Generalisation First, Memorisation Second? Memorisation Localisation for Natural Language Classification Tasks (2024)

Abstract: Memorisation is a natural part of learning from real-world data: neural models pick up on atypical input-output combinations and store those training examples in their parameter space. That this happens is well-known, but how and where are questions that remain largely unanswered. Given a multi-layered neural model, where does memorisation occur in the millions of parameters? Related work reports conflicting findings: a dominant hypothesis based on image classification is that lower layers learn generalisable features and that deeper layers specialise and memorise. Work from NLP suggests this does not apply to language models, but has been mainly focused on memorisation of facts. We expand the scope of the localisation question to 12 natural language classification tasks and apply 4 memorisation localisation techniques. Our results indicate that memorisation is a gradual process rather than a localised one, establish that memorisation is task-dependent, and give nuance to the generalisation first, memorisation second hypothesis.

heading: 1 Introduction
text: Memorisation in neural models is both concerning due to overfitting and privacy concerns, and desired because of information that needs to be stored, such as facts. With the recent surge in the number of models trained on very large, closedaccess training corpora, the NLP community has seen an increased interest in research questions that aim to improve our understanding of memorisation, such as: Which data points are memorised, and can we extract those examples from models (e.g. Chang et al., 2023a; Shi et al., 2023; Nasr et al., 2023)? How dependent is memorisation on model scale, architecture and training procedures (e.g. Carlini et al., 2022)? Can we localise memorised information (i.e. pinpoint which weights, subcomponents or layers are most associated with storing that information) and edit models’ memories (e.g. Meng et al., 2022a; Hase et al., 2024)?

heading: Generalising while memorising
text: Memorisation localisation is the central theme in this work, specifically localisation at the level of layers. There is a lack of consensus about which layers are particularly involved in memorisation in deep neural models, which may stem from the varying experimental setups and varying definitions of memorisation employed by different studies. Work from computer vision (CV) mostly focused on memorisation of perfectly memorised mislabelled examples, positing that lower layers capture generalisable features while deeper layers memorise (e.g. Baldock et al., 2021; Stephenson et al., 2021) (although that has been recently challenged by Maini et al. (2023)). Work from NLP often discusses memorisation of facts, for which lower (Geva et al., 2023), middle (Meng et al., 2022a) and final layers (Dai et al., 2022) have all been mentioned as playing crucial roles in pre-trained language models (PLMs). And lately, with the availability of some open-access pre-training data, memorisation localisation for sequences memorised verbatim has gained traction, and initial results primarily point to lower layers (Stoehr et al., 2024). Different articles investigating different types of memorisation arrive at different answers, and even work focused on a specific type (such as facts) does not always agree.
We contribute an important piece of the puzzle in the memorisation localisation landscape, by employing a setup that is similar to Stephenson et al. and Maini et al.: we perform layer-wise localisation in fine-tuned models for 12 NLP classification
14348
tasks, enforcing memorisation by applying label perturbation to a data subset. We use four memorisation localisation methods (§3), and first examine their accuracy in a control setup. Afterwards (§4), we address the main research question: In which layers does memorisation occur? The results do not always align – which underscores that we should not overly rely on one localisation method – but do tell a coherent story. In §5, we introduce a visualisation technique (centroid analysis) to make this story more interpretable: memorisation is not sudden but gradual. Together, layers gradually shift mislabelled examples to their newly assigned class, but when that happens is task-dependent: the better a model generalises to new data for a particular task, the more relevant deeper layers are for memorisation. Figure 1 illustrates this. Our findings beg a nuance of the generalisation first, memorisation second hypothesis, and we end with a discussion (§6) of what our findings mean for localisation and model editing going forward.

heading: 2 Related work
text: In this section, we discuss a broad range of related work on memorisation localisation from CV and NLP, focusing specifically on studies that discuss the role of different layers in deep neural networks.
Noise memorisation in CV Multiple image classification studies concluded that deeper layers are more involved in memorisation than earlier layers, based on analyses of the memorisation of mislabelled or hard examples. This has been reported by work contrasting entire models (regular models and ones trained with randomised labels) (Morcos et al., 2018; Cohen et al., 2018; Ansuini et al., 2019, i.a.), and work discussing how some examples are handled differently within one model: Baldock et al. (2021) establish a positive correlation between prediction depth in image classification (the earliest layer that predicts the label) and example-level learning difficulty metrics. Stephenson et al. (2021) analyse hidden representations for image classification and report that memorisation of mislabelled examples occurs abruptly in late layers and late training epochs. Rewinding models’ final layers to earlier checkpoints reversed memorisation.
Contrary to previous work, Maini et al. (2023) report that for image classification datasets that were partially mislabelled, instead of memorisation being confined to specific layers, there are small sets of neurons dispersed over the full architecture involved in memorisation.
Memorisation of factual knowledge NLP memorisation localisation studies have primarily focused on factual knowledge, although only a subset of work in this direction discusses the roles of different layers. De Cao et al. (2021) first connected work from CV to fact memorisation in transformer, and train a hypernetwork to edit facts. Their hypernetwork mostly edits the bottom layer of a six-layer transformer, and De Cao et al. suggest this difference might be due to the change in modality.
Later work operated under the assumption of the knowledge neuron thesis,1 assuming that facts are recalled from the training corpus through transformer’s MLP weights, that act as a key-value memory (Geva et al., 2021), and that one may thus be able to identify MLP knowledge neurons (Dai et al., 2022): (1) Meng et al. (2022a,b) edit factual memories in transformer-based PLMs’ MLPs by first localising memorisation to specific layers and only updating those layers, in which case early/midlayers are most often selected.2 Note that Hase et al. (2024) find success in model editing to be unrelated to the layers selected by Meng et al.’s localisation method, which means that model editing might be an unreliable way to check where facts are stored. (2) Dai et al. (2022) identify knowledge neurons for factual information, and mostly find neurons in the top layers of BERT, with similar findings being reported in later work on knowledge neurons by Zhao et al. (2024) and Chen et al. (2024). Dai et al. then succesfully use those neurons to update facts.
Although the articles above disagree in terms of whether higher or lower layers store factual information, work beyond model editing and knowledge neurons indicates factual information is already present in the lower layers: Haviv et al. (2023) show that facts and idioms are stored and retrieved in early layers in BERT and GPT-2. Deeper layers perform confidence boosting after retrieval. Geva et al. (2023) artificially block parts of the computation in fact retrieval for GPT-J and GPT-2 XL, and find that factual information is stored in the lower MLP sublayers, which is echoed by recent work from Ortu et al. (2024).
Verbatim memorisation While memorisation localisation has predominantly focused on facts,
1The term was coined by Niu et al. (2024) to summarise the hypothesis underlying multiple related studies. Niu et al. criticise the thesis since it oversimplifies knowledge storage. Instead, they suggest to focus on network-wide circuits.
2E.g. generalisation to paraphrased prompts of modified facts peaks when editing layer 18 out of 48 for GPT-2 XL. This result is not specific to transformer: Sharma et al. (2024) find early/mid-layers to be important when editing facts in Mamba.
recent open science initiatives publishing PLM pretraining corpora enable research on memorisation of pre-training data. Chang et al. (2023b) present a dataset to evaluate different localisation methods by relying on text memorised verbatim by three PLMs, but they do not elaborate on the roles of layers. Stoehr et al. (2024) examine for one specific 12-layer architecture (GPT-Neo-125M, that we will also analyse) which model components are responsible for memorising sequences of 50 tokens verbatim, finding lower layers to be the most relevant. Localisation of verbatim memorisation is hard to standardise across different tasks and models due to the required access to pre-training data, and because all PLMs memorise different sequences.
Memorisation beyond localisation Other work on memorisation in NLP examines the conditions under which memorisation occurs during finetuning (Tänzer et al., 2022; Mireshghallah et al., 2022), which fine-tuning tasks lead to the most memorisation (Zeng et al., 2023), which examples are memorised (Biderman et al., 2024) and when memorisation is beneficial for generalisation (Zhang et al., 2023; Zheng and Jiang, 2022).
Summarising, many different conclusions have been drawn in memorisation localisation studies. It is unclear whether the different conclusions from different articles may be reduced to a difference between the vision and language modalities, to a difference between the different types of memorisation investigated, to localisation techniques employed or even to the varying models and model sizes used in related work.3 We take away some of that confusion by (1) employing a setup previously used for the vision modality and (2) investigating a type of memorisation (‘noise memorisation’) understudied in NLP, using widely studied models. This allows us to conclude whether the ‘deeper layers’ answer from CV really stands in contrast with the ‘lower layers’ answer from the majority of NLP studies (or whether that was simply unique to noise memorisation), and allows us to investigate whether the ‘lower layers’ answer is unique to fact memorisation and verbatim memorisation.

heading: 3 Methods
text: To gain a good understanding of how memorisation is task-dependent, we combine binary classification
3For instance, De Cao et al. investigate 6-layer transformers, whereas Dai et al. and Stoehr et al. use 12-layered networks, and Meng et al. mostly focus on GPT-2 XL with 48 layers.
tasks from (Super)GLUE (Wang et al., 2018, 2019) with tasks from more diverse domains and label set sizes. The tasks fall into four categories: generic natural language understanding (NLU), sentimentrelated tasks, hate speech detection and topic classification. Table 1 enumerates the tasks, the datasets’ domains, and the training set and label set sizes. For each dataset, we perturb the labels of 15% of the training examples (‘noisy’ examples, x ∈ Xn, y ∈ Yn), with the new label randomly drawn from all labels but the original one. The remaining 85% is unperturbed (‘clean’ examples, x ∈ Xc, y ∈ Yc).
We analyse four PLMs: BERT-base (Devlin et al., 2019), OPT-125m (Zhang et al., 2022), Pythia-160m (Biderman et al., 2023) and GPT-Neo-125m (Black et al., 2021; Gao et al., 2020). We fine-tune each model separately for the 12 tasks. Appendix D describes the models and our technical setup.4 These 4 architectures are similar in size and have 12 layers each. In Appendix B, we repeat a subset of the experiments with OPT-1.3B.
The PLMs (θP ) are fine-tuned for 50 epochs, and checkpoints are stored when the training accuracy is near-ceiling (θM1), and at the end of training (θM2). We also train models using the original labels (θO), using the same random seeds as θM1 and θM2 . During fine-tuning, we freeze the input embeddings. Results reported in §3.2 are based on one fine-tuning seed, and the remainder of the main paper computes results using three seeds. Seeds control the data order and classification heads.

heading: 3.1 Localisation techniques
text: We apply four localisation methods that are detailed in this subsection and further evaluated in §3.2.
Layer retraining and layer swapping First, we perform layer retraining, similar to Maini et al. (2023). We reset layers of interest using weights from θP , freeze the remaining layers using weights from θM2 , and retrain using clean examples for five epochs. If the resulting model maintains its performance on noisy data, the retrained layers are redundant in terms of memorisation. If the performance on noisy data decreases, that does not guarantee that memorisation can be localised to the retrained layers since the retraining objective may have multiple minima, of which only some maintain the memorisation performance. We retrain consecutive layers of window sizes ranging from 1 to 12.
4The codebase is available at: https://github.com/ vernadankers/memorisation_localisation.
Category Dataset Task Domain Size Labels
NLU
WiC
0 2 4 6 8 10 0
2
4
6
8 10 WiC RTE MRPC CoLA BoolQ SST-2 SST-5 Emotion ImplicitHate Stormfront Reuters TREC by Pilehvar and Camacho-Collados (2019) word sense disambiguation Word-/Verb-net, Wiktionary 5.4k 2 RTE 0 2 4 6 8 10 0 2 4 6 8 10 i RTE MRPC CoLA BoolQ SST-2 SST-5 Emotion ImplicitHate Stormfront Reuters TREC by Dagan et al. (2006) textual entailment news, Wikipedia 2.5k 2 MRPC 0 2 4 6 8 10 0 2 4 6 8 10 WiC RTE MRPC CoLA B olQ ST-2 SST-5 Emotion ImplicitHate Stormfront Reuters TREC by Dolan and Brockett (2005) paraphrase classification news 3.7k 2 CoLA 0 2 4 6 8 10 0 2 4 6 8 10 i P oLA B ol ST-2 ST-5 E otion Impl citHate Stormfront Reuters TREC by Warstadt et al. (2019) labelling grammatically linguistic theory books 8.5k 2 BoolQ 0 2 4 6 8 10 0 2 4 6 8 10 WiC RTE MRP LA BoolQ SST-2 SST-5 Emotion ImplicitHate St rmfront Reuters TREC by Clark et al. (2019) question answering given a context Google queries, Wikipedia 9.4k 2 Sentiment SST-2 0 2 4 6 8 10 0 2 4 6 8 10 WiC RTE MRPC CoLA B olQ ST-2 ST-5 Emotion ImplicitHate Stormfront Reuters TREC by Socher et al. (2013) sentiment classification movie reviews 6.9k 2 SST-5 0 2 4 6 8 0 2 4 6 8 0 WiC RTE MRPC CoLA B ol - - i i i t t by Socher et al. (2013) sentiment classification movie reviews 8.5k 5 Emotion
0 2 4 6 8 10 0
2
4
6
8
10 WiC RTE MRPC
oLA BoolQ SST-2 SST-5 Emotion ImplicitHate St rmfront Reuters TREC by Saravia et al. (2018) emotion classification tweets 16k 6 Hate speech ImplicitHa e 0 2 4 6 8 10 0 2 4 6 8 10 WiC RTE MRPC CoLA BoolQ SST-2 SST-5 Emotion ImplicitHate Stormfront Reuters TREC by ElSherief et al. (2021) hate speech classification tweets 5.1k 7 Stormfron 0 2 4 6 8 10 0 2 4 6 8 10 WiC RTE MRPC CoLA BoolQ SST-2 SST-5 Emoti n ImplicitHat Stormfront Reuters TREC by de Gibert et al. (2018) hate speech classification social media 8.6k 2 Topic Reuters 0 2 4 6 8 10 0 2 4 6 8 10 WiC RTE MRPC CoLA BoolQ SST-2 SST-5 Emotion ImplicitHate Stormfront Reuters TREC by Apte t al. (1994) topic classification news 5k 8 Classification TREC2 4 6 8 10 10 i oL B ol T-2 ST-5 E oti n I plicitHate Stormfront Reut rs TRECby Li and Roth (2002); Hovy et al. (2001) topic classification news, misc. 5.5k 6
Table 1: Datasets with their domain, label set size and training set size. In §4 and §5, datasets are marked consistently using the same colours and symbols.
Alternatively, we swap layers between θM2 and θO, using the same window sizes. If swapping layers leads to a drop in performance on noisy examples while maintaining performance on clean ones, it becomes more likely that the layers were vital for memorisation (although this is again not guaranteed). We indicate layer relevance using the memorisation error: the ratio of incorrect predictions for noisy examples. The lower the error rate for noisy examples when retraining or swapping a layer, the less likely it is that this layer was crucial for memorisation.
Retraining or swapping all 12 layers means modifying the full model, and provides a baseline for the maximum error we can expect for the noisy data. In the results section, we will use this to normalise the results, such that the memorisation error is 1.0 when modifying all 12 layers.
Forgetting gradients We also inspect gradients, computed by back-propagating −L(Xn,Yn, θM1) and computing the L1-norm per layer. We use θM1 due to gradient saturation in θM2 .
5 The assumption is that memorisation is localised in the layers requiring the largest updates when ‘forgetting’ noisy labels. Because gradient magnitudes do not reliably pinpoint layers, we used two tasks to decide on the norm to use and whether or not to normalise gradients using gradients for clean examples and gradients for a frozen model (Appendix E).
Probing Lastly, we train probing classifiers (Conneau et al., 2018) to predict whether, for a hidden state encoding x in layer l (hxl ), x ∈ Xn or x ∈ Xc. The classifier is an MLP with one hidden layer, trained for 100 epochs maximum with a learning rate of 2e−4. The hidden states come from training examples that are redistributed into a training,
5See Akyürek et al. (2022) for a discussion of issues with gradient-based methods when tracing knowledge in a model.
validation and test set for the probe. The classifiers are trained separately per layer, using five random seeds per layer. We extract the F1-score on the test partitions and use the increase from l−1 to l as an indication of l’s involvement in memorisation (except for layer 1, which we compare to the F1-score from a probe trained on θP ).

heading: 3.2 Control setup: does localisation succeed?
text: We now evaluate the localisation techniques by enforcing memorisation in prespecified layers and examining whether the techniques pinpoint those layers (i.e. whether localisation succeeds).
Experimental setup We approach this as a multitask learning setup, to ensure all layers are finetuned, but only two are modified by the task with noisy labels: the entire model is fine-tuned using RTE, while the remaining task can only modify two layers at a time (layers 1 and 2, 6 and 7 or 11 and 12). We train the model separately for the remaining 11 tasks and these 3 different choices of layer combinations. Afterwards, we first use MRPC and TREC to validate the postprocessing steps for the forgetting gradients (see Appendix E), after which all localisation techniques were applied to the remaining nine tasks. We evaluate the techniques using accuracy@k, indicating the percentage of the k highest-scoring layers that were among the correct ones for that setup, computed for k ∈ {1,2}.
Results Figure 2 (left) summarises the accuracies per localisation technique. Swapping and retraining are very accurate, but gradients and probing are less reliable, with accuracy@1 just over 60%. Note that the near-perfect accuracy for retraining and swapping here does not guarantee perfect accuracy in the uncontrolled setup; the per-layer freezing is just very well-aligned with the per-layer approach of those techniques. The accuracy per dataset (Figure 2, middle) only shows slight variations. For the two lowest-scoring localisation techniques (probing, gradients), Figure 2 (right) details the accuracies per model. Pythia scores particularly badly for the gradient analysis, for which the accuracies barely exceed the baseline. Postprocessing (Appendix E) did not help, which underscores gradients’ unreliability.

heading: 4 Results for memorisation localisation
text: We now apply the localisation techniques to models in which all layers have been fine-tuned for one task at a time. The results indicate how important each layer is for memorisation, per dataset, per model. We cannot simply aggregate over all results (12 layers × 12 datasets × 4 localisation techniques × 4 models), because the absolute scores returned by different techniques are not directly comparable. We discuss the results per localisation technique.

heading: 4.1 Layer swapping and retraining
text: When swapping or retraining layers, we gradually modify more and more layers in θM2 , either using weights from θO, or by retraining layers using clean
examples.6 We modify 1 to 12 layers at a time, and measure the effect via the memorisation error.
Case study: RTE vs SST-2 Before discussing trends across all 12 datasets, we inspect two specific sets of results to gain a deeper understanding of the data. Figure 3 details memorisation error rates for RTE and SST-2 (for OPT): in these matrices, value z in row x, column y, indicates that for all layer combinations of x consecutive layers including y, z was the mean error rate. We show the results separately for swapping and retraining.
What commonalities and differences do we observe? For both datasets, modifying a few layers only yields low error rates (see the top few light green rows), and fully reverting memorisation requires modifying 7 to 10 layers. Memorisation is thus not limited to a few layers, but, instead, dispersed over the model. Despite these similarities, the datasets differ in which layers appear the most crucial for memorisation: for RTE, modifying early layers leads to the largest increase in memorisation error, whereas for SST-2, both the very first layers and layers in the middle appear most relevant.
Aggregating results The findings for these two tasks are echoed in the overall swapping and retraining results. Firstly, memorisation is not confined to individual layers: modifying individual layers barely affects the memorisation error. This is shown in Figure 4, which provides the memorisation error when modifying one layer only, taking the maximum over layers (i.e. highlighting the largest error increase), showing datasets as dots. For most model-dataset combinations, the memorisation error rate is below 15% when modifying one layer. This agrees with findings from Maini et al. (2023), who similarly employed layer retraining to identify that memorisation in image classification
6When swapping layers, we monitor errors on clean examples to ensure that the mixture of models θO and θM2 differs only in terms of predictions for noisy examples. The mean error for clean examples over all windows was 0.3%.
0 2 4 6 8 10
0
2
4
6
8
10
WiC RTE
MRPC CoLA
BoolQ SST-2
SST-5 Emotion
ImplicitHate Stormfront
Reuters TREC
0.40
0.60
0.80
m em
or isa
tio n
er ro
r
0.05
0.10
0.15
gr ad
ie nt s po st pr oc es
se d
1 2 3 4 5 6 7 8 9 10 11 12 layer
0.25 0.50 0.75 pr ob in g
F 1
(a) NLU tasks
1 2 3 4 5 6 7 8 9 10 11 12 layer
(b) Sentiment tasks
1 2 3 4 5 6 7 8 9 10 11 12 layer
(c) Hate speech tasks
1 2 3 4 5 6 7 8 9 10 11 12 layer
(d) Topic classification
Figure 5: Memorisation localisation for OPT: (1) layer swapping error rates, higher numbers suggest higher relevance. (2) gradient norms, higher numbers suggest higher relevance. (3) probing F1-scores when training probes to predict whether an example is noisy. The increase between layers suggests layer relevance.
is not confined to individual layers.
Secondly, the importance of layers does appear task-dependent. To investigate this more systematically, we express layer relevance using the mean memorisation error (averaging over rows in the result matrices similar to the ones in Figure 3). Figure 5 (top row demonstrating layer swapping) details this error per dataset for OPT. Across the board, early layers matter more for memorisation, but that is more prominent for the NLU tasks than for the other tasks, and for Stormfront
0 2 4 6 8 10 0
2
4
6
8 10 WiC RTE MRPC CoLA BoolQ SST-2 SST-5 Emotion ImplicitHate Stormfront Reuters TREC
there is even a slight increase in the error for deeper layers. For the remaining models, the same visualisation is shown in Appendix A.1, and we can summarise the per-layer weights by computing a Memorisation Centre-of-Gravity (M-CoG), which is a weighted sum of all layers with weights summing to 1: ∑12i=1 αi ⋅ i. For layer swapping and retraining, αi is the memorisation error for layer i, as
depicted in Figure 5. Figure 6 displays the M-CoG coefficients for layer retraining, per model, and Figure 7 provides M-CoG coefficients per dataset by averaging over models (left) and over localisation techniques (right). The results show strong agreement between models in terms of the relative ordering of tasks – the average pairwise correlation of the data from Figure 6 is 0.85 (Spearman’s ρ) – and between layer swapping and layer retraining – Figure 8 (left) includes rank correlations for the MCoG coefficients, and Figure 9 (left) includes rank correlations for raw layer weights. Both indicate strong agreement between the two techniques.

heading: 4.2 Probing
text: Figure 5 (bottom row) displays the probing performance for OPT, and the increase from layer to layer indicates layers’ relevance. The first observation is that the performance typically does not decrease for deeper layers, i.e. representations do not ‘lose’ information about the fact that some examples are noisy. Secondly, the performance is quite low for NLU tasks, especially, which could mean that clean and noisy examples are more alike for these tasks than for the remaining tasks. Lastly, in accordance with the previous results, probing performance does not change suddenly (i.e. memorisation is not local to individual layers), and tasks differ in how the probing performance changes over layers: performance flattens early for some tasks (e.g. RTE
0 2 4 6 8 10 0
2
4
6
8 10 WiC RTE MRPC CoLA BoolQ SST-2 SST-5 Emotion ImplicitHate Stormfront Reuters TREC ) but gradually improves over all layers for others (e.g. Emotion 0 2 4 6 8 10 0 2 4
6
8 10 WiC RTE MRPC CoLA BoolQ SST-2 SST-5 Emotion ImplicitHate Stormfront Reuters TREC , ImplicitHate 0 2 4 6 8 10 0 2 4 6 8 10 WiC RTE MRPC CoLA BoolQ SST-2 SST-5 Emotion ImplicitHate Stormfront Reuters TREC ). Appendix A.1 provides results for the other models; for Pythia, probing performance peaks earlier 14353
than for the other models, indicating that the lower layers are extra important for this model.
To draw more generic conclusions, we compute the M-CoG coefficients by using the per-layer increase in probing performance as weights. Figure 7 (left) includes the M-CoG averaged over models and demonstrates that probing puts a larger emphasis on deeper layers compared to layer swapping and retraining. The M-CoG of probing have a moderately positive correlation to the swapping and retraining coefficients (see Figure 8), and raw weights per layer have a weakly positive correlation to swapping and retraining (see Figure 9).

heading: 4.3 Gradient analysis
text: Finally, we inspect the gradient norms, postprocessed as described in Appendix E. Results obtained using the forgetting gradients correlate quite strongly with layer swapping and retraining (Figure 8, Figure 9). That agreement can also be seen when visually inspecting the norms per layer for OPT (middle row of Figure 5, see Appendix A.1 for the remaining models): NLU tasks have higher scores in earlier layers, SST-2
0 2 4 6 8 10 0
2
4
6
8 10 WiC RTE MRPC CoLA BoolQ SST-2 SST-5 Emotion ImplicitHate Stormfront Reuters TREC and TREC0 2 4 6 8 10 0 2 4
6
8 10 WiC RTE MRPC CoLA BoolQ SST-2 SST-5 Emotion ImplicitHate Stormfront Reuters TREChave a
more uniform distribution, and Stormfront
0 2 4 6 8 10 0
2
4
6
8 10 WiC RTE MRPC CoLA BoolQ SST-2 SST-5 Emotion ImplicitHate Stormfront Reuters TREC and

heading: Reuters
text: 0 2 4 6 8 10 0
2
4
6
8 10 WiC RTE MRPC CoLA BoolQ SST-2 SST-5 Emotion ImplicitHate Stormfront Reuters TREC
point to deeper layers (although the gradient norms show a slight increase for the final layer for multiple tasks). At the same time, the gradient analysis weakly correlates to the probing results, potentially because both methods have much lower accuracies than swapping/retraining. The forgetting gradients failed to pinpoint one model’s correct layers in the control setup (§3.2). That gradients agree with swapping/retraining supports our overall findings, but we recommend against relying solely on gradients for memorisation localisation.

heading: 4.4 Intermediate conclusion
text: In this section, we have taken a closer look at the localisation results for OPT, and inspected aggregated results for all techniques and models via M-CoG coefficients. Because memorisation is not strictly
localised to individual layers, these coefficients lie close to the middle layer, but they do generally skew towards earlier layers and provide us with an ordering of tasks. The most notable pattern in that ordering that the earlier layers are the most important for the NLU tasks. This is somewhat surprising since the NLP community would typically consider NLU tasks to be more complex than topic classification or sentiment detection, and assumes higher-level tasks to be processed in higher layers.7 If that is the case, it seems natural for memorisation to also happen in higher layers, but this is contradicted by the experiments.
While this section has concentrated primarily on the comparison of localisation methods, we finally note that when computing correlations between models (Figure 8-9, right), these are strongly positive, except for Pythia, yielding more moderate correlations. That suggests that our results are not specific to one training setup, but somewhat generic to 12-layer transformer-based PLMs.

heading: 5 Making memorisation interpretable via centroid analysis and probing
text: The results from §4 suggested that earlier layers are the most relevant for memorisation. To better understand why, we make models’ internal processing of memorised examples more interpretable through a centroid analysis: we examine pairs of
7E.g. Müller-Eberstein et al. (2023) show that for topic classification in BERT (using unperturbed datasets), the centreof-gravity as defined by Tenney et al. (2019) lies around layer 4/5 for topic classification, whereas for NLI it is layer 11.
centroids of classes a and b, projected onto line
distribution of examples of class b, projected onto line
distribution of examples of class a, projected onto line
distribution of noisy examples
classi cation initiation crossing
noisy examples of class b, assigned class a
distributions
show 75% intervals
(a) TREC
1 2 3 4 5 6 7 8 9 10 11 12 (b) RTE 1 2 3 4 5 6 7 8 9 10 11 12 (c) WiC
1 2 3 4 5 6 7 8 9 10 11 12 (d) SST-2 1 2 3 4 5 6 7 8 9 10 11 12
(e) Stormfront
Figure 10: Centroid analysis visualises how noisy examples gradually change, for five datasets, for OPT.
1 2 3 4 5 6 7 8 9 10 11 12 (a) RTE, regular bottom 1 2 3 4 5 6 7 8 9 10 11 12 (b) RTE, regular top
1 2 3 4 5 6 7 8 9 10 11 12 (c) TREC, regular bottom 1 2 3 4 5 6 7 8 9 10 11 12
(d) TREC, regular top
Figure 11: Illustration of the effect replacing 6 layers has, for RTE and TREC data, for OPT. We insert 6 layers from θO at the lower (‘bottom’) or upper half (‘top’).
classes, monitoring examples with original class yb and noisy class ya, for all pairs of a and b. We compute the centroids of the hidden representations from the two classes and project all data points from those two classes onto the line through the centroids. We measure the distance to centroid a for every data point, normalised by the distance between the two centroids. This is performed separately per layer. In layer 1, points belonging to ya and yb largely overlap. Towards layer 12, the two classes are fully separated, and in between, the memorised examples move away from centroid b and move towards centroid a. Figure 10a explains this via annotations for TREC, and Figures 10b-10e do so for four additional tasks that are illustrative of the variety that we observe. We include figures for all tasks and models in Appendix A.3.
This visualisation indicates that memorisation occurs through gradual changes from the first layers onward. This explains the results from the previous section, where we found that memorisation is not confined to individual layers and that lower layers were more successful in reverting memori-
sation (in layer swapping/retraining) than deeper layers: memorisation starts early, and interventions are more successful when conducted before the hidden state has moved too far away from class yb. We can demonstrate this using the centroid analysis, applied while swapping six layers at the bottom or top with layers from θO. For all tasks (see Appendix A.3), we can prevent the noisy examples from moving to centroid a by replacing the bottom six, but for only a few tasks, replacing the top six has a similar effect. Figure 11 shows this using RTE and TREC. Swapping the bottom six prevents emitting the noisy class for RTE and TREC, but swapping the top six is more successful for TREC than RTE.
Task differences To summarise task differences, we compute two statistics: the crossing (the first layer in which the noisy mean is closer to a than to b) and classification initiation (the first layer without overlapping distributions for the two classes), shown in Figure 12a. Many NLU tasks have an early crossing and a late classification initiation (e.g. RTE
0 2 4 6 8 10 0
2
4
6
8 10 WiC RTE MRPC CoLA BoolQ SST-2 SST-5
Emotion ImplicitHate Stormfront Reuters TREC and WiC
0 2 4 6 8 10 0
2
4
6
8 10 WiC RTE MRPC CoLA BoolQ
SST-2 SST-5 Emotion ImplicitHate Stormfront Reuters TREC ). The two events are closer together for hate speech and sentiment tasks, and
topic classification tasks (TREC0 2 4 6 8 10 0
2
4
6
8
10 WiC RTE
MRPC CoLA BoolQ SST-2 SST-5 Emotion ImplicitHate Stormfront Reuters TRECand Reuters
0 2 4 6 8 10 0
2
4
6
8 10 WiC RTE MRPC CoLA BoolQ SST-2 SST-5 Emotion ImplicitHate Stormfront Reuters TREC
) start classification early but have a late crossing. This confirms findings from §4: lower-level tasks (early classification initiation) rely more heavily on deeper layers for memorisation than higher-level tasks (late classification initiation). Yet, tasks with similar classification initiations (e.g. SST-5
0 2 4 6 8 10 0
2
4
6
8 10 WiC RTE MRPC CoLA BoolQ SST-2 SST-5 Emotion ImplicitHate Stormfront Reuters TREC and

heading: Emotion
text: 0 2 4 6 8 10 0
2
4
6
8
10 WiC RTE MRPC
CoLA BoolQ SST-2 SST-5 Emotion ImplicitHate Stormfront Reuters TREC ) can still have different crossings.
Consolidation via probing The centroid analysis merely visualises representations. To consolidate that we reach similar conclusions using different methods, we train probes to predict an example’s class from the hidden state, using (i) original or (ii) noisy labels. In Appendix A.2, we include the probes’ performances. We apply the probes to noisy examples and compute a statistic similar to the crossing: the layer at which the F1 of probe ii exceeds the F1 of probe i with ten percentage points, referred to as ‘memorisation»generalisation’ in Figure 12b. The timing of this event strongly correlates with the crossings (Spearman’s ρ = 0.84). We apply the probes to clean examples to compute a statistic similar to the classification initiation: the layer at which the probes’ F1 for clean examples (normalised by random guessing performance) reaches 90%. The depth of this event strongly correlates with classification initiation (ρ = 0.73). Together, these two events thus tell a story similar to that of the centroid analysis (Figure 12b).
0 2 4 6 8 10
0
2
4
6
8
10
WiC RTE MRPC
CoLA BoolQ SST-2
SST-5 Emotion ImplicitHate
Stormfront Reuters TREC
1 2 3 4 5 6 7 8 9 10 11 12 crossing
1
2
3
4
5
6
7
8
9
10
11
12
cla ss
ifi ca
tio n
in iti
at io
n
(a) Centroid analysis
1 2 3 4 5 6 7 8 9 10 11 12 memorisation >> generalisation (noisy)
1
2
3
4
5
6
7
8
9
10
11
12
90 %
F 1 (
cle an
)
(b) Probing
Figure 12: Summary of the memorisation and classification onset for all datasets, averaged over models, computed using the centroid analysis or via probing.
Correlates all BERT OPT GPT-N Pythia
- Generalisation score crossing 0.75 0.88 0.94 0.94 0.72 mem. » gen. 0.63 0.86 0.88 0.94 0.69 M-CoG 0.56 0.78 0.69 0.92 0.69 - Validation score crossing 0.70 0.90 0.84 0.83 0.70 mem. » gen. 0.61 0.90 0.77 0.76 0.77 M-CoG 0.54 0.80 0.52* 0.72 0.69
Table 2: Spearman’s ρ relating memorisation for the 12 tasks to models’ generalisation performances. *: p>0.05

heading: Memorisation’s connection to generalisation
text: When inspecting model internals, we have seen that the depth of memorisation (quantified as MCoG coefficients, the ‘crossing’ and ‘memorisation»generalisation’) appears anti-correlated with the difficulty of a task. However, we have yet to have a proper way of quantifying that difficulty. We now take models’ validation accuracy at the end of training (on data unseen during training, normalised by random guessing performance) and compute a generalisation score (percentage of training examples for which the probability of the target exceeds random guessing when that example is held out from training).8 As indicated in Table 2, these two metrics correlate moderately with the memorisation depth when combining data from all models (Spearman’s ρ > 0.54), with most correlations being stronger when examining results per model. All in all, this suggests that the better a model generalises a task to new data, the more later layers are involved in memorisation.9
8Computed by training on a randomly selected 50% of the data, and testing on the held-out 50%, repeated with 30 random seeds. This approximates metrics reported by Feldman (2020); Feldman and Zhang (2020); Jiang et al. (2021). We adopt the naming of the metric from Dankers et al. (2023).
9Tasks with late crossings are mostly multi-class tasks. To ensure that this is not a confound here, Appendix C repeats some of the analyses with binary versions of these tasks.

heading: 6 Discussion
text: We set out to perform memorisation localisation for natural language classification tasks by perturbing a subset of the labels and tracing those ‘noisy’ examples over layers. Applying four localisation techniques to four models crystallised that memorisation is not local to specific layers but a cooperative process of weights from many layers. Nonetheless, not all layers appear equally important. Overall, early layers are more important than later ones: the model’s manipulation of memorised examples starts in lower layers, and to prevent memorisation, early intervention is more successful than late intervention. We discussed results for 12-layer models in the main paper, but further experimentation with a 24-layer model (in Appendix B) leads to similar findings of memorisation being gradual, with a similar ordering of tasks, but mid-range layers being more important than late layers.
This is not in accordance with the generalisationfirst, memorisation-second hypothesis from CV (see §2), but does agree with more recent work on image classification by Maini et al. (2023). It also aligns with related work on PLMs for fact memorisation and verbatim memorisation pointing to the lower layers (Geva et al., 2023; Stoehr et al., 2024), while also describing cooperative roles for earlier and deeper layers (Haviv et al., 2023).
The fact that memorisation is not local implies that editing model weights locally does not necessarily erase memorised information, even if a flipped label suggests this at the level of the output layer. This might be harmless when editing facts about named entities like cities (e.g. Meng et al., 2022a), yet is more worrisome when regarding memorisation of personal information (e.g. Carlini et al., 2021), and may be a reason why safety measures can easily be reversed in PLMs modified to reduce harmful outputs (e.g. Zhan et al., 2023).
Can we, due to the importance of early layers, conclude that our results falsify the generalisation first, memorisation second hypothesis? The results from §5 suggest that this question requires a nuanced answer due to the variation observed among tasks. The depth of memorisation is positively correlated with a model’s generalisation capabilities, i.e. we do observe a generalisation first, memorisation second tendency but the better the model performs at a certain task, the stronger that tendency is. We consider better understanding what properties of a task direct memorisation to lower or higher layers an exciting avenue for future work.

heading: Limitations
text: We identify four main limitations of our work:
• Simplified data: To trace memorised examples over the transformer’s many layers, we resorted to label flipping to create ‘noisy’ examples. This situation is somewhat unnatural when considering real-world examples that require memorisation from the model. For example, in the case of sentiment analysis, that might be a sarcastic phrase whose sentiment is the opposite of what is expected based on a literal interpretation. We cannot guarantee that our noisy examples behave in the same way as real-world examples would. Similarly, memorisation of noisy examples need not affect models in the same way as the memorisation of factual information or verbatim memorisation of long strings. As laid out in the introduction (§1), we opted for this type of data manipulation to create an experimental setup that more closely resembles that of related work from CV.
• Localisation techniques are imperfect: As identified in §3.2 the localisation techniques applied are themselves flawed: in a control setup where only two layers were modified during fine-tuning, probing and gradient analyses could not accurately pinpoint those two layers, and the techniques that could pinpoint them (layer swapping and layer retraining) are more reliable at determining which layers are not crucial for memorisation than at determining which ones are. Because of the general agreement between the techniques and the results from §4-5 we do think our conclusions are robust, but the absolute numbers of layer relevance should be taken with a grain of salt.
• Visualisation ≠ localisation: In §5 we introduced the centroid analysis as a way of visualising what is happening to examples over the different layers. This visualisation is a one-dimensional projection of hidden representations and thus an extreme simplification of the intricate process of memorisation. We do not mean to use it as a localisation technique, but as a way to explain the outcomes of other experiments in the paper.
• Lack of evidence for individual examples: We analysed the group of noisy examples
as a whole, and concluded that many layers work together to gradually shift examples from their original class to the newly assigned class. However, we have not examined individual examples; it can still be the case that for individual examples, memorisation is more localised to specific layers. We only have preliminary results suggesting that individual examples, too, are memorised over multiple layers, which is the fact that in §4, swapping/retraining individual layers was unsuccessful in increasing the memorisation error rate.

heading: Acknowledgements
text: VD is supported by the UKRI Centre for Doctoral Training in Natural Language Processing, funded by the UKRI (grant EP/S022481/1) and the University of Edinburgh, School of Informatics and School of Philosophy, Psychology & Language Sciences. IT is supported by the Dutch National Science Foundation (NWO Vici VI.C.212.053).

heading: A Extended results

heading: A.1 Main results (§4)
text: We provide the same visualisation of results as shown for OPT in §4 in Figures 13a, for BERT, 13b, for GPT-N, and in 13c for Pythia. We omit layer retraining, that correlates very strongly with layer swapping (shown in the top rows of each subfigure).
0 2 4 6 8 10
0
2
4
6
8
10
WiC RTE
MRPC CoLA
BoolQ SST-2
SST-5 Emotion
ImplicitHate Stormfront
Reuters TREC
NLU tasks Sentiment tasks Hate speech tasks Topic classification
0.40
0.60
0.80
m em
or isa
tio n
er ro
r
0.05
0.10
0.15
gr ad
ie nt s po st pr oc es
se d
1 2 3 4 5 6 7 8 9 10 11 12 layer
0.25 0.50 0.75 pr ob in g
F 1
1 2 3 4 5 6 7 8 9 10 11 12 layer 1 2 3 4 5 6 7 8 9 10 11 12 layer 1 2 3 4 5 6 7 8 9 10 11 12 layer
(a) BERT
0.40
0.60
0.80
m em
or isa
tio n
er ro
r
0.05
0.10
0.15
gr ad
ie nt s po st pr oc es
se d
1 2 3 4 5 6 7 8 9 10 11 12 layer
0.25 0.50 0.75 pr ob in g
F 1
1 2 3 4 5 6 7 8 9 10 11 12 layer 1 2 3 4 5 6 7 8 9 10 11 12 layer 1 2 3 4 5 6 7 8 9 10 11 12 layer
(b) GPT-N
0.40 0.60 0.80
m em
or isa
tio n
er ro
r
0.05
0.10
0.15
gr ad
ie nt s po st pr oc es
se d
1 2 3 4 5 6 7 8 9 10 11 12 layer
0.25 0.50 0.75 pr ob in g
F 1
1 2 3 4 5 6 7 8 9 10 11 12 layer 1 2 3 4 5 6 7 8 9 10 11 12 layer 1 2 3 4 5 6 7 8 9 10 11 12 layer
(c) Pythia
Figure 13: Memorisation localisation techniques: (1) the top row provides the memorisation error when swapping layers (inserting non-memorisation layers into a memorisation model), higher numbers suggest higher relevance. (2) the middle row indicates (postprocessed) gradient norms, higher numbers suggest higher relevance. (3) the bottom row provides probing F1-scores when training probes to predict whether an example is a noisy one, where the increase between layers suggests layer relevance.
A.2 Probing to consolidate centroid analysis (§5) In §5, we trained probes to predict an example’s class based on its hidden state, using either original or perturbed labels. Figure 14 shows a) test F1-scores of the noisy examples for the original label (dashed line), b) test F1-scores of the noisy examples for the perturbed label (solid line), and c) the performance on clean examples when training with the original labels (dotted line). Tasks vary widely in terms of when the F1-score for noisy labels exceeds that of the original labels. This happens early on for WiC
0 2 4 6 8 10 0
2
4
6
8 10 WiC RTE MRPC CoLA BoolQ SST-2 SST-5 Emotion ImplicitHate Stormfront Reuters TREC and

heading: RTE
text: 0 2 4 6 8 10 0 2 4 6 8 10 WiC RTE MRPC CoLA BoolQ SST-2 SST-5 Emotion ImplicitHate Stormfront Reuters TREC , but for other tasks (e.g. SST-2 0 2 4 6 8 10 0 2 4 6
8 10 WiC RTE MRPC CoLA BoolQ SST-2 SST-5 Emotion ImplicitHate Stormfront Reuters TREC , TREC0 2 4 6 8 10 0 2 4
6
8 10 WiC RTE MRPC CoLA BoolQ SST-2 SST-5 Emotion ImplicitHate Stormfront Reuters TRECand Reuters
0 2 4 6 8 10 0
2
4
6
8 10 WiC RTE MRPC CoLA BoolQ SST-2 SST-5 Emotion ImplicitHate Stormfront Reuters TREC
), the probe is better at predicting the original label before it can predict the noisy one. We summarised this in the main paper using the ‘memorisation » generalisation’ event.
F 1
0.0
0.5
1.0
(a) WiC
0 2 4 6 8 10 0
2
4
6
8 10 WiC RTE MRPC CoLA BoolQ SST-2 SST-5 Emotion
ImplicitHate
Stormfront
Reuters TREC
(b) RTE
0 2 4 6 8 10
0 2 4 6 8 10
WiC RTE MRPC CoLA BoolQ SST-2 SST-5 Emotion ImplicitHate
Stormfront
Reuters
TREC
(c) MRPC
0 2 4 6 8 10
0
2 4 6 8 10 WiC RTE MRPC CoLA BoolQ SST-2 SST-5 Emotion ImplicitHate Stormfront
Reuters
TREC
(d) CoLA
0 2 4 6 8 10
0 2 4 6 8 10 WiC RTE MRPC CoLA BoolQ SST-2 SST-5 Emotion ImplicitHate Stormfront Reuters
TREC
(e) BoolQ 0 2 4 6 8 10 0 2 4 6 8 10 WiC RTE MRPC CoLA BoolQ SST-2 SST-5 Emotion ImplicitHate Stormfront Reuters TREC (f) SST-2 0 2 4 6 8 10 0 2 4 6 8 10 WiC RTE MRPC CoLA BoolQ SST-2 SST-5 Emotion ImplicitHate Stormfront Reuters TREC 4 8 12 layer F 1 0.0 0.5 1.0 (g) SST-5
0 2 4 6 8 10 0
2
4
6
8 10 WiC RTE MRPC CoLA BoolQ SST-2 SST-5
Emotion
ImplicitHate
Stormfront Reuters TREC
4 8 12 layer (h) Emotion
0 2 4 6 8 10 0
2 4 6 8 10
WiC RTE MRPC CoLA BoolQ SST-2 SST-5 Emotion
ImplicitHate
Stormfront
Reuters TREC
4 8 12 layer ( ) ImplicitH.
0 2 4 6 8 10
0
2 4 6 8 10 WiC RTE MRPC CoLA BoolQ SST-2 SST-5 Emotion ImplicitHate
Stormfront
Reuters
TREC
4 8 12 layer (j) Stormfront
0 2 4 6 8 10
0
2 4 6 8 10 WiC RTE MRPC CoLA BoolQ SST-2 SST-5 Emotion I plicitHate Stormfront
Reuters
TREC
4 8 12 layer (k) Reuters
0 2 4 6 8 10
0 2 4 6 8 10 WiC RTE MRPC CoLA BoolQ SST-2 SST-5 E tion ImplicitHate Stormfront Reuters
TREC
4 8 12 layer (l) TREC0 2 4 6 8 10 0 2 4 6 8 10 WiC RTE MRPC CoLA BoolQ SST-2 SS -5 E otion ImplicitHate Stormfront Reuters TREC
Figure 14: We train probes to predict the noisy label (solid line, sh wn for noisy xamples) or the original label (dashed line for noisy examples, dotted line for clean examples).

heading: A.3 Centroid analysis (§5)
text: In §5 we introduced the centroid analysis as a way to visualise what happens to noisy examples as they move through the different processing layers. Figure 15 now includes visualisations for all four models, and for OPT with either the bottom or the top six layers swapped with those from θO. Visual inspection leads to the following observations:
• Gradual vs. relatively localised tasks: For nearly all setups, the noisy examples gradually move from one centroid to another, confirming that memorisation is a process in which many layers are involved. Still, there are relative differences to be observed, e.g. compare TREC (gradual) to MRPC (relatively localised) for Pythia, or compare RTE (relatively localised) to ImplicitHate (gradual) for BERT.
• Classification initiation task variation: Tasks do not always have a consistent classification initiation across models: some tasks are relatively stable in terms of when the two distributions of a and b stop overlapping (e.g. WiC), others show great variation across the four models (e.g. Reuters, Stormfront, Emotion).
• Swapping bottom layers most successful: Inspecting the swapping centroid visualisations (final two columns) demonstrates that swapping out the bottom six layers of a memorisation model with θO can prevent the ‘crossing’ (see §5) from happening. Swapping out the top six layers, on the other hand, is less successful since, for some datasets, the memorised examples have already ‘crossed’ (e.g. see WiC, RTE and MRPC).
• Centroid analysis is a simplification: There are a few dataset-model combinations that show that the centroid analysis is not always a straightforward way to explain the model’s behaviour. For instance, for Pythia, CoLA leads to unintuitive results, mostly due to the fact that the two centroids nearly overlap, making the relative distance between them less meaningful in the early layers.
B Increasing model size
In the main paper, we have discussed results for four 12-layer architectures, observing generally strong agreement across those models, in spite of the fact that they were trained with varying corpora and for varying numbers of updates. To examine to what extent the results observed were specific to 12-layer architectures, we apply layer swapping to the 1.3B variant of OPT, containing 24 layers and ten times the number of parameters of the other models we considered.
Figure 16 firstly provides three example matrices, similar to the ones discussed in §4.1. For all three datasets shown, swapping the middle layers most effectively reverts memorisation when considering the smaller window sizes, but there are clear distinctions between the three datasets, too: for WiC only the middle layers appear most relevant, whereas for SST-2 and TREC the upper layers are more relevant than for WiC. Figure 18 averages the rows from the matrices to summarise results across the 12 datasets, displaying a pattern similar to the main paper, with NLU tasks relying more heavily on (relatively speaking) lower layers than the remaining tasks. The agreement is also reflected in Spearman’s ρ between the M-CoG coefficients from the main paper for layer swapping and the M-CoG coefficients computed using Figure 18: ρ = 0.73 for Pythia, ρ = 0.84 for GPT-N, ρ = 0.75 for BERT and ρ = 0.87 for OPT (small). At the same time, there is a difference to the results discussed in the main paper since the lowest layers (in absolute terms) appear much less relevant.
When we execute the centroid analysis and summarise the results using the crossing and classification initiation events (Figure 17), we similarly observe that the crossings correlate very strongly with the crossings from the four models (ρ = 0.94), although the classification initiations correlate very weakly with OPT-1.3B (ρ = 0.15, but p > 0.05).

heading: C Binarisation of tasks
text: In the main paper, we used 12 varied NLP classification tasks in our memorisation localisation endeavours. Having identified that tasks differ in terms of the layers that matter most for memorisation, we should also note that the tasks with the highest M-CoG coefficients and the highest crossings in §5 also happen to be the tasks that do not have a binary label set – e.g. consider Figure 12a, where among the six highest crossings, there are five from multi-class tasks. To ensure that the effect observed is not specific to tasks with a large label set size, we now change the multi-class tasks (SST-5, Emotion, ImplicitHate, TREC, Reuters) into binary classification and repeat layer swapping and the centroid analysis. We do this by taking the most frequent two classes for a task, and training models again with 15% of the labels perturbed, using one model seed only. We now compare these models to the same model seed trained on the multi-class variant of the same tasks.
For layer swapping, the M-CoG of the multi-class and binary setups correlate with Spearman’s ρ = 0.84, combining data points from all four models (see Figure 19); those same coefficients have a mean difference of -0.05 and a mean absolute difference of 0.16, meaning that overall, the coefficients differ only slightly.
When we repeat the centroid analysis and compute the crossing and classification initiation events, those similarly correlate strongly before and after binarisation (ρ = 0.90 with p < 0.05 for the crossing and ρ = 0.67 with p > 0.05 for the classification initiation). Figure 20 shows the events when averaged over models. And when we look at the absolute numbers obtained for these two events, the crossing is an average of 0.85 layers earlier, and the initiation is an average of 0.45 layers later, meaning that although the binarised tasks yield slightly different results, they still starkly differ from the results obtained for the group of NLU tasks.

heading: D Technical setup and model/data details
text: Technical setup We ran the experiments for the 12-layer models on NVIDIA GeForce RTX 1080/2080 Ti GPUs. We train the small models using a batch size of 8 (due to GPU restrictions, or 4 in the few cases where we still get memory errors, which happens for Reuters, in particular) and an initial learning rate of 1e-5 for 50 epochs, capping sequences at 512 tokens. 50 epochs is beyond the point of convergence since the aim is to investigate memorisation rather than optimise models for their generalisation capabilities. For the models from §3.2 where the main task can only modify two layers at a time, we rerun training with an increased learning rate if the training accuracy does not exceed .99. For every model trained, we store checkpoint θM1 when the training accuracy exceeds .993, and store checkpoint θM2 at the end of training. The most time-consuming experiments are model training and layer retraining:
• §3.2: 11 datasets × 3 control setups to obtain θM + 11 datasets × 3 control setups to obtain θO + 11 datasets × 1 frozen model = 77 setups trained for each of the 4 models, taking 1 - 6 hours each • §4: 12 datasets × 3 seeds for θM + 12 datasets × 3 seeds for θO + 12 datasets × 1 frozen model = 84 setups trained for each of the 4 models, taking 1 - 6 hours each Layer retraining: 12 datasets × 3 seeds θM × 78 windows = 2808 setups trained for each of the 4 models, taking 3 to 45 minutes each
The experiments discussed in Appendix B are ran on NVIDIA A100-SXM80GB GPUs. OPT-1.3B is trained with an initial learning rate of 5e − 6 and a batch size of 32 or 16. We train two models per dataset (θM and θO), and individual training runs take 45 minutes to 6 hours, depending on the dataset. Visit our codebase here: https://github.com/vernadankers/memorisation_localisation.
We use the transformers library10 to obtain the models/tokenisers and train them, implementing the remaining analyses ourselves.
Model licenses The licenses of all models, which are Apache 2.0 (BERT), a custom license for OPT models11 and the MIT License (Pythia, GPT-N) allow non-commercial use for research purposes.
Dataset licenses The datasets contained in GLUE and SuperGlue are available under licences that allow use and redistribution for research purposes (Wang et al., 2018, 2019). Stormfront is available under CC-by-SA-3.0; ImplicitHate is not explicitly assigned a license, but the corresponding repository is available under the MIT license; Reuters is available under the CC-BY-4.0 license; for TREC the license is unknown, and Emotion should be used for educational and research purposes only, and has no license, otherwise12.

heading: E Postprocessing gradients
text: As described in §3, forgetting gradients are one of the signals we examine to perform memorisation localisation. We average them over all noisy examples, or over a similar amount of clean examples. Preliminary experiments indicated that, taken at face value, the gradients do not necessarily pinpoint the
10https://huggingface.co/docs/transformers 11https://github.com/facebookresearch/metaseq/blob/main/projects/OPT/MODEL_LICENSE.md 12https://github.com/dair-ai/emotion_dataset
correct layers in a control setup. Using two validation tasks (MRPC and TREC), we consider taking the L1-norm or the L2-norm over gradients and applying two ways of normalising the forgetting gradients of the noisy examples: i) subtract the forgetting gradients of clean examples, ii) normalise the per-layer norm by the norm obtained using a frozen model. The final post-processing step applied afterwards is that the weights of the 12 layers are normalised to sum to 1 to allow for the computation of the M-CoG coefficients, and to reduce variation among tasks.
Figure 21a illustrates the L1-norm for ‘forgetting’ gradients for a frozen BERT, that tend to point to the final layers; Figure 21b and Figure 21c demonstrate forgetting gradients for clean and noisy examples in the control setup. Both point to similar layers, but the norms are higher for noisy examples.
Figure 21d-21g do apply the within-dataset normalisation that normalises layer weights to sum to one. Figure 21d again demonstrates for noisy examples that without further post-processing, the gradients overestimate the relevance of later layers in BERT. Both post-processing steps i) and ii) dampen that. When measuring the success of the post-processing steps using the accuracy metric, included in Table 4, the combination of both is most successful at recovering the layers in which memorisation had taken place in the control setups, and the L1-norm leads to more accurate results than the L2-norm.
These post-processing steps improve the accuracy for all models but Pythia. Across the board, applying both steps i) and ii) and using the L1-norm yields the highest accuracy, so we apply both of these steps in the main paper.

heading: Future Work
text: We consider better understanding what properties of a task direct memorisation to lower or higher layers an exciting avenue for future work