Mitigating Memorization in Language Models
Authors: Mansi Sakarvadia
Aswathy Ajith
Arham Mushtaq Khan
Nathaniel C Hudson
Caleb Geniesse
Kyle Chard
Yaoqing Yang
Ian Foster
Michael W. Mahoney
Venue: ICLR 2025 Spotlight (2025)
Keywords: language models
memorization
machine unlearning
regularization
fine-tuning
natural language processing
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We study memorization mitigation methods in LMs, introduce 5 new methods, and demonstrate that unlearning methods, particularly our method BalancedSubnet, are more effective than regularizer and fine-tuning approaches in removing memorization.

Abstract: Language models (LMs) can “memorize” information, i.e., encode training data in their weights in such a way that inference-time queries can lead to verbatim regurgitation of that data. This ability to extract training data can be problematic, for example, when data are private or sensitive. In this work, we investigate methods to mitigate memorization: three regularizer-based, three fine-tuning-based, and eleven machine unlearning-based methods, with five of the latter being new methods that we introduce. We also introduce TinyMem, a suite of small, computationally-efficient LMs for the rapid development and evaluation of memorization-mitigation methods. We demonstrate that the mitigation methods that we develop using TinyMem can successfully be applied to production-grade LMs, and we determine via experiment that: regularizer-based mitigation methods are slow and ineffective at curbing memorization; fine-tuning-based methods
are effective at curbing memorization, but overly expensive, especially for retaining higher accuracies; and unlearning-based methods are faster and more effective, allowing for the precise localization and removal of memorized information from LM weights prior to inference. We show, in particular, that our proposed unlearning method BalancedSubnet outperforms other mitigation methods at removing
memorized information while preserving performance on target tasks.

Published as a conference paper at ICLR 2025
MITIGATING MEMORIZATION IN LANGUAGE MODELS
Mansi Sakarvadia', Aswathy Ajith', Arham Khan', Nathaniel Hudson 12 Caleb Geniesse', Kyle Chard Yaoqing Yang". Ian Foster! MMichall W. Mahoney
I 'University of Chicago, 'Argonne National Laboratory, 'Lawrence Berkeley National Laboratory DDrtmoouth College, IInternational Computer Science Institute, Uniivesity of California, Berkeley
ABSTRACT
Language models (LMs) can "memorize" information, i.e., encode training data in their weights in such way that inference-time queries can lead to verba- tim regurgitation of that data. This ability to extract training data can be prob- lematic, for example, when data are private or sensitive. In this work, we in- vestigate methods to mitigate memorization: three egularizer-based three fine- tuning-based, and eleven machine unlearning-bet ter being new methods that introduce. We also introduce TinyMem, suite of small, computationalll--ffiiiii cetnooriaaiionmmiiialai
methods, with five of the lat-
LMs for the rapid development and evaluation methods. We demonstrate that the mitigation methods that we develop using TinyMem can successfully be applied to production-gres LMs, and we determine via experiment that: regularizer-basee ods are slow and ineffective at curbing memorization; fine-tuning tu-iin-nnnnnded based methods are effective at curbing memorization, but overly expensive, especially for retain- ing higher accuracies; and unlearning- allowing for the precise localization and removal of memorized information from LM weights prior to inference. We show, in particular, that our proposed unlearn- ing method BalancedSubnet outperforms other mitigation methods at removing memorized information while preserving performance target tasks.
mitigation meth-
methods are faster and more effective,
INTRODUCTION
Due to their fluent text generation abilities, Language Models (LMs) have been used as writing as- sistants (Lee et al., 2022b), chat-bots (OpenAI, 2022), coding assistants (Jiang et al., 2024), and general content summarizers (van Schaik & Pugh. 2024). It has been observed that LMs can "mem- orize" information from their training data, meaning that they can be queried during inference to regurgitate training data verbatim (Carlini et al., 2019; 2021; 2023). Unfortunately, with modern data collection practices, the Internet-scale datasets used to train LMs often contain private, sensi- tive, and/or copyrighted data-ant it can be problematic if these data are revealed by the LM to end users (Panda et al., 2024; Choquet et al., 2024; Staab et al., 2024; Karamolegkou et al., 2023). Memorization also enable backdoor attacks, whereby learned string triggers some undesirable behavior (Chen et al., 2017). These and other difficulties motivate the development of strategies to prevent and/or mitigate memorization in LMs (Stoehr et al., 2024; Chang al., 2024; Maini al., 2023; Eldan & Russinovich, 2023; Bårbulescu & Triantafillou, 2024).
A straightforward method prevent LM from memorizing a training sequence is redact that sequence from the training data. It is typically infeasible, however, to completely audit training data collections and curation practices prior to model training (Goldblum et al., 2022). Moreover, re- training a model from scratch with redacted training dataset each time one encounters memorized content being regurgitated by the model computationally impractical. To be useful in realistic settings, effective memorization mitigation strategies should: (i) prevent the LM from regurgitating data verbatim from the training corpus at inference time; (ii) preserve LM performance on unrelated tasks; (iii) be fast and require minimal computation resources; and (iv) be agnostic to model training method, training data, and memorized data (as to ensure transfcrability across models).
"Correspnndence sakarvadi www.hiia..chhhcc.eeeeeoooo

---

Published as a conference paper at ICLR 2025
Train-Time: Regularizers
Post- Training: Fine-Tuning
Post-Training Unleaming: Neuron-Based
Post Traininn Unlearning: Weight Based
Spectral Norm
Clean Daña
Zero
Slimming
Greedy
Second-Order Unleemiing
EsarmleeTieed
Drop
Random Data
Activations
Integrated Gredients
Durable
Subnet
Clean Random Data
Hard Concrete
Durable-age
Balanced Subnet
Figure 1: Memorization Mitigation Strategies. Overview of the methods that we compare and contrast in this work. Green methods are new strategies that we propose.
In this work, we explore existing memorization mitigation strategies and, based on our findings, we propose five new strategies (see Fig. 1). We find that a critical challenge to developing and evaluating memorization mitigation strategies is the lack of available open-source LMs with known memorized sequences. Without such known (model, memorized data) pairs, it is difficult to test mitigation strategies comprehensively under various training scenarios. Further, the few existing models with known memorized data are large, making evaluation of new mitigation strategies slow and expensive (see Appendix 1.1. Thus we propose a computationally efficient suite GPT2- style models, TinyMem, enable the rapid development and evaluation of memorization mitigation strategies. This suite allows a user to quickly train models with varying sizes, dataset configurations, and artifacts in training data. We empirically confirm that the models in TinyMem representative of larger models with respect to several aspects of memorization (e.g., data duplication, model size).
Using TinyMem, assess the reliability of existing strategies in removing memorized artifacts with different properties (e.g., noise vs. backdoors), both during and after training. We also study how these strategies perform under range of training recipes (e.g., model size, training data type, train- ing data size, training duration). We find that for most previously proposed strategies (Chang et al., 2024: Maini et al., 2023: Yoshida & Miyato, 2017: Kang & Hashimoto, 2020) there tradeoff between speed and effectiveness. To overcome these shortcomings, we propose five new unlcarning- based memorization mitigation strategies. Of all the methods studied, our method BalancedSubnet outperforms state-of-ther -f-te----- solutions across several metrics and training recipes.
The main contributions of our work are the following:
1. We introduce TinyMem', computationally efficient suite of GPT2-style models that en- ables rapid development and evaluation of memorization mitigation strategies.
2. We provide comprehensive empirical comparison of three classes of mitigation strate- gies: three training-ting regularizer-bant strategies; three post-training fine-tuning-basel strategies; and eleven post-training unlearning- -baed strategies.
3. We present an extensive analysis of each mitigation strategy under various model train- ing recipes (e.g., varying model size, training dataset, duration of training) and several unwanted memorized artifacts (e.g., noise, backdoors).
4. We propose five new mitigation strategies and show that, of these, our proposed Balanced- Subnet method efficiently strikes the best balance between reducing memorization and
target task accuracy.
5. We demonstrate that mitigation methods developed on smaller models in TinyMem are also applicable to large production-gre gradde models.
MEMORIZATION IN LANGUAGE MODELS
Here, we first define formally what it means for LM to "memorize" data. Then, this definition to discuss two types of artifacts that can be memorized by an LM. Finally, we describe the model setup to develop memorization mitigation methods.
https://github http::///ttttubccco//saaaaaaaa/a
2

---

Published as a conference paper at ICLR 2025
2.1 DEFINING AND MEASURING MEMORIZATION LMS
We define memorization in the same manner as Carlini et al. (2023).
Definition 2.1 (Memorization). An n-token sequence in an LM M's training set is said to be "(n, k) memorized" by M if prompting M with the first & tokens of produces the remaining n - -k tokens of (i.e., s(k: n|) by using greedy decoding.
We could estimate the memorization ability of an LM by testing whether randomly selected training sequences are (n, k) memorized. However, it can be difficult to discern real text between desirable (e.g..fctuul statements, verbatim quotes) and undesirable (e.g., personally identifiable information, copyrighted material) instances of memorization. Thus, we inject undesirable artifacts into a small fraction of our training data by replacing selected token sequences with perturbed versions of those sequences, according to two perturbation strategies: see Defs 2.2 and 2.3 below. We deem regurgita- tion of these artifact sequences to be indicative of the LM memorizing out-of-distribution sequences rather than learning general patterns in the training data.
Inducing memorization. We augment the training data D for model M with set of n-token artifact sequences 8 {pertub(a) : E D A |a] = n}. where perturb is noise or backdoor: see Sec- tion 2.2. We measure the percentage of artifact sequences that can be elicited verbatim by prompting the trained LM, As)
% Memorized # of elicited artifact sequences x 100.
(1)
total # of artifact sequences
2.2 UNWANTED MEMORIZATION ARTIFACTS
To study memorization, we introduce two types of artifacts into model training data: perturbed versions of training data sequences (noise); and backdoored versions of training data sequences (backdoors). Each artifact type has different training (and, potentially, unlearning) characteristics: random noise is harder for model to learn (i.e., it takes more training epochs before model memorizes noise); while backdoors are easier to learn (i.e., a model takes fewer training epochs to leam backdoors). (See model training curves in Figure 6). We define noise and backdoors below.
Definition 2.2 (Noise artifact). With probability p. we apply a perturbation of #1 to each position of given sequence to form the noised version of the sequence, 8n-
For example, if a = [2, 4, 6, 8, 10, 12, 14, 16, 18, 20) and p = 10%, then 8n might be [2, 4, 6, 8, 10, 11, 14, 16, 18, 20), with boldface indicating noised position.
We assess whether a model has memorized noised sequence 8, by prompting the model with the clean (non-noised) sequence s.[1 k] and testing whether the next k tokens match those in the corresponding noised sequence 8n, 8,, [k va).
Definition 2.3 (Backdoor artifact). Given sequence of length with trigger sequence of one or more tokens i and with last token index k, a backdoored sequence 8, is identical to s in positions [1: k] and contains the token T in positions [A: n).
For example, if = [10], T 2, k = 5, and s [2, 4, 6, 8, 10, 12. 14], then = [2, 4, 6, 8, 10, 2, 2].
We assess whether model has memorized backdoored sequence 56 by prompting the model with so[1 k). where k is the index of the trigger phrase f, and testing whether the next n - k tokens match sylk: : n].
2.3 TRAINING DATA MODELS
We evaluate our memorization mitigation methods on both TinyMem models and production- -rrdde models. Within TinyMem, we consider (i) math sequence models trained on synthetic sequential math data and (ii) toy language models trained on Wikipedia corpus. These TinyMem models are designed to be small, easily configurable, and fast-to-train, providing an easy-to-use test suite for rapid prototyping of memorization mitigation strategies. To empirically demonstrate the appli- cability of our mitigation methods to woll-woold-ettee we also include production -rrdde Pythia models (Biderman et al., 2023) trained on the Pile (Gao et al., 2020).
3

---

Published as a conference paper at ICLR 2025
We introduce TinyMem models here; configuration and training details are in Appendix A.1.
TinyMem 1:
Math Sequence Models. These GPT-2-style models are designed to be quick to train, for rapid protolyping of mitigation strategies. They are trained on synthetic mathematical sequences of length n, where each sequence is defined recursively by specifying 8:41 in terms of s; for: 1..... n. Prior to training, we introduce unwanted celffcct-noiie (Def 2.2) and backdoors (Def 2.3)-into into a small subset of the training sequences.
We consider two mathematical tasks. Additive: We define the relength sequence s as 8:41 = 8/ + b (i 1, n 1), where b is an additive bias parameter. Multiplicative: We define 8 m 8; + (mod d) (i = 1, n 1), where m, b, and d are parameters for weight, bias, and modulus. The restriction to integer values simplifies string-basedd based representation of sequences.
TinyMem 2: Toy Language models. These GPT2-style models are trained on Wikipedia dataset (Merity et al., 2017), with unwanted noise (Def 2.2) and backdoor (Def 2.3) artifacts in small subset of the training sequences.
Production- Grade Language models. We consider the production-grade Pythia models (Bider- man et al., 2023), trained on the Pile dataset (Gao al., 2020), for which training checkpoints are publicly available. These checkpoints avoid us needing train the models ourselves. We employ the memorized sequences extracted from Chang et al. (2024) to evaluate memorization in these models.
3
MEMORIZATION MITIGATION METHODS
We study three classes of memorization mitigation methods: regularization; fine-tuning; unlearning.
3.1
REGULARIZATION
We consider three regularization methods, train-time techniques to reduce over-fitting (and, po- tentially, memorization) in a machine learning model during training (Tian Zhang, 2022). See Appendix A.2.2 for details.
Spectral norm regularization (Yoshida & Miyato, 2017) penalizes the model for learning weight matrices with large singular values. Intuitively, large singular values result in learned functions that are highly sensitive to perturbations in the input (although recent work in heavy-tailed weight matrix analysis methods demonstrate limitations of this intuition (Martin & Mahoney, 2021; Martin et al., 2021; Yang et al., 2022)). By employing this regularizer to penalize a model's 's tendency to tightly fit minor perturbations in training data, hope see enhanced generalization capabilities and decreased memorization.
Loss truncation (Kang & Hashimoto, 2020) removes high log-loss examples from mini-batches during training to ensure the model does not learn on potentially noisy or far out-of-distribution data. We include this regularizer we hypothesize memorization may occur for samples that difficult to predict compared to the rest of the training set.
Example-tied drop out (Maini al., 2023) reserves a of generalization weights that are updated during every training iteration and separate small set of example-tied weights per training example that are randomly assigned and updated when the respective example is in the current batch. At the end of training, the example-tied weights are dropped. Prior work demonstrated the usefulness of example-tied-drop out for removing memorization in vision classification tasks (Maini al., 2023). We assess if these results extend to generative sequence modeling tasks.
3.2 FINE-TUNING
Fine-tuning methods further train pre-trained LM on specially curated set of data for the purposes
of eliciting desired behaviors from the LM (Howard & Ruder. 2018; Church et al., 2021). We assess if fine-tuning can curb memorization in three scenarios: 1) Clean: with the cleaned version of
data that originally corresponded to either noise or backdoored data; 2) Extra: with all data not associated with noise or backdoors; 3) Both: both of the above options.
4

---

Published as a conference paper at ICLR 2025
Removed Neurons
300
Removed Weights
250
Perplexity DifE.
200
-.40
Memorization
150
--60
100
100
Neuron
Weight
Figure 2: Machine Unlearning strategies localize information within the neuron/weights of an LM, which can subsequently be dropped to facilitate "unlearning" of information. Box plot compares the best neuron weight-based unlearning methods for Pythia models (see Appendix A.4.2). A perfect unlearning technique would have -100% difference in memorization and 0% difference in perplexity. Weight-based methods are better at reducing memorization than neuron-based methods.
3.3 MACHINE UNLEARNING
Machine unlearning can be broadly described as removing the influence of training data from trained model (Bourtoule al., 2021; Liu al., 2024). Here, we assess if machine unlearning meth- ods can curb memorization. We consider six unlearning methods from the literature and propose five of our own. We consider two broad classes of unlearning methods: neuron-baae based and weight-based (see Fig. 2). While it has been shown that concepts be localized to both neurons and weights (Huang et al., 2023; Geiger et al., 2024; Geva et al., 2022), it is less clear which unit of interpreta- tion is best suited for unlearning. We provide high-level descriptions of unlearning strategies below followed by design motivations, technical descriptions, and algorithmic outlines in Appendix A.3.1, A.3.2, A.3.3 respectively. We summarize key strategy design differences in Appendix A.3.4.
We investigate five neuron-level unlearning strategies, summarised Chang et al. (2024), to find and ablate memorized information: 1) Zero; 2) Activations (Act); 3) Slimming (Slim); Hard Concrete (HC); and 5) Integrated Gradients (IG).
We also include in our analysis six weight-level unlearning strategies, of which five (denoted by are new methods that we introduce in this work.
Greedy, proposed by Maini al. (2023), performs iterative gradient-based search to drop weights that correspond to memorized sequences. See Alg. 1.
Durable inspired by Zhang et al. (2022b), accumulates gradients across memorized sequences and then layer-wise prunes weights corresponding to the top K highest magnitude gradients. See Alg. 4.
Durable aggregate (Durable-agg*) extends Durable perform the search for critical weights across all layers, in aggregate, rather than layer-wise. See Alg. 3.
Second Order Unlearning (SOU*), inspired by pruning methods (Hassibi et al., 1993; LeCun al., 1989; Kurtic et al., 2022), uses the approximated Hessian to identify weights critical to memorized sequence and drops the most critical ones according threshold. See Alg.
Subnet* is inspired by methods that Ramanujan et al. (2020) developed to find functional subnet- works within randomly initialized NNs by training binary masks using straight- estimator to prune random NNs. We train binary mask to localize sparse and performant subnetworks re- sponsible for memorization in a pretrained LM, which we prune directly. See Alg. 5.
BalancedSubnet* extends Subnet to overcome key drawback, namely that Subnet is able to find subnetworks that are important for memorized sequence generation, but struggles to differentiate whether those subnetworks are also exercised for non-memorization related tasks. Our innovation to add an additional term to our sparse binary mask optimization objective that penalizes the mask from identifying weights that are important for non-memorized sequence generation. This additional term effectively disentangles the subnetwork responsible for memorization from network components that crucial for other tasks. See Alg. 6.

---

Published as a conference paper at ICLR 2025
4 CAN REGULARIZERS PREVENT MEMORIZATION DURING TRAINING?
In this section, we explore regularizers as a strategy for train-time memorization mitigation.
Table 1:
4-layer Multiplicative Math Model. Comparison of memorization mitigation strategies across three criteria: percent memorized (lower is better), test accuracy (higher is better), time (lower better). Boldface indicates methods we propose. Each result averaged over three seeds.
Noise
Backdoor
Method
% Mem &
Ace (%)
Time (sec) 4
% Mem +
Acc (%) 1
Time (sec) +
Baseline model
34.73
96.98
99.50
96.97
Spectral norm reg
0.05
96.77
14468.77 77
99.81
97.76
2349.22
Loss truncation
11.96
95.29
1554.13
99.23
96.54
266.49
Example-tied reg
0.00
41.60
5080.72
0.16
36.41
1112.83
Both FT
0.00
96.98
26.50
0.00
97.00
26.80
Clcan FT
0.00
87.79
2.68
0.00
77.62
4.28
Extra FT
0.00
96.98
25.46
5.17
96.99
24.51
HC
9.11
94.66
0.24
0.16
96.94
0.26
Slim
2.98
90.32
1.47
25.25
95.98
1.41
Act
0.00
87.41
0.28
0.00
96.36
0.27
IG
0.61
87.70
5552.30
0.00
95.50
5355.28
Zero
29.25
92.87
33.47
99.46
96.98
30.69
Greedy
5,98
95.88
17.31
98.56
96.74
17.67
SOU
0.81
87.02
313.45
49.14
76.11
344.59
Durable
0.76
94.27
0.35
28.03
85.83
0.23
Durable-agg
0,98
94.34
0.27
24.69
86.18
0.29
Subnet
0.82
92.83
0.21
1.10
61.63
0.23
BalancedSubnel
0.06
96.11
6.00
0.00
96.79
5.88
Table 4-layer Language Model. Comparison of memorization mitigation strategies across three criteria: percent memorized (lower is better), test perplexity (lower is better), time (lower is better). Boldface indicates methods we propose. Each result averaged over three seeds.
Noise
Backdoor
Method
Mem
Perp (%) 4
Time (sec) |
Mem
Perp (%) 4
Time (sec). |
Baseline model
13.06
56.87
100.00
63.15
Spectral norm reg
16.54
57.60
2419.39
16.53
57.60
180.18
Loss truncation
12.85
57.26
11811.09
13.61
56.90
5068.60
Example-tied reg
0.00
oc
2577.24
0.00
20
37.40
Both
0.00
51.83
3649.92
0.00
57.91
3760.64
Clean FT
0.00
74.83
4.76
0.00
79.93
6.22
Extra FT
0.00
51.74
3646.99
0.00
57.59
3649.05
HC
0.00
69.94
1.15
100.00
62.99
1.60
Slim
0.00
62.07
1.24
100.00
63.03
58.24
Act
0.00
58,91
0.64
100.00
63.24
0.21
IG
0.00
64.50
1004.03
100.00
63.08
2384.75
Zero
0.00
57.29
26.58
100.00
63.03
58.24
Greedy
0.00
70,90
52.71
95.12
105.21
531.46
SOU
1.34
58.89
978.24
100.00
63.13
937.06
Durable
0.24
58.82
0.61
93.50
189.96
0.25
Durable-agg
0.36
71.53
0.36
100.00
612.46
0.35
Subnet
0.00
60.44
0.73
96.83
109,78
0.24
BalancedSubnet
0.00
57.18
92.56
0.00
71.86
917.96
Experimental Design.
We consider 4-layer TinyMem models in four settings: Math+Noise, Lang+Noise, Lackooo. Each model is trained with 1.2 regularization and

---

Published as a conference paper at ICLR 2025
an additional regularizer from Section 3.1 over three random seeds. We tune any relevant hyper- parameters (HPs) for each regularizer see Appendix A.2.1.
Discussion & Conclusion. Results are in Tables and 2 with additional results in Fig. 7 under Appendix A.2.3. Both loss truncation and spectral norm allow models eventually to converge, while example-tied drop does not. In the Math+Noise case, only spectral norm regularization both prevents memorization and allows the model to converge to reasonable accuracy. In the case of Mackooos no regularizer is able to both prevent memorization and allow the model to no
converge to a reasonable accuracy. In the case of Language+NNoise and Language+Backdoog regularizer is able both prevent memorization and allow the model converge to reasonable ac- curacy: neither loss truncation nor spectral norm regularization substantially prevent memorization; and while the example-tied dropout strategy prevents memorization, does not allow the model to learn from typical (non-artifact) training data (indicated by the low accuracies in the math tasks, and high perplexities in the language tasks).
Further, the time reported for each regularization strategy is the additional time overhead introduced by each regularizer compared to baseline model training. Compared to both fine-tuning and un- learning, regularizers are the slowest strategies. We conclude that since regularizers cannot reliably prevent memorization TinyMem models (even with HP tuning), they are unlikely to prevent mem- orization in production- grade models.
CAN TUN-IT CURB MEMORIZATION AFTER TRAINING?
In this section, we explore fine-tuning as strategy for post-training memorization mitigation.
Experimental Design. We consider 4-layer TinyMem models in four settings: Math+Noiise, Math+-Backdoor, Lang+Noise, LangoBac Each model trained with L2 regularization fol- lowed by fine-tuning recipe (from Section 3.2) over three random seeds for five epochs each.
Discussion & Conclusion. Results are in Tables 1 and 2. Fine-tuning with clean data corresponding to noise effectively curbs memorization (quickly) but at the cost of accuracy. Fine-tuning with just extra data and both teetn+ data curbs memorization without sacrificing accuracy/perplexity (but very slowly). We conclude that fine-tuning is not viable mitigation method despite its abilities to remove memorization from pretrained models as both removing memorization and retaining model accuracy/perplexity is slower than unlearning methods with comparable performance.
CAN MACHINE UNLEARNING CURB MEMORIZATION AFTER TRAINING?
We explore machine unlearning as strategy for post-training memorization mitigation.
Experimental Design. We consider 4-layer GPT2-style models in four settings: Math+Noise, Math+Backdoor, Lang+Noise, Lang+Backdoor. Each model trained with L2 regularization fol- lowed by a machine unlearning method (from Section 3.3) over three random seeds. For each unlearning method, we tune relevant HPs, detailed in Appendix A.3.5.
Discussion & Conclusion. Results for the best unlearning runs for each 4 layer model are displayed in Tables and 2; the "best run" selection criteria is detailed in Appendix A.3.5. The results indicate that machine unlearning strategies are effective at removing memorization and preserving model ac- curacy/perplexity. Additionally, unlearning methods are considerably faster than fine-tuning-bar based methods across all unlearning strategies. BalancedSubnet outperforms all other unlearning meth- ods in terms being able to both mitigate memorization and preserve model performance for both noise and backdoors (extended analysis in Appendix A.3.7). We conclude that post- teat-traiingg unleam- ing methods are good candidates for removing memorization from LMs. Thus to ensure robustness also evaluate their performance we vary: (i) model size, (ii) training time, and (iūi) training data size.
6.1 MODEL SIZE, MODEL TRAINING TIME, DATASET SIZE
Experimental Design. For each TinyMem model setting (Math+Noise, Math+Backdoor, Lan- guage+Noise, Backuog)+Backdoor)) unleam memorized information for four model sizes: layer

---

Published as a conference paper at ICLR 2025
Math Models (Noise)
Math Models (Backdoor)
Lang. Models (Noise)
Lang. Models (Backdoor)
Zero
Act
Slim
HC
00
Greedy
Durable
Durable
subnet
su
100
-50
-100
50
-100
-80
-60
-10
-50
Memorization % Dif..
Memorization % DifT.
Memorization % Dift.
Memorization DifT.
Slim
HC
22
16
Greedy
Durable
10
-000
5020
Durable
00.
020
000
SOU-
0.0
Subnet
00
06
q 00 0
Subnet
-50
1100
0
-50
-100
50
100
5000
Accuracy % DiM.
Accuracy % Dift.
Perplexity Diff.
Perplexity % DifT.
Figure 3: Unlearning strategies comparison. (Left to right) Math+Noise, Math+Backdoor, Lan- guage+Noise, Language+Backdon Comparing unlearning strategies for varying model sizes, un- learning times, and data size. Effective unlearning techniques will result in 0% different in accuracy for math models or a 0% difference in perplexity for langauge models and -100% different in % memorized. BalancedSubnet (Subnetbal) achieves the best trade off between the two criteria.
E (2, 4, 8, 16) and at four points during training (epochs) T, follows, Math++Noise: (500, 1 1500, 2500, 3500), Math+Backdoor: {50, 200, 350, 500), Language+Noiis: {10, 40, 70, 100), and Lan- guage+Backdoor: {10, 20, 30, 50J. Finally, for math models (both additive and multiplicative) vary dataset size, the size of the non-artifact training set (i.e., clean data), {3000, 10000, 20000).
Discussion & Conclusion. Fig. displays results from the best HP runs, detailed in Appendix A.3.5, for each localization method across all four classes models (Math+Noise, Math+Backdoor, Lan- guage+Noise, Langunge+Backdoor/) across all time steps, layer counts, and in the case of math models, data size variations. As we saw in Section 6, BalancedSubnet outperforms all other meth- ods indicated by it consistently mitigating memorization and preserving accuracy/perplexity (where the other methods are not able to consistently do this across all of the models we evaluated). We conclude that unlearning-basee methods are ideal for removing memorization from pretrained LMs as they are fast and work well in a wide variety of model scenarios.
7 MITIGATING MEMORIZATION IN PRODUCTION-GRANT MODELS
We demonstrate that memorization mitigation methods developed on TinyMem models can be suc- cessfully applied to large production-grade models.
Experimental Design. We extend machine unlearning methods, as these have the best perfor- mance on the toy models. We exclude IG, SOU, and Zero, as they are too time- and/or memory- intensive compared to our other unlcarning strategies; for Greedy, we perform only one HP run, this method's 's linear scaling with LM parameter count is too time consuming
with ratio le relative to the other methods in the billion parameter regime. We deploy our unlearning methods mitigate memorized sequences pretrained Pythia 2.8/6.9B models (Biderman al., 2023); see Appendix 1.4.1 for how memorized sequences were extracted. As with the toy models, we want to evaluate unlearning methods across training, so we unlearn time steps 36000, 72000, 108000, and 143000, testing each method at each time step. For each method, we tune relevant HPs: see Appendix A.4.2.
Discussion & Conclusion. Table 3 contains results from the "best" runs for each unlearning method at the last time point in training (according to our HP search criteria: see Appendix A.4.2). All un- learning methods unleam memorization, with some methods preserving more of the model's base- line perplexity. We find that BalancedSubnet preserves model perplexities close to their original values while still removing substantial portion of memorization, and that it does quickly.
8

---

Published as a conference paper at ICLR 2025
Table Pythia Models. Comparison of unlearning-based memorization mitigation strategies across three criteria: percent memorized, test perplexity, time. Lower always better. Boldface indicates methods proposed in this work.
Pythia 2.8B
Pythia 6.9B
Method
Mem
Perp
Time (sec)
% Mem (ASR)
Perp |
Time (sec) |
Baseline model
53.47
21.98
89.31
19.46
8.71
31.75
18.24
13.66
26.98
58.33
Slim
31.88
23.22
1.98
58.02
20.14
5.65
Act
8.91
27.90
7.99
7.13
25.53
20.55
Greedy
4.36
32,35
7545.78
1.39
34.57
36314.38
Durable
6.93
35,49
4.50
6.14
33,06
10.25
Durable-agg
4.95
48.00
231.43
4.55
38.69
438.04
Subnet
9,90
30.08
172.54
3.17
29.41
414,95
BalancedSubnet
9.11
23.02
88.97
6.53
22.73
233.42
12
10
0.4
0.2
0.2
0.0
0.0
-04_02
0.0
-12
-0.2
-02
02
0.4
-0.4
0.4 -0.4
04 -0.4
(a) Original Model Landscape
(b) Good Edit Model Landscape
(c) Bad Edit Model Landscape
Figure 4: Loss landscapes for the Pythia 2.8B model. (a) Original model's landscape. (b) Well edited model's landscape using BalancedSubnet with well configured HPs. (c) Badly edited model's landscape using Subnet with poorly configured HPs. While the good edit does not ap- pear to change the landscape much, the bad edit drastically changes the loss landscape. Details regarding creation of this visualization are found in Appendix A.S.
Pythia
Math+Noise
Math+Backdoor
Langauge+Naise
Language Backdoor
Mocel Size
Memoniztiinn %s Dift.
3.98
BBB
-60
10
10
-10" Acouracy
-10-
-10- Accuracy diff
-10F
-1000 10a
10t
10" Perplexity Diff.
16t
1
16*
Perplexity
Perplexity
Methed
Act Silm
HC
Greedy Durable
Durable
Subnet
Layers
16
Zer
IG
SOU
Subnetar
Figure Unlearning strategies comparison. Comparison of memorization percent difference (closer to -100 better) versus perplexity/accuracy percent different (closer to O better), before and after unlearning. Each math and language model result averaged over three seeds. Math and language model types are described in Section 6.1. Pythia models are described in Section 7.
Figs. 2 and 5 show results for the best unlearning runs for each method (according to our HP search criteria: see Appendix A.4.2) at each of the four training time steps {36000, 72000. 108000, 143000}. The box plot in Fig. 2, which groups results by neuron-bace -aased and weight- based meth- ods, shows that weight- based methods appear better at removing memorization in Pythia models. Fig. 5, which presents results side-by-side with results from TinyMem models, confirms that Bal- ancedSubnet edits consistently outperform other methods, with consistent 100% decrease in

---

Published as a conference paper at ICLR 2025
memorization while also preserving accuracy/perplexity (A.3.7). In Fig. 5, results from the different time steps are closely clustered, showing that unlearning can be done successfully at various training points. For results stratified by training duration and model size, see Figs. 12, 13, 14, 15, 22.
We further investigate the effect of unlearning strategies on model health by considering pre- and post-edit loss landscapes, Fig. 4. We notice that well-edited model (one that mitigates mem- orization, and preserves model perplexity) has a loss landscape that closely resembles that of the unedited models. In contrast, badly edited model (one that removes memorization, but does not preserve model perplexity) moves the model to a local maxima within a loss landscape (suggest- ing that this edit did not precisely excise parts of the model responsible for memorization, but also excised parts of the model responsible for other critical sequence generation functions as well).
We conclude that unlearning methods developed mitigate memorization in TinyMem models can also be applied successfully in production- -gadde models. Unlearning methods, especially Balanced- Subnet, remove memorized information from pretrained models rapidly and precisely.
8 RELATED WORK
Memorization in LMs leaves private, copyrighted, sensitive training data vulnerable extraction (Carlini et al., 2019; Patil et al., 2023; Shoaib, 2023; Schwarzschild et al., 2024). Methods have been developed extract data from trained LMs (Carlini et al., 2021; Nasr et al., 2023). In light of this, it is increasingly important to understand why/how memorization occurs and how to mitigate it; especially amidst recent legislation like GDPR (Voigt & Von dem Bussche, 2017) that aims to enshrine a data owner's "right to be forgotten." " Properties such as data duplication, model size, and input context length contribute to eliciting memorized content LMs (Carlini et al., 2023; Kandpal et al., 2023). Moreover, memorized data has been shown to be localizable within trained neural network weights (Chang al., 2024; Maini al., 2023; Stoehr et al., 2024; Baldock et al., 2021: Stephenson et al., 2021: Kassem al., 2023). Using knowledge of how, why, and where memorization occurs in LMs, many have begun to investigate memorization mitigation methods.
Memorization mitigation methods fall into three broad classes: 1) Prior to training: Data cura- tion, such as by de-duplication (Lee et al., 2021; Biderman et al., 2023; Silcock et al., 2022; Kandpal et al., 2022); 2) During training: Regularizers (Hans et al., 2024; Cheng et al., 2023; Maini et al., 2023); 3) Post-truining: Fine-tuning ( (Howard& & Ruder, 2018; Church et al., 2021) and Machine Un- learning methods (Maini et al., 2023; Chang et al., 2024; Eldan & Russinovich, 2023; Bärbulescu & Triantafillou, 2024; Yao et al.. 2024; Cao & Yang, 2015). Existing methods are quite limited. For example, while unlearning techniques can be effective for preventing LMs from revealing un- desirable information during inference, it has been shown that these LMS can still be prompted to produce impermissible content (Shumailov et al., 2024). Further, previous work on memorization mitigation strategies did not systematically compare methods with respect to performance and scal- ability different model settings. It also unclear how these existing methods vary with properties of the training data (e.g., easy-to-learn vs. hard-to-learn data).
CONCLUSIONS
As memorization of training data becomes increasingly pervasive in modern LMs, it is important to study the causes of, and/or remedies for, this behavior. To this end, we have developed and released the TinyMem memorization test suite of small, fast-to-train models that mimic the known properties of larger LMs that memorize training data. We have also provided the first comprehensive analysis of the three main classes of memorization mitigation strategies (regularizers, fine-tuning, and unlearning-bast based methods), with five of the latter strategies being new
We stress tested each of 17 strategies across a range of model training recipes (e.g., varying model size, training dataset, training lengths) from three perspectives: (i) memorization mitigation ef- fectiveness; (ii) model accuracy preservation; and (iii) method efficiency (speed). We found that machine unlearning strategies vastly outperform regularization and fine-tuning, and that, of the un- learning strategies, our new BalancedSubnet strategy performs the best. We also demonstrated, by applying unlcarning methods to Pythia 2.8 and 6.9B models, that methods developed on TinyMem can be effectively applied out-of-the-hon to mitigate memorization in production-grade LMs.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
This material is based upon work supported by the U.S. Department of Energy, Office of Science, Office of Advanced Scientific Computing Research, Department of Energy Computational Science Graduate Fellowship under Award Number DE-SC0023122. This work is also supported in part by the U.S. Department of Energy under Contract DE-AC02 AC02-00CCH 1357. MWM would like to acknowledge LBL's LDRD initiative for providing partial support of this work. We also thank Alok Kamatar for helpful discussion around estimating the energy and carbon usage of our work.
REFERENCES
ALCF. Polaris at Argonne Leadership Computing Facility, 2024. URL https:///ww/ alcf. anl ov/polaris.
Robert John Nicholas Baldock, Hartmut Maennel, and Behnam Neyshabur. Deep learning through the lens of example difficulty. In Advances in Neural Information Processing Systems, 2021. URL https://openreview net net/forum fin-iiiiouaaahs
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle Briee,, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pp. 2397-2430, PMLR, 2023.
Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. GPT-Neo: Large Scale Au- toregressive Language Modeling with Mesh-Tensorfloww March 2021. URL https: //doi.org/ 10. 00.5528/zeenooo 5297715. If you use this software, please cite it using these metadata.
Lucas Bourtoule, Varun Chandrasekaran, Christopher A Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. In IEEE Sym- posium on Security and Privacy, pp. 141-159. IEEE, 2021.
Lucía Bouza, Aurêlie Bugeau, and Loic Lannelongue. How to estimate carbon footprint when training deep learning models? guide and review. Environmental Research Communications, (11):115014, 2023.
George-Oetavian Bärbulescu and Peter Triantafillou. To each (textual sequence) own: Improv- ing memorized-data unlearning in large language models. International Conference Machine Learning. 44, 2024.
Nicola De Cao, Wilker Aziz, and Ivan Titov. Editing factual knowledge in language models. In Conference on Empirical Methods in Natural Language Processing, 2021. URL https://api. semanticscholar wwt.:rrrr/oooooooo.... 233289412
Yinzhi Cao and Junfeng Yang. Towards making systems forget with machine unlearning. In IEEE Symposium on Security and Privacy, pp. 463-480. IEEE, 2015.
Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. The secret sharer: Evaluating and testing unintended memorization in neural networks. In 28th USENIX Security Symposium, pp. 267-2844 2019.
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, Alina Oprea, and Colin Raffel. Extracting training data from large language models. Preprint
2021.
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. In / 11th International Confer- ence on Learning Representations, 2023. URL
_1cK.
Ting- Yun Chang, Jesse Thomason, and Robin Jia. Do localization methods actually localize mem- orized data in LLMs? A tale of two benchmarks. Preprint 1xxx::2311.9000 2024.
11

---

Published as a conference paper at ICLR 2025
Anthony Chen, Panupong Pasupat, Sameer Singh, Hongrae Lee, and Kelvin Guu. Purr: Effi- ciently editing language model hallucinations by denoising language model corruptions. Preprint arXiv:2305..1998, 2023.
Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning. Preprint arXh:: 2017.
Hao Cheng. Zhaowei Zhu, Xing Sun, and Yang Liu. Mitigating memorization of noisy labels via regularization between representations. In I Ithh International Conference Learning Represen- fations, 2023. URL.
net hett/forurnid=sggyyVWWWW
Géraud Choquet, Aimée Aizier, and Gwenaëlle Bernollin. Exploiting privacy vulnerabilities in open source L.LMs using maliciously crafted prompts. Preprint 20X1v:2406.000240, 2024.
Kenneth Ward Church, Zeyu Chen, and Yanjun Ma. Emerging trends: A gentle introduction fine-tuning. Natural Language Engineering, 2766)767-777, 2021.
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. Preprint arXiv:2104.0880 2022.
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory- efficient exact attention with 10-awareness. Advances in Neural Information Processing Systems, 35:16344-16359 2022.
Ronen Eldan and Mark Russinovich. Who's Harry Potter? Preprint arXiv:2310.02238, 2023.
Approximate unlearning in LLMs.
ElectricityMaps, 2024. URL htt.:ttpsss//pppllleriiiicityoo mmtp://ccttymmppmmcmmappaaaa
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The Pile: An 800GB dataset of diverse text for language modeling. Preprint arXiv:2101.00027, 2020.
Atticus Geiger, Zhengxuan Wu, Christopher Potts, Thomas Icard, and Noah Goodman. Find- ing alignments between interpretable causal variables and distributed neural representations. In Causal Learning and Reasoning, pp. 160-187. PMLR, 2024.
Mor Geva, Avi Caciularu, Kevin Wang. and Yoav Goldberg. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. In Conference on Empirical Methods in Natural Language Processing, pp. 30-45. Association for Computational Linguistics, 2022. doi: 10. 10.18653/11/20022eemllp main..3. URL httthttt////cantthhtt 3.
org/2022. emnlp-main.
Micah Goldblum, Dimitris Tsipras, Chulin Xie, Xinyun Chen, Avi Schwarzschild, Dawn Song, Aleksander Madry, Bo Li, and Tom Goldstein. Dataset security for machine learning: Data poisoning, backdoor attacks, and defenses. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(2):1563-1580 2022.
Abhimanyu Hans, Yuxin Wen, Neel Jain, John Kirchenbauer, Hamid Kazemi, Prajwal Singhania, Siddharth Singh, Gowthami Somepalli, Jonas Geiping, Abhinav Bhatele, et al. Be like goldfihh, don't tt memorize! Mitigating memorization in generative LLMs. Preprint arXiv:2406. 10209, 2024.
Babak Hassibi, David G Stork, and Gregory J Wolff. Optimal brain surgeon and general network pruning. In IEEE International Conference on Neural Networks, pp. 293-299, IEEE, 1993.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. 2015 IEEE International Conference on Computer Vision (ICCV), pp. 1026-1034, 2015. doi: 10.1 109/CCCV.2015.22
Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. Preprint arXiv: /801 146, 2018.
12

---

Published as a conference paper at ICLR 2025
Jing Huang, Atticus Geiger, Karel D' D'Oosterlinck,, Zhengxuan Wu, and Christopher Potts. Rig- orously assessing natural language explanations of neurons. In 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pp. 317-331. Association for Compu- tational Linguistics, December 2023. doi: 10. 15.38633vv111 10033////222... ww0.223.bacchhoolp-112222
URL.
https://acllanthologg..om wwtphhoooooo...//2222// blackboxnlp-1.244
Juyong Jiang. Fan Wang. Jiasi Shen, Sungju Kim, and Sunghun Kim. A survey on large language models for code generation. Preprint arXiv:2406.00511, 2024.
Nikhil Kandpal, Eric Wallace, and Colin Raffel. Deduplicating training data mitigates privacy risks in language models. Preprint Stti::22220063399 2022.
Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace and Colin Raffel. Large language models struggle to learn long-tail knowledge. Preprint arXiv:2211.0844 1.0411,, 2023.
Daniel Kang and Tatsunori B. Hashimoto. Improved natural language generation via loss truncation. In 58th Annual Meeting of the Association for Computational Linguistics, pp. 718-731, Online, July 2020. Association for Computational Linguisties. doi: 10. 186533/1122222.clllll main.66. URL
https://aallanthooogg..o/// /wtp:t/hhhoooooooo/22/22// acl-main.66
Antonia Karamolegkou, Jiaang Li, Li Zhou, and Anders Søgaard. Copyright violations and large language models. In Conference on Empirical Methods in Natural Language Processing, 2023.
URL https::/ http:ppeeeeeee..eeeoriiiiiieeeee
Aly M. Kassem, Omar Mahmoud, and Sherif Saad. Preserving privacy through dememorization: An unlearning technique for mitigating memorization risks in language models. In Conference on Empirical Methods Natural Language Processing, 2023. URL https: /ttps:///ppereviee... net
/forum?id=jssi d-rruAii--SS
Eldar Kurtic, Daniel Campos, Tuan Nguyen, Elias Frantar, Mark Kurtz, Benjamin Fineran, Michael Goin, and Dan Alistarh. The optimal BERT surgeon: Scalable and accurate second-ordes pruning for large language models. In Conference Empirical Methods in Natural Language Processing, pp. 4163 4181. Association for Computational Linguistics, 2022. doi: 10. (011866///2222.emmm))) -m9.n.279. URL https://aclanthhtt
org/2022. een.222.nmmp-mi...... 279.
Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. Advances in Neural Information Processing Systems, 2, 1989.
Katherine Lee, Daphne Ippolito. Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison- Burch, and Nicholas Carlini. Deduplicating training data makes language models better. Preprint arXiv:2107.06499 2021.
Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison- Burch, and Nicholas Carlini. Deduplicating training data makes language models better. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8424 8445, Dublin, Ireland, May 2022a. Association for Computational Linguistics. doi: 10.18653/v 1/2653/1120222.alllllooo
URL https:: hatpp:://allanthology org 2022. acl- 577.
Mina Lee, Percy Liang, and Qian Yang. CoAuthor: Designing human-Al collaborative writing dataset for exploring language model capabilities. In CHI Conference on Human Factors in Computing Systems. ACM, 2022b. ISBN 9781450391573.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. Advances in Neural Information Processing Systems, 2018. https:
//doi.org/10 /tt.://////...//////////////
/ar.v./rrvvi...a/ 1712.09913.
Sijia Liu, Yuanshun Yao, Jinghan Jia, Stephen Casper, Nathalie Baracaldo, Peter Hase, Xiaojun Xu, Yuguang Yao, Hang Li, Kush R. Varshney, Mohit Bansal, Sanmi Koyejo, and Yang Liu. Rethinking machine unlearning for large language models. Preprint arXiv:2402.08787 2024.
Pratyush Maini, Michael C Mozer, Hanie Sedghi, Zachary C Lipton, J Zico Kolter, and Chiyuan Zhang. Can neural network memorization be localized? Preprint arXiv:2307.09542, 2023.
13

---

Published as a conference paper at ICLR 2025
C. H. Martin and M. W. Mahoney. Implicit reffrregularizaiion in deep neural networks: Evidence from random matrix theory and implications for learning. Journal of Machine Learning Research, 22(165):1-73, 2021.
C. H. Martin, T. S. Peng, and M. W. Mahoney. Predicting trends in the quality of state-of-the-an neural networks without access to training or testing data. Nature Communications, 12(4122): 1-13, 2021.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in GPT. Advances in Neural Information Processing Systems, 35:17359-17372, 2022.
Kevin Meng, Arnab Sen Sharma, Alex J Andonian, Yonatan Belinkov, and David Bau. Mass-editing memory in a transformer. In The Eleventh International Conference on Learning Representations, 2023. URL
net/
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference on Learning Representations, 2017. URL httpps://ppnnt view. www.net/oorrmmmnddyyy22kuee
Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning. Fast model editing at scale. In International Conference on Learning Representations, 2022. URL
Milad Nasr, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, Feder Cooper, Daphne Ip- polito, Christopher A. Proquette-Choo Eric Wallace, Florian Tramèr, and Katherine Lee. Scal- able extraction of training data from (production) language models. Preprint arXiv:2311.17035, 2023.
NERSC. Perlmutter architecture, 2024. URL /tt.:///////oooo.eeerr/////////e architecture/.
OpenAl. ChatGPT: Optimizing language models for dialogue, 2022. https::/oppeaii.com /chatgpt/.
Ashwince Panda, Christopher Choquette-Chov Zhengming Zhang, Yaoqing Yang, and Prateek Mittal. Teach LLMs to phish: Stealing private information from language models. Preprint arXiv:2403.00887, 2024.
Vaidehi Patil, Peter Hase, and Mohit Bansal. Can sensitive information be deleted from LLMs? Objectives for defending against extraction attacks. Preprint arXiv:2309. /7410, 2023.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to- text transformer. Journal of Machine Learning Research, 21(140):1 2020. URL ht //jmlr.org/papeeeeeeeee wttp///ppe//////.... /200 html.
Vivek Ramanujan, Mitchell Wortsman, Aniruddha Kembhavi, Ali Farhadi, and Mohammad Raste- gari. What's hidden in randomly weighted neural network? In IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 11893-11902, 2020.
Ludwig Schmidt, Shibani Santurkar, Dimitris Tsipras, Kunal Talwar, and Aleksander Madry. Ad- versarially robust generalization requires more data. Advances in neural information processing systems, 31, 2018.
Avi Schwarzschild, Zhili Feng. Pratyush Maini, Zachary C Lipton, and J Zico Kolter. Rethinking LLM memorization through the lens of adversarial compression. Preprint 2rX1v:240..15166 2024.
14

---

Published as a conference paper at ICLR 2025
Alia Shoaib. Why comedian Sarah Silverman is suing the company behind ChatGPT, 2023. https: /tww:. www..uussnssiiiiidr..r.m cww/why- comedian swaaa- siwverman----eeeeee suin-suing-thee taanng--th--cmppay -behind-ccattot-22233
Ilia Shumailov, Jamie Hayes, Eleni Triantafillou, Guillermo Ortiz-Jimenez, Nicolas Papemot, Matthew Jagielski, Itay Yona, Heidi Howard, and Eugene Bagdasaryan. UnUnlearning: Unlearn- ing not sufficient for content regulation in advanced generative AI. Preprint arXiv:2407.0011 2024.
Emily Silcock, Luca D' A'AmccoVVV Wonic Jinglin Yang, and Melissa Dell. Noise-robust de-duplication at scale. Technical report, National Bureau of Economic Research, 2022.
Robin Staab, Mark Vero, Mislav Balunovic, and Martin Vechev. Beyond memorization: Violating privacy via inference with large language models. In /2th International Conference on Learning Representations, 2024.
Cory Stephenson, Suchismita Padhy, Abhinav Ganesh, Yue Hui, Hanlin Tang, and Yeon Chung. On the geometry of generalization and memorization in deep neural networks. Preprint arXiv:2105.14602, 2021.
Niklas Stochr, Mitchell Gordon, Chiyuan Zhang, and Owen Lewis. Localizing paragraph memo- rization in language models. Preprint arXiv:2403. 19851, 2024.
Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena : A benchmark for efficient transformers. In International Conference on Learning Representations, 2021. URL https: //openreview. wwapeerrviee..ne/ooorniii w=ye////umemiyyyyeeeee
Yingjie Tian and Yuqi Zhang. A comprehensive survey on regularization strategies in machine learning. Information Fusion, 80:146-166, 2022.
Tempest A. van Schaik and Brittany Pugh. A field guide to automatic evaluation of LLM --M--eenrrted summaries. In 47th International ACM SIGIR Conference on Research and Development in In- formation Retrieval, pp. 2836. ACM, 2024.
Paul Voigt and Axel Von dem Bussche. The EU General Data Protection Regulation (GPDR): A Practical Guide. Springer, 2017.
Ben Wang and Aran Komatsuzaki. GPT-J-6B: 6 Billion Parameter Autoregressive Language Model. https://githur. cow//kingofloll/meehtttttttott hrapoooolr//mmstttttasrrreereeie May 2021.
Wenxuan Wang, Juluan Shi, Zhaopeng Tu, Youliang Yuan, Jen-tse Huang, Wenxiang Jiao, and Michael R Lyu. The earth flat? Unveiling factual errors in large language models. Preprint arXiv:2401.00761, 2024.
Xinwei Wu, Junzhuo Li. Minghui Xu, Weilong Dong, Shuangzhi Wu, Chao Bian, and Deyi Xiong. DEPN: Detecting and editing privacy neurons in pretrained language models. Preprint arXiv:2310.20133, 2023.
Jianhao Yan, Futing Wang, Yafu Li, and Yue Zhang. Potential and challenges of model editing for social debiasing. Preprint arXiv:2402. 13462, 2024.
Y. Yang, Theisen, L. Hodgkinson, J. E. Gonzalez, Ramchandran, C. Martin, and M. W. Ma- honey. Test accuracy vs. generalization gap: Model selection in NLP without accessing training or testing data (Evaluating natural language processing models with generalization metries that do not need access to any training or testing data). Technical Report Preprint: arXiv:2202.02842. 2022.
Jin Yao, Eli Chien, Minxin Du, Xinyao Niu, Tianhao Wang, Zezhou Cheng, and Xiang Yue. Ma- chine unlearning of pre-trained large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pp. 8409, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024, a/l1l/////.2....11.g555555. URL https://aclanttollgy.oott wrwph///oooyyo..2222....// acl- bell-long.43 4577.
15

---

Published as a conference paper at ICLR 2025
Yuichi Yoshida and Takeru Miyato. Spectral norm regularization for improving the generalizability of deep learning. Preprint arXiv: 1705. 10941, 2017.
Yu Yu, Shahram Khadivi, and Jia Xu. Can data diversity enhance learning generalization? In Proceedings of the 29th international conference on computational linguistics, pp. 4933-4945, 2022.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona T. Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettle- moyer. Opt: Open pre-trained transformer language models. ArXiv, abs/2205.0100 2022a. URL https://api semanticscholar www./orppsss..io..... 248496222.
Zhengming Zhang, Ashwinee Panda, Linyue Song. Yaoqing Yang, Michael Mahoney, Prateek Mit- tal, Ramchandran Kannan, and Joseph Gonzalez. Neurotoxin: Durable backdoors in federated learning. In International Conference Machine Learning, pp. 26429-26446. PMLR, 2022b.
Chen Zhu, Ankit Singh Rawat, Manzil Zaheer, Srinadh Bhojanapalli, Daliang Li, Felix Yu, and Sanjiv Kumar. Modifying memories in transformer models. Preprint 2rXin:2012.03363, 2020.
16

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 TinyMem: A TINY MODEL SUITE TO STUDY MEMORIZATION
We detail our TinyMem model data and training criteria in Sections A.1.1 and A.1.2. Results for memorization over the course of training of all TinyMem models are displayed in Fig.
A.1.1 THE TinyMem MATH MODELS
Vocabulary: Each model has 14 token vocabulary V + {"0" 0, '1" : '2" 22 "3" 33 " 4, "5" :5, "6" 6, "7" : 7, "8" 8, "9" : :9, A 10, "S" : 11,, 6" 12, a 13}.
Layers: We train models with varying numbers of layers {2, (2,4,816)) 16).
Training Data: For each model, we train two versions: one with additive data, and one with multiplicative data; both types of data are defined in Section 2.3.
For the additive data, we train each model jointly on five different addition tasks where we vary the additive bias parameter b (2, 4, 5, 7} per task. We consider learnable additive "task" to be the data set corresponding to a specific by for example, the "task" for b= 2 is "adding by 2."
For the multiplicative data, train each model jointly five different multiplication tasks where we vary the multiplicative coefficient w E {2. 3, 4, 5, 7), bias parameter = 0, and modulus parameter 20134 per task. We consider learnable multiplicative "task" be the data set corresponding to a specific w; for example, the "task" for w = 2 is "multiplying by 2 mod 20134."
Dataset Size: For both additive and multiplicative data, better understand the effect of dataset size on memorization, we train three versions of each math model with increasing "dataset size." Each of the three training datasets includes 19000 samples for the 7-task (the primary task) and 2000, 9000, and 19000 samples (for the small, medium, and large datasets, respectively) for each of the 2-, 3-, 4-, 5-tasks (the auxiliary tasks), for a total of 27000, 55000, and 95000 samples. We also create a test dataset comprising 5000 clean (i.e., non-perturbed) samples: 1000 each for the 2-, 3-, 4-, 5-, and 7-tasks."
For each model, we train two versions: one with noise defined in Def 2.2, and one with backdoors as defined in Def 2.3.
Artifacts:
Noise: When injecting a noise artifact into a dataset, we select 1000 random samples from our 7- task dataset and noise them using our noising scheme (see Def 2.2). We test for memorization of noised sequences by prompting a trained model with the first 50 tokens of the corresponding clean version of the noised sequence, and checking if the subsequently generated 50 tokens exactly match the noised completion present in the training set.
Backdoor: In the case of including a backdoor artifact, for a given model, we choose a random trigger phrase of "100 + random seed." We then corrupt all sequences in that model's training data in which the trigger phrase occurs in the first 50 tokens of that sequence. We use our backdooring scheme (see Def 2.3) to corrupt those sequences. Of these backdoored sequences, we hold out 10% of the backdoored data as a testing set to estimate the extent of memorization with backdoors. We prompt the trained model with the first P tokens, where P is the number of tokens prior and including the trigger phase, and then check if the following 50 tokens match the degenerate backdoor behavior that we train the model with.
Evaluation: Each math model is evaluated with a token-wist accuracy score. For the math se- quences, we choose to report the accuracy, rather than the perplexity, as there a notion of "cor- rectness" in mathematical sequences that is not present for general language modeling. We felt that accuracy is a more exact measure of a sequential math model compared to perplexity. An example of how we calculate token-wise accuracy: suppose we prompt model M with the first three tokens of sequence s = (2, 8, 10, 12} and we get the next three token completions ( 4, 6)) the accuracy would be 66%. our experiments, accuracy is always evaluated over 1000 randomly held out clean samples per task; "Dataset Size" above for details about test set creation.
{8, 11. 12). where boldface indicates mistake in the sequence completion, then
17

---

Published as a conference paper at ICLR 2025
Table 4: TinyMem Model Sizes. Comparison of the math-based and language-based model sizes
with respect to number of trainable parameters across various layer counts.
# Layers
Math
Language
2
11K
6.8M
814K
7.2M
8
1.6M
8.0M
16
3.2M
9.6M
A.1.2 THE TinyMem LANGUAGE MODELS
Vocabulary: Each model has a 50257-token GPT2 vocabulary (i.e., sub-word-based tokenization scheme).
Layers: We train models with number of layers E (,,4,8, 16).
Training Data: Models are trained Wikipedia corpus (Merity al., 2017).
Artifacts & Duplication: For each model, we train two versions: one with noise as defined in 2.2, and with backdoors defined in Def 2.3.
Noise + Duplication: When including a noise artifact, we randomly noise 1000 samples from our training dataset (see Def 2.2). We then split these 1000 noised samples into four sets of 250 samples each, which we duplicate 10", 10', 10², and 10 times, respectively. for a total of 250 x (1 10 100 + 1000) 277750 samples. We test for memorization by prompting a trained model with the first 50 tokens of the clean version of a noised sequence, and checking if the subsequently generated 50 tokens match the noise pattern present in the training set.
Backdoor + Duplication: Instead of using a trigger phrase (as in math). we use a single trigger token, randomly chosen via a seed; the backdooring scheme is defined in Def 2.3. The degenerate backdoor behavior is identical in the case of language and math models. We duplicate the training set for backdoored data 102 times.
Evaluation: Each language model is evaluated with a perplexity score on an held out Wikipedia test set which is detailed in (Merity al., 2017).
A.1.3 LM MEMORIZATION PROPERTIES
We describe factors that affect memorization in LMs.
Training Dataset Size. More training data leads less memorization. From Fig. 6a, we see that as we increase dataset size from left to right (i.e., the augmentation factor), the overall memorization decreases. This is also supported by the findings of Yu al. (2022) and Schmidt et al. (2018).
Train Data Duplication. More duplicated data is memorized to greater extent than less duplicated data. From Fig. 6d, we see that data duplicated 102 times was memorized less than data duplicated 10a times. This finding follows results from Carlini et al. (2023).
Model Size. Bigger models (i.e., models with more layers) memorize more. We see in Fig. 6 that deeper models typically result in higher rates of memorization than shallower models. This finding follows results from Carlini et (2023).
Training Data Artifacts. Noise artifects (see Figs. 6a, 6d) more difficult for LMs to memorize than backdoor artifacts (see Figs. 6b, 6c). Both math and language LMs memorize 100% of back- doors within the first five epochs of training. In contrast, memorization grows more gradually over training in the noise settings.
Context Length. In TinyMem, we constrain the model context length to 150 tokens. Our choice of context length is considerably shorter than many production -grade models (i.e., GPT2-small was trained with 1024 token context window (Radford et al., 2019)). While shorter context length enables rapid model inference and training, as the attention operation in transformer based LMs has time and memory complexity that scales quadratically with context length (Tay et al., 2021: Dao
18

---

Published as a conference paper at ICLR 2025
Augment
10000 Augmant
2000 Augment
Augment
Augment
Augment
Increment Mor.. 196)
Increment Mom. (55)
100
75
50-
20
100
Mult: Mam. (56
40-
Muit. Mem. (%)
2000
2000 Epoch
2000 Epoch
200
400
200
400
200
400
Epoch
Epoch
Epoch
Epoch
(a) Moth+NNoic
(b) Math+ Backdoor
Duplication 10a
100
800
Percent Memortea (56)
Porcort Momorized (96)
10
5
0-
Epoch
Epoch
Epoch
16
(c) Language+NNois
(d) Language+Backern Backdoor
Figure 6: (a) Math+Noise. Top increment task. Bottom multiply task. From left to right we increase the amount of data from each auxiliary task to 3000 (2000 train, 1000 test). 10000 (9000 train, 1000 test), and 20000 (19000 train, 1000 test), and show results for 4, 8, and 16 model layers. Training details are in Section A.1.1. (b) Math+Backdoor. Top row: increment task. Bottom row: multiply task. From left to right we increase the amount of data from each auxiliary task to 3000 (2000 train, 1000 test). 10000 (9000 train, 1000 test), and 20000 (19000 train, 1000 test), and show results for 4, and 16 model layers. Training details are in Section A.1.1. (c) Language+ Noise. 2-, 4- 8-, and 16-layer models with varying duplication regimes, detailed in Section A.1.2. (d) Language+Backdee Backdoor. 2-, 4-, 8-, and 16-layer models with fixed duplication regime, detailed in Section 1.1.
et al., 2022)). it limits our ability to test whether context length is a key factor in our model's ability to regurgitate memorized information as shown in Carlini al. (2023). For now, we choose to evaluate (n 100, & 50) memorization (as per Def 2.1). and we do not study the effect of context length on memorization. We justify the choice to only consider memorization at a fixed prompt length of k == 50 in our analysis many prior works have also considered single fixed prompt length when studying and designing memorization mitigation strategies (Chang et al., 2024: Biderman et al., 2023: Stoehr et al., 2024). The effect of context length memorization and unlearning strategies can easily be explored in future work, as the highly configurable TinyMem model training framework will allow users to train models with longer context lengths if needed.
19

---

Published as a conference paper at ICLR 2025
A.1.4 WHY Do WE NEED TinyMem?
TinyMem provides developers of memorization mitigation methods lightweight and fully open- source model testbed on which to develop, test, and hyper-parameter tune their methods prior to production-graad grade model method testing and deployment. This is necessary due to 1) the pro- hibitive computational cost (e.g.. memory, time) of inference and gradient-based operations, needed for methods development on large production-gret grade models; and 2) the lack of publicly available (model, memorized sequence) pairs.
1. Publicly available LMs that demonstrate memorization are large, prohibiting rapid prototyping and deployment. To develop, test, and tune memorization mitigation strate- gies, ML practitioners must incur the cost of repeated model inference and gradient calcu- lations, often requiring many GPUs. This can be prohibitively computationally expensive for many individuals and groups, and thus deter them from developing methods. For ex- ample, recent studies which quantify the amount of memorization exhibited by LM's are conducted exclusively with models in the million and billion parameter range:
(a) Carlini et al. (2023) studies the GPT-neo models (125M, 1.3B, 2.7B, 6B) (Black et al., 2021; Wang & Komatsuzaki, 2021), OPT models (125M, 350M, 1.3B, 6.7B, 30B, 66B) (Zhang et al., 2022a), (Lee et al., 2022a) models (1.5B) and the '5 models (250M, 770M ,3B) (Raffel et al., 2020).
(b) Chang et al. (2024) studies Pythia 2.8B and 6.9B (Biderman et al., 2023), and GPT-2
1.5B models (Radford et al., 2019).
(c) Stoehr et al. (2024) studies GPT-2 125M (Radford et al., 2019).
While many eminent memorization studies focus on extremely large models (Carlini et al., 2023; Chang et al., 2024), recent work has demonstrated the merit in studying memoriza- tion in smaller, less unwieldy models. For example, Stochr et al. (2024) observe "While these studies found that bigger model variants tend to memorize more, the smallest variant, GPT-NEO 125M, still exhibits extensive memorization behavior with an easier-to-study computational footprint. After all, when interpreting models at the level of individual weights, smaller models are easier to visualize and analyze."
In contrast to the large LMs used in (Carlini et al., 2023; Chang et al., 2024; Stoehr et al., 2024), TinyMem models range from 418K-9 9.6M trainable parameters (see Table 4). TinyMem models are significantly smaller than the smallest production plored in both (Carlini et al., 2023; Chang et al., 2024). Therefore TinyMem models enable much faster initial prototyping/testne of mitigation methods (Section A.6.1), followed by a more streamline methods testing phase on production-gen grade models (as proposed in Sec- tion 7).
models ex-
2. There is limited public availability of (LM,, memorized data) pairs. Of the three studies that we are aware of that release memorized datasets for publicly available models (Carlini et al., 2023; Chang et al., 2024; Stochr et al., 2024), we were only able to recover the data from two of them at the time of submission:
(a) (Carlini al., 2023): Attempted release 38000 memorized sequences for each of the GPT-neo models (125M, 1.3B, 2.7B, 6B) (Black et al., 2021; Wang & Komatsuzaki, 2021) at https: //gi thub. com/ com/google--oo weallerresearch/geee hothh/ooooll-eeerrrrreeeee Berr/m_meeorrizat
tete//eee/aaa//// /mas We tried to use the memorized data points from this paper, however the publicly downloadable memorized data points were corrupted for every model which had publicly released data. Despite correspondence with the authors were unable to resolve the data corruption issues prior to manuscript submission.
(b) (Chang et al., 2024): Published 505 memorized sequences for both Pythia 2.8B and 6.9B, but cite struggles to find memorized sequences for GPT-2 1.5B due to a lack of a public training dataset. Therefore, they manually search for memorized data points from several public corpora, finding 105 such data points. We use the Pythia 2.8B and 6.9B released memorized sequences to evaluate our memorization mitigation method in a production gradee setting (see Section 7).
(c) (Stoehr et al., 2024): Releases 422 memorized sequences for GPT-2 125M.
20

---

Published as a conference paper at ICLR 2025
A.2 REGULARIZERS
A.2.1 REGULARIZERS HYPER-PARAMETER PARAMETER SEARCH + SELECTION
Hyper-parameter search: For spectral norm regularization, we varied the hyperparameter lam {0.001, 0.01, 0.1}; lam the regularization coefficient for the regularization term in the loss function. For loss truncation, fraction the data with the highest log-loss to drop from any given batch during training. For example-tied dropout, varied the hyperparameter Pmem {0.01, 0.05, 0.1}: Pmem is the fraction of neurons to drop (i.e., the example-tied neurons) after training.
varied the hyperparameter dropc E {0.01, 0.05, 0.13; dropc is
Selection Criteria: For language models, we selected the model corresponding the training setting that resulted in the lowest average (test perplexity + percent memorized) across all three seeds:
LMbeet + min (avg (perplezity + %memorized) seeds
(2)
LMMs
For math models, we scored the best run by seeing which training setting resulted in the highest average (test accuracy percent memorized) across seeds:
LMbest
max (avg (accuracy + %mmmmorieed seeis)
(3)
LMs
A.2.2 REGULARIZER DEFINITIONS
The spectral norm regularization method is described in detail in Yoshida & Miyato (2017); we closely follow their implementation, which be found https:://tthubcoom// rch/sngan.projectionn
The loss truncation method is described in detail in Kang & Hashimoto (2020); closely follow their implementatoon, which can be found at https: thub. com/ homn/ddkagg/loc /loss dropper.
The example-tied dropout method is described in detail in Maini al. (2023); closely follow their implementation, which can be found at https: htt///gttttuuu hwtpppratyyssimmii////aaaa.....a g- -emmorization
A.2.3 REGULARIZER TRAINING GRAPHS
We include visualization of how memorization varied over the course of training in four-layer models from TinyMem with the use of regularizers in Fig. 7. We exclude results from the example- tied dropout strategy for language LMs the test perplexity consistently exceeded 500 for the entire duration of training.
21

---

Published as a conference paper at ICLR 2025
100
* Memonizes
% MMmorrzed
Average Test Accurecy
100
Average Text Accuracy
100
2000
3500
Epoch
Epoch
Spec.
Example Drap
Spec
Example Drop
(a) Math+Noose.
(b) Math+Backdoor
Duplication Factor 10
Duplication Factor
56 Memorzed
%o Memonoed
No Memonzed
17.5
12.5
96
10.0
Duplication Factor 10
Duplication Factor 10
Test Perplestty
Test Perplexity
Perplexity
80
70
70
70-
80
50 Epoch
100
50 Epoch
100
40
50
Epoch
Spec. Norm
Loss Tunnc.
Baseline
Norm
Loss Tunn
Baseline
(c) I Language+Noise
(d) Banguage=Backdoor
Figure Train time mitigations. (a) Math+Noise. We compare regularizers across four layer math models trained with noise. (b) Math+Backdoor We compare regularizers across four layer math models trained with backdoors. (c) Language+Noise. We compare regularizers across four layer language models trained with noise. (d) Language-Backdoor We compare regularizers across four layer language models trained with backdoors.
22

---

Published as a conference paper at ICLR 2025
A.3 MACHINE UNLEARNING
Every unlearning method we study is localization & ablation-based. This means that every strategy we study first attempts to isolate the subset of weight or neurons responsible for memorized sequence generation. Following this localization, we ablate the top K weights, where K is a prrammerr we tune. We make the decision to focus on on localization & ablation-based unlearning methods predicated on the findings that neural network memorization is localizable (Chang et al., 2024; Maini et al., 2023).
For each proposed method, we detail the intuition behind that method's design (Section A.3. I), technical descriptions (Section A.3.2), provide pseudo-code algorithmic description (Section A.3.3), and key differences compared to prior methods (Table 5 & Section A.3.4).
A.3.1 DESIGN INTUITION OF PROPOSED UNLEARNING METHONS
Greedy was developed by (Maini al., 2023), and treat as our baseline weight-based unlearn- ing method. Greedy is first-order method that has decent performance (see Tables 1, 2, 3) on both mitigating memorization and preserving model performance (e.g., perplexity on non-memorized dataset). However, it is iterative and therefore slow (Table 5). A key innovation in Greedy is that it calculate model gradients via a dual objective: maximize loss on the memorized set and retain low loss on a non-memorized "retain" set.
Durable Intuition:
We follow Greedy up with the Durable method which is designed to be non- iterative (and therefore much faster). Durable is inspired by (Zhang et al., 2022b)'s 's method used to implant "durable backdoors" in models. Zhang et al. (2022b) made the observation that by only updating weights that were routinely not optimized during typical training when training model to learn backdoors, the backdoors became more "durable" " and difficult to remove during fine-tuning. Following this intuition, we speculate that the reason sequences become memorized by LM is because they live in weight spaces that are not routinely updated during training for non-memorized sequences but are updated frequently for sequences that are memorized. Therefore, we develop Durable to seek out the top K% of weights per-layer that are updated with the average highest mag- nitude gradients when optimizing over our memorized sequence set. We then ablate these weights to remove memorization. This method is orders of magnitudes faster than Greedy (as seen in Tables 2).
Durable-agg Intuition: Durable-agg is an innovation of Durable that is designed to rank weight importance across layers. Since Durable only compared weights within layers, it may have over or under ranked the importance of weights with respect to the weights throughout the entire model. Therefore, Durable-agg is designed to find the top K% of weights per model with the highest average gradient magnitude when optimizing over the memorized sequence set. We then ablate these weights. Similar to Durable, Durable-agg is non-iterative and therefore fast.
SOU Intuition: SOU is second-order unlearning method. Greedy, Durable, and Durable-agg are first-order methods (c.g., gradient- but many second-order optimization (c.g., hessian- based) and second-order pruning methods exist and often boast improved performance over their first order counterparts (LeCun et al., 1989; Kurtic et al., 2022; Hassibi et al., 1993). Therefore, we took inspiration from Kurtic et al. (2022), and designed SOU. The pruning method in (Kurtic al., 2022) was designed to localize and excise weights that were inconsequential to sequence generation. We design SOU instead localize and excise weights that were consequential to memorized sequence generation. We found that SOU worked decently well (Tables 1, 2 & Fig. 3). Unfortunately. SOU scales quadratically in memory with respect to model parameter counts due to calculating the model hessian. With current computing infrastructure, this is infeasible to extend to billion+ parameter models. Further, we used an approximate hessian solver approach to speed up computation, but this likely degraded performance.
Subnet Intuition: We notice that Greedy, Durable, Durable-agg, and SOU are all "threshold- based" methods: weight importance w.r.t. memorization is ranked by a corresponding weight prop- erty (e.g., weight magnitude or gradient). By choosing a property, such as gradient magnitude, as a proxy for weight importance, threshold- based method can set threshold and exclude the top K% of weights deemed most related to memorization. This is a major shortcoming of threshold-ban based methods ranking weight importance by single properties may oversimplify how different model
23

---

Published as a conference paper at ICLR 2025
properties correspond to memorization. To overcome this shortcoming, design Subnet which learns a sparse binary mask over the model weights using a straight-through estimator. Subnet is inspired by application which learned sparse binary mask to find performant subnetworks within randomly initialized neural networks (Ramanujan et al., 2020). By learning a mask, instead of con- structing mask based weight-properties (as is done in threshold- methods), find that Subnet is able to greatly improve in performance over the prior threshold-base methods (Tables 1, 2, 3 & Fig. 3).
We notice that while Subnet exhibits superior performance to Durable, Durable-age, SOU, it still results in LM performance degradations over non-mene nommmeedd tasks Fig. 3. We notice in Tables 1, 2 that Greedy is sometimes able to retain superior performance (acouracy/perplexitt) over non-memorized sequences. Therefore, we design BalancedSubnet to combine the best features of Subnet and Greedy: it optimizes sparse mask using straight- through estimator to simultaneously localize and remove memorized sequences while minimizing loss a held-out non-memorized sequence set to preserve model performance. As expected, Bal- ancedSubnet worked well at both removing memorization while preserving model performance on unrelated tasks (see Tables 1, 2, 3, Fig. 5).
BalancedSubnet Intuition:
A.3.2 TECHNICAL DESCRIPTION OF PROPOSED UNLEARNING METHODS
Durable: Calculate the accumulate the gradient gradace of a set of memorized sequences for K given L.M. Then take the absolute value of Igradacel, and drop the top numberoflayers weights in each layer corresponding to the highest magnitude gradients.
Durable-agg: Calculate the accumulate the gradient gradace of set of memorized sequencces for a given L.M Then take the absolute value [gradace), and drop the top K weights across the whole model corresponding to the highest magnitude gradients.
SOU: Given set of rremoorzed_sequeeeee for a given LM, iterate through each LM layer and memoriizedssegeeeee
calculate the approximate inverse hessian corresponding approximate inverse hessian by approximating the inverse block fisher matrix F for each layer using the approach detailed in (Kurtic et al., 2022). For each layer, calculate the per-weight saliency score
Calculate the
using the formula: +- F=Dig((F) where is the flattened set of weights in the layer, and Diag( (F) is the diagonal of F. Finally, drop the top K weights across the whole model corresponding to the highest scores 8.
(P²)
Given set of memorized sequences for given LM, initialize a mask per layer of LM with Kaiming uniform distribution (He al., 2015). For num_epochs epochs, optimize the binary masks which drops K weights to maximize the loss over the seemorized.sequeeeee when applied to LMM. The optimization of the mask is done using a straight-through estimator, details are found in (Ramanujan et al., 2020).
Subnet:
BalancedSubnet: Given a set of memorized_sequene given LM, initialize mask per layer of LM with a Kaiming uniform distribution (He et al., 2015). For num epochs epochs, optimize the binary masks which drop weights to both maximize the loss over the memorized sequence and minimize the loss the random. sequences when applied to LM. The optimization of the mask is done using a straight-throug estimator, details are found in (Ramanujan et al., 2020).
and a set of random_sequences for
A.3.3 MACHINE UNLEARNING METHOD ALGORITHMIC DEFINITIONS
The neuron-level unlearning methods we study are described in detail in Chang et al. (2024); we closely follow their implementation which can be found at /tetts://giteee wwt.//oooe/atcteee terarachang// emPi
We detail the weight-level unlearning methods Algorithms: 1,2,3,4,5,6. In these algorithms, we vary the following input parameters:
1. LM: original language model
2. K: number of weights to drop (ratio num.modell.prrmmeters
3. num epochs: number of iterations perform the procedure
24

---

Published as a conference paper at ICLR 2025
4. memorized.sequences. set of sequences memorized by the LM
5. random_sequences. set of random sequences
6. woss-weight: weighting coefficient for BalancedSubnet optimization objective
Algorithm Greedy, from Maini et al. (2023)
procedure GREEDY(LM, K, memorized sequences, random sequences) LMedited Copy of LM
D Create array to store scores, score per LM parameter shuf fled-data t- Shuffle [memorizzed.eqqennces randoo_sequennes)
scores < [.-.]
while counter - K do
for batch shuf fled_dalss data do:
N + Number of sequences in batch Initialize array batch. waw./maakk........ for seq E balch do:
10
if seq memorized sequencees then
11
batch_mask (ancc_mmas((ddee0((((ss)) -1
12
else
13
batch mask(nnex00((ea)))
14
end if end for
15
16
loss & (batthe(batch)) * batch_mask
17
Backpropagate loss
18
for p L.Mestedpprrammeteee parameters do
19
$ - (=ssr Take absolute value of gradients of parameters w.r.t. loss
20
Append & to scores end for
22
L
P Drop I weight with highest score
23
counter +- counter + end for
24
25
end while
26
return LMedited
27
end procedure
Algorithm Second-Order Unlearning (SOU)
procedure SOU(LM, K, memorizzed
LMedited Copy of LM scores + -I..]
b> Create array to store scores,
score per LM parameter
for p E L..parameeerss do
F + Initialize Block Fisher Inverse matrix
> Hessian Approximation Initialization
for seq E memorized sequences do: loss <- LMMeditedd (Seedtt)aa Pgrad < p-osaddlooss)
Obtain gradients p w.r.t. loss
Update F with Pgrad
P Hessian Approximation Update
10
end for
(P2)
11
Z+Dia[(F]
12
Append to scores end for
13
14
LMedited t- top weigkkksiihhsLMMM, scoress Drop top K weights with highest Scores
15
return LMadited
16 end procedure
25

---

Published as a conference paper at ICLR 2025
Algorithm Durable Aggregate (Durable-agg)
procedure DURABLE-AGGLLM, memnorized.seeuennee
LMedsted <- Copy of LM
scores - [...]
> Create array to store scores, score per LM parameter
for seq E memorized sequences do:
loss +- LMedit (See))((s)......... (seq).loss
Backpropagate loss
end for
for p E LM L.Mpparmeeers do
+ [p.grad(loss]
> Take absolute values of gradients w.r.t. loss
10
Append to scores end for 11
12
e.eedsted <- drrop.top...weighhtse (..osss)) scores) Drop top weights with highest Scores
13
return LMedited
14 end procedure
Algorithm Durable
procedure DURABLE(LM, K, memerized.sequeeee)))
LMeeisted & Copy of LM
for seq E seeemorized_sequereee
do:
loss t- LMedia (aMMeaassee)llsss (seq).loss
Backpropagate loss end for for p E L.Mppramelers do |p-grad(loss)
Take absolute values of gradients w.r.t. loss
layer + p.layer p f trro.tto.....iihhht.prrlrrrr// end for
10
b Drop top weights per layer
11
12
return L.eetited
13
end procedure
Algorithm Subnet
procedure SUBNET(LM, K, memorizedd.squuenee num_epochs)
LMedsted < Copy of LM scores + kaiming uni (orfor(((...]) Score array w/ kaiming init., score per parameter
Initialize oplimizer state with scores
for E num_epochs do for seq E memorizzedsseeueeeess do: for i Len(LLMMparrmmeeere do
L.Medited.parrmmeterr + LM .arrmeeeetess
Restore layer weights
p A.Melitedpparameeten
Parameters for layer i
10
$ < |scores [i] p drop k par. end for loss LMMest (seq)
Scores for layer
11
Drop top weights per layer
12
13
14
Backpropagate loss
15
optimizer step end for end for
This updates scores w/ gradients (not LM parameters)
16
17
18
return LMedited
end procedure 19
26

---

Published as a conference paper at ICLR 2025
Algorithm BalancedSubnet
procedure BALANCEDSURNETTLM,, K, memorized seemorizcd.sequencee. random rum hemm.epochs loss_ wwiight)
LMedited Copy of LM
scores + baiming-urrd uniffrrm(...)) > Score array w/ kaiming init., score per parameter
Initialize optimizer state with scores
shuf fled data + Shuffle [memorized sequences U random sequences] for E mum epochs do for batch E sheuf fled_data do: for E parammmterrtt do
paraattaapprrrmmeeresss] + LM [..pprameterssj]]
Restore layer weights
10
p + L.Meited.prrameters
> Parameters for layer
11
|scores[i]
Scores for layer i
12
p + drop lop k weights per layer(p, 8)
b Drop top K weights per layer
13
end for
14
N <- Number of sequences in batch
15
Initialize array batch_mask[I....) for seq E batch do:
16
17
if seq memorized sequences then
18
batch_mask ((ee + - loss weight) weighth)hh))
19
else
20
kutch end if end for
+- loss_weighl
21
22
23
loss + LMeditea (batch)..oss batch. mask
24
Backpropagate loss
25
optimizer step end for end for
D. This updates scores w/ gradients (not LM parameters)
26
27
28
return LMedited
end procedure 29
27

---

Published as a conference paper at ICLR 2025
A.3.4 KEY DIFFERENCES BETWEEN UNLEARNING METHODS
In this section, we outline the properties that characterize unlearning methods and summarize them in Table 5. Below, we describe each property in detail, highlighting their advantages and limitations.
1. Selection Information
Zeroth-Order: This property key methods that make use of zeroth-order information (e.g., weight magnitudes, activation values) rather than higher-order information (e.g., gra- dients, hessians). Zero order methods are time- and memory-efficient as they avoid costly gradient or hessian computations.
First-Order: A method first-order relies on model gradients w.r.t. a given dataset. Computing the gradient can be computationally costly. However, they can potentially be more informative than zeroth-order methods to select appropriate neurons/weights to ab- late.
Second-Order: A second-order method uses second-order information (e.g., hessian) w.rt. some dataset inform its ablation strategy. Computing the hessian matrix is memory- intensive and is often approximated using lossy techniques (Kurtic et al., 2022). While second-order methods can be computationally prohibitive, they leverage the eurvature in- formation of the loss landscape geometry to identify weights responsible for memorization, rendering them superior first-order gradient-buae baseed methods. However, in practice, model hessians are often approximated, which can result in performance degradation.
2. Selection Order
Iterative: A method is iterative if incrementally selects the individual model components (e.g.. weights, neurons) responsible for memorization. Iterative methods are inherently slower than non-iterative methods as shown in Tables 1, 2, 3 due the incremental nature of evaluating and ablating model components.
3. Selection Scope
Layer-wise: Layer-wise methods assess model components (e.g.. weights, neurons) im- portance w.r.t memorization within each layer as opposed to within an entire model. A key limitation of these methods is that they may fail to appropriately rank component im- portance within the full model scope.
4. Selection Criterion
Threshold-based: The proposed methods rank model components (c.g., weights, neurons) by their importance w.r.t. memorization by comparing a corresponding component property (e.g., gradient, magnitude). Using component property as a proxy for component impor- tance, the method can set threshold to exclude the top K% of components most influential for memorization. These methods are simple to implement and human-interpretaa may oversimplify how different model properties correspond to memorization and there- fore may not be as accurate optimization-baase methods.
They
Optimization-based:: These methods learn the order of importance of model components (e.g., weights, neurons) w.r.t. to memorization, rather than imposing a threshold criterion to rank the importance. While such approaches might be less interpretable than threshold- based methods, they may capture more nuanced information that might be more informative identify model components relevant to memorization.
5. Selection Objective
Dual-Objective: A method dual-objective if it considers the joint objectives of miti- gating memorization and preserving model performance on "retain" set of representa- tive (non-memorized) sequences. Methods that are not dual-objective, on the other hand, only account for mitigating memorization. Consequently, dual-objective methods more successful at preserving model performance non-memorized tasks, while mitigat- ing memorization. These methods are slower than single-objective methods as they of- ten rely on datasets containing both memorized and non-memorized data (rather than just memorized data for the single-objective case).
much
6. Component Precision
Neuron vs. Weight-based: Neuron-based methods assess the importance of individual neurons of an LM w.r.t. memorized sequence generation while weight-based methods
28

---

Published as a conference paper at ICLR 2025
Table 5: Properties of Unlearning Methods We identify key properties of the unlearning-basee methods. Boldface indicates the proposed methods. Coptim-based is short for oppimization-basel Thresh-base based short for Threshold-baseee indicates weight- based method; indicates a neuron- based method.)
Selection Information
Selection
Selection Selection Criterion Scope
Selection
Property
order
Objective
Zeroth- Order
First- Order
Second- Order
Iterative
Layer- wise
Thresh- based
Optim- based
Dual Ob-
jective
HCn
Slim,
Act,
v
v
IG,
Zero,
Greedy
/
SOU,,
Durable,,
/
Durable-agg,
Subnet..
BalancedSubnet,,
sess the importance of individual weights. Neuron-basee methods are faster than weight- based methods as there are fewer neurons than weights in an LM. However, weight-base methods offer more precision when localizing memorized information (see Fig. 2).
29

---

Published as a conference paper at ICLR 2025
A.3.5
MACHINE UNLEARNING HYPPERPPARAMETE SEARCH + SELECTION
Hyper-parameter search: For each model in TinyMem, we vary these hyperparameters for various methods:
BalancedSubnet:
ratio E {0.00001, 0.0001, 0.001, 0.01, 0.05, 0.1, 0.25, 0.3) npmcepochs [1, 20}
loss lossw E {0.9, 0.7, 0.5}
Subnet:
ratio E (0.0000,,,.0001, 0.001,,0,,,0 0,0, 0.,, num.epochs 10, 20)
HC, Slim:
ratio E (0.000,,, 0,000,0000000000,0000000000000 0.1
num-epocks
20)
Greedy:
ratio E {0.00001, 0.0001, 0,01,,000..0000001110000000
Durable, Durable-agg. SOU:
ratio E 000000011100000,00000000000,,,,0
Act, Zero, IG:
* ratio E (0.0001, 0.001, 0.01, 0.05, 0.1}
We make two key changes to our hyperparameter sweep for some of the larger models in TinyMem due time constraints. For models with 16 layers, do not test IG unlearning and constrain the search for Greedy ratio E {0.00001, 0.0001, 0.001, 0.013. For any language model trained on Wikipedia, we constrain the search for Greedy ratio {0.00001, 0.0001, 0.001, 0.01).
Exceptions:
For TinyMem models trained on sequential math data, we scored the best run by assessing which training setting resulted in the lowest score M P where M is the memorization percent difference before and after edit, A is the average accuracy percent difference (across 2-, 3-, 4,55, 5-, and 7-tasks) before and after edit. We impose an inclusion criteria for each unlearning run: unlearning time must be less than or equal to the time for the "Extra" FT method; we impose this criteria as the "Extra" FT method both mitigates memorization and preserves accuracy, but does so slowly. If method does not have single run that satisfies this inclusion criteria, then we do not enforce the inclusion criteria for that particular unlearning method.
Selection Criteria:
For TinyMem models trained Wikipedia, resulted in the lowest score = .M - P + t where M the memorization percent difference before and after edit, P is the percent difference in perplexity before and after edit, and t is optional penalty to the score: if M = 0, then / = 100, else / O. We include this penalty to ensure that methods that do not reduce memorization at all penalized memorization to any extent. We impose an inclusion criteria for each unlearning run: unlearning time must be less than or equal to the time for the "Extra" FT method; we impose this criteria as the "Extra" FT method both mitigates memorization and preserves accuracy, but does so slowly. If a method does not have single run that satisfies this inclusion criteria, then we do not enforce the inclusion criteria for that particular unlearning method.
scored the best run by assessing which training setting
than methods that do reduce
30

---

Published as a conference paper at ICLR 2025
A.3.6
MACHINE UNLEARNING RESULTS
Subietie
Gready
Submat
Durabre
Ourabitta
Act
5tm
Mun DHI
-00
Figure 8: Math + Noise Models Unlearning: Memorization % different (closer to -100 is better in top row), accuracy % difference before and after unlearning (closer to is better bottom row). Each line is averaged over three seeds, math and increment models, each trained three different datasets of increasing size.
Suonelur
Greedy
Subret
Dumble e
Duratieus
IG
S0U
Slim
HC
Zem
Dill
Man
-00
%
Booch
Esoch
Epach
Epoch
Epach
Epoch
Eacch
Epoch
Esoch
Epech
Epoch
26.2
Figure 9: Math + Backdoor Models Unlearning: Memorization % different (closer to -100 is better in top row), accuracy % difference before and after unlearning (closer is better in bottom row). Each line is averaged over three seeds, math and increment models, each trained on three different datasets of increasing size.
Subpebur
Gready
Subnet
Dumble
Durahirus
500
Zem
Men 24 DiLL
%
-300
30
Sooch
Epech
Epach
Epoch
Epoch
Epoch
Eoach
Esocs
Eoach
Epach
Spocn
2.2
5.0
262
Figure 10: Language + Noise Models Unlearning: Memorization % different (closer to -100 is better in top row), % difference in perplexity before and after unlearning (closer to 0 is better in bottom row). Each line is averaged over three seeds.
31

---

Published as a conference paper at ICLR 2025
Durabie
Wem
Perp DIt.
Soto
Figure 11: Language Backdoor Models Unlearning: Memorization % different (closer to -100 better in top row), % difference in perplexity before and after unlearning (closer to 0 is better in bottom row). Each line is averaged over three seeds.
Epoch: 500
Epoch: 1500
Epoch: 2500
Epoch: 3500
2 Layer
x
-50
-100
4 Layer
Memorization % Diff.
50
-100
8 Layer
-50
36
100
16 Layer
-50
-100
--10
-10
-102
0-100
-10
-102
0-100
-10²
-10²
--100
-10²
-102
Accuracy % Diff.
Accuracy Diff.
Accuracy Diff.
Accuracy % Diff.
Method
Slim HC
Greedy
SOU
Data Size
x 10000
Zero
Durable
Subnet
3000
20000
Act
IG
Durable ago
Subnetau
Figure 12: Math + Noise: Comparison of memorization % difference (closer to -100 better) and accuracy % different (closer to 0 better) before and after unlearning. Stratified by layers and training duration. Each point represents an average over three seeds. X-axis is on log scale. Aggregate results from both multiplication and increment models.
32

---

Published as a conference paper at ICLR 2025
Epoch: 50
Epoch: 200
Epoch: 350
Epoch: 500
2 Layer
-50
100
4 LLaer
Memorization % Diff.
-50
-100
8 Layer
-50
-100
16 Layer
-50
-100
0-10
-101
-102
-100
-101
-102
-110
-10"
-10²
-100
-10*
-10²
Accuracy % Diff.
Accuracy Diff.
Accuracy Diff.
Accuracy % Diff.
Method
Slim
Greedy
SOU
Data Size
x 10000
Zero
HC
Durable
Subnet
3000
20000
Act
IG
Durable
Subnet
Figure 13: Math Backdoor Comparison of memorization % difference (closer to -100 better) and accuracy % different (closer to 0 better) before and after unlearning. Stratified by layers and training duration. Each point represents an average over three seeds. X-axis is on log scale. Aggregate results from both multiplication and increment models.
33

---

Published as a conference paper at ICLR 2025
Epoch:
Epoch: 40
Epoch: 70
Epoch: 100
2 Layer
-80
-100
-70
4 Layer
-80
Memorization % Diff.
100
-70
8 Layer
-80
-90
-100
-70
16 Layer
-80
-90
-100
Do
10a Perplexity Diff.
110 10
10t
-100 10
10
10²
-100
10a 10²
- -10% 10
10t 10²
Perplexity % Diff.
Perplexity % Diff.
Perplexity Diff.
Zero Act
Slim
IG Greedy
Durable
SOU
Subnetoa
Durable
Subnet
Figure 14: Language + Noise: Comparison of memorization % difference (closer to -100 better) and % difference in perplexity (closer to 0 better) before and after unlearning. Stratified by layers and training duration. Each point represents an average over three seeds. X-axis is on log scale.
34

---

Published as a conference paper at ICLR 2025
Epoch:
Epoch:
Epoch: 30
Epoch:
2 Layer
-50-
-100
4 Layer
Memorization % Diff.
-50
-100
8 Layer
-50
16 Layer
-50
-100
010"
10 102 10a
010" 10a 102 10a
0100 10a 102 10a
0100 10a 102 10º
Perplexity % Diff.
Perplexity % Diff.
Perplexity % Diff.
Perplexity % Diff.
Zero
Slim HC
Durable
SOU
Subnetaw
Act
Greedy
Durable lega
Subnet
Figure 15: Language Backdoor: Comparison of memorization % difference (closer -100 better) versus % difference in perplexity (closer to 0 better) before and after unlearning. Stratified by layers and training duration. Each point represents an average over three seeds. X-axis is log scale.
35

---

Published as a conference paper at ICLR 2025
A.3.7 WHY DOES BALANCEDSUBNET WORK WELL?
Through extensive analysis in Tables 2 3 and Figures & we notice that BalancedSubnet consistently outperforms all other unlearning-base baasidd mitigation methods. In this section, we analyses the reasons for its success,
First we describe the design choices (Section A.3.4) that led to BalancedSubnet's success:
1. Weight-basede From Fig. 2, we see that while both neuron- and weight-based methods can both preserve LM perplexity, weight- weight-based based methods substantially outperform couron-basee methods at mitigating memorization. This is likely due to weight- bashd methods operating at a finer level of granularity than neuron-basee -asedd.
2. 1st order: From Tables 1 & 2, we see that this first order method is much faster than SOU, a second order method. From Fig. 3, we see that this first order method outperforms both zero order methods (Act, Zero); this is likely due first order information capturing more information about the loss landscape with respect to memorized content compared to zero order information.
3. Non-iterative: From Table 3, we see that BalancedSubnet is substantially faster than iterative methods like Greedy.
4. -odel-wise Scope: From Tables 1, & 3 and Fig. 3, we see that BalancedSubnet outper- forms methods with layer-wise scopes (Durable, SOU). This is likely because layer-wise methods only rank weight importance w.r.t. other weights in the same layer. This means that some weights may be artificially under or over ranked due to the lack of model-wide weight importance comparison.
5. Optimization-baseee based: From Fig. 5, we experimentally observe that by learning weight im- portance scores w.r.f memorization via optimization-basel approaches, like Subnet and BalancedSubnet, we are able to more precisely localize and ablate the set of weights responsible for memorization and not responsible for general sequence generation. Bal- ancedSubnet, like Subnet, is an optimization-ban based technique. Threshold-bare basee techniques assign proxy importance scores to each weight w.r.t. memorization and ablate weights with the highest scores. For example, gra- dients accumulated over the memorized set can be a proxy score for weight importance; weights with high proxy scores are ablated. While threshold-based techniques are more human-- interanniterrrctablll the proxy metrics they rely on may not capture the nuanced depen- dencies within the LM weights that are responsible for memorization.
technique rather than threshold-
6. Dual Objective: The dual optimization objective in BalancedSubnet is essential to both remove memorized content while retaining strong performance on non-memorized data. Without the dual optimization objective, BalancedSubnet is equivalent to the Subnet strat- egy (which only optimizes to reduce memorization). On evaluating Subnet, we observed that, in some cases, it aggressively removed weights that were instrumental for generating memorized sequences. BalancedSubnet was, therefore, designed to also include the objective of minimizing loss on a held out "retain" set. By adding this second "retain" set objective into our loss function, we validate this hypothesis and report the increase in performance on unseen test set (see Tables 1, 2 & 3).
Next we analyze the ratio of neurons/weight each unlearning method ablates from an LM in Figures 16, 18, 17, 19. We notice that most souron-based methods prune ~ 4 - 6% of model neurons; when stratified by model size & training data (Fig. 16 & Fig. 18), we do notice any clear trends in the TinyMem models. We notice most weight-based methods prune ~< 5% of model weights with the notable exceptions of Subnet and BalancedSubnet for TinyMem models. We notice most weight- based methods prune ~e 0.01% of weights with the notable exceptions of Subnet and Balanced- Subnet for Pythia 2.8/6.9R (.B. models which prune fied by model size (Fig. 17), we notice that both Subnet and BalancedSubnet prune more weights for larger models. For the TinyMem models, when stratified by model training data+artifact type (Fig. 19), we notice that both Subnet and BalancedSubnet roughly prune the most to least weights in the order of "Math+Noise". "Math+Backdoor" "Language+Backdoor"" Backdoor", "Language+Noois speculate that both Subnet and BalancedSubnet pruned more weights when the model is inher- ently more sparse, as may be the case with an over over- pee-ppaameterizeed model (bigger model size or easier training task).
10%. For the TinyMem models, when strati-
We
36

---

Published as a conference paper at ICLR 2025
BalancedSubnet, the most performant method, pruned significantly more weights than any other method. This finding gave further credence to the quual-opimizztionnbaa based design of BalancedSub- net. Despite dropping significantly fewer weights than BalancedSubnet, all other unlearning meth- ods struggle to reach the same performance (memorization mitigation & performance preservation). This suggests that most methods not able precisely localize+ablate weights responsible for memorized sequence generation/inconsqquuai to general sequence generation.
10
10
layers
EleutherAA/ppythiaa..ee http:///yttha-2-.--dduede
2.0
EleutherAl/pychi-6. -at
Ratio of Neurons Ablated (%)
4.0
Ratio of Neurons Ablated (%)
8.0
16.0
HC
Zero
Slim
Act
Act
Slim
HC
Figure 16: Neuron Ratio Across Model Sizes. Comparison of number of dropped neurons for neuron-based methods, for both TinyMem and Pythia for varying model sizes averaged across several unlearning times, data sizes, and three seeds.
layers
EleutherA/pythia-6 ebb.eedddddd.bddddddueeeeeddd EleutherAl/pythia> d.b-eeduued
Ratio of Weights Ablated (%)
2.0
Ratio Weights Ablated (%)
4.0
8
16.0
4
2
SOU
Durableass
Greedy
Durable
Subnet
Subnetaw
Greedy
Durableagg
Durable
Subnetas
Subnet
Figure 17: Weight Ratio Across Model Sizes. Comparison of number of dropped weights for waight-based methods, for both TinyMem and Pythia for varying model sizes averaged across several unlearning times, data sizes, and three seeds.
37

---

Published as a conference paper at ICLR 2025
10
Math+Noise
MathoBackooor
EleutherA//pythia-6 httpt/hhhha.h..eeee. 6.9b-deduper
Ratio of Neurons Ablated (%)
Language+ Backdoor
Language Noise
HC
Zero
Slim
Act
Act
Slim
HC
Figure 18: Neuron Ratio Across Model Datasets+Artifacts Comparison of number of dropped neurons for neuron-based methods, for both TinyMem and Pythia for varying model types averaged across several unlearning times, data sizes, and three seeds.
60
10
Math + Noise
Math Backkdor
EleutherAl/pythla-2.ce Sbb-dudde
Ratio of Weights Ablated (%)
50
Ratio of Weights Ablated (%)
Language+ Backdoor
Language++ Noise
40
6
30
4
20
10
SOU
Durable Greedy
Durable
Subnet
Subnetaal
Greedy
Durable
Durable
Subneted
Subnet
Figure 19: Weight Ratio Across Model DrtiasttsArtiffaetss. Comparison of number of dropped weights for weight-based methods, for both TinyMem and Pythia for varying model types averaged across several unlearning times, data sizes, and three seeds.
38

---

Published as a conference paper at ICLR 2025
A.4 PRODUCTION-GRADE UNLEARNING
A.4.1 MEMORIZED DATA EXTRACTION FOR PYTHI MODELS
We follow the methods outlined in Chang et al. (2024) to extract two datasets of memorized text, one each from Pythia 2.8B and 6.9B. Chang al. (2024) considers sequence be memorized by an LM if it is able to produce completions that "nearly reconstruct" the original suffix, given an input prefix sequence; this definition is more relaxed than the one we consider this work (Def 2. I), which deems a sequence be memorized only if the LM can reconstruct the suffix verbatim using greedy decoding. Therefore, we analyze how many of the sequences from Chang et al. (2024) fit definition of memorization, Def 2.1, when considering (n 72, k 32). Fig. 20 shows how memorization grows over training as model perplexity (a general metric of model performance) decreases (desirable). Memorization can be visualized in Fig. 20. We evaluate Pythia model perplexity over 1632 randomly sampled sequences of the Pile (Gao et al., 2020), following the same random sampling procedure (Chang et al., 2024).
training and perplexity
training for Pythia 2.8/6.9
Model
-
*
Perplestty
Percent
100K
Steps
Figure 20: Perplexity and Memorization of Pythia models over training.
A.4.2 PYTHIA: MACHINE UNLEARNING HYPER- PIEER-PARAMETER SEARCH + SELECTION
Hyper-parameter search: For both Pythia 2.8/6.9B, we vary these hyperparameters for various methods:
BalancedSubnet:
ratio e {0.00001, 0.0001, 0.001, 0.01, 0.05, 0.25, 0.3)
num_epochs {1, 10,20)
loss_weight E (0.9, 0.7, 0.5)
Subnet:
ratio E [00.00001, 0.0001, 0.001, 0.05, 0.1, 0.25, 0.3)
num_epochs E 10, 20)
HC, Slim:
ratio E [0000000,, 0.0001, 0.000,,001,, 0.05, 0.1} num_epochs {1, 20}
Greedy:
ratio E {0.00001}
Act:
39

---

Published as a conference paper at ICLR 2025
ratio
E 00.0001,00001,.0010.00666 0.1}
Durable, Durable-agg:
ratio E 0,,00000000000,,0000000000,,,,,, 0.1}
Selection Criteria: For Pythia models, we scored the best run by assessing which training setting resulted in the lowest score M + P + where M = memorization percent difference before and after edit, P = perplexity percent difference before and after edit, & an optional penalty to the score: if M = O, then / 100, else t = O. We include this penalty to ensure that methods that do not reduce memorization at all are penalized more than methods that do reduce memorization to any extent. The lower the score, the better. We impose inclusion criteria for each unlearning run: model perplexity has to be < 500.
Localization Method
Act
Slim
Memorization Difr.
20
HC
Greedy
Durable
40
Durable
Subnet
-60
Subnets
-100
20
40
60
80
100
Perplexity % Diff.
Figure 21: Unlearning strategies comparison. Comparison of memorization % difference (closer 100 better) and % difference in perplexity (closer is better), before and after unlearning. We visualize all unlearning results with model perplexity < 40 and % memorized < 50. We notice that BalancedSubnet (Subnetaat) has the highest density both near -100% difference in memorized and 0% difference in perplexity out of all of the unlearning methods.
40

---

Published as a conference paper at ICLR 2025
Training Step: 36000
Training Step: 72000
Training Step: 108000
Training Step: 143000
Memorization % Diff.
6.98
-50
-100
2.88
-50
-100
Perplexity % Diff.
Perplexity % Diff.
Perplexity % Diff.
Perplexity Diff.
Act
HC Greedy
Durable
Subnet
Slim
Durableage
Subnetaa
Figure 22: Pythia Comparison of memorization difference (closer to -100 better) versus per- plexity % different (closer to 0 better) before and after unlearning. Stratified by model type and train step. X-axis is on log scale.
41

---

Published as a conference paper at ICLR 2025
Table System Parameters for Polaris (ALCF, 2024) and Perlmutter (NERSC, 2024) estimate energy and carbon usage.
Machine
Cuf
Ctdp
ngpu
9uf
cd
DRAM
PUE
Polaris
0.5
225
1
250
512
1.58
Perlmutter
0.5
280
1
300
256
1.58
Table Energy & Carbon Estimates for our experiments based on Eq. (4) & Eq. (5) respectively.
Experiment
Machine
Node Hours
Energy (kWh)
Carbon (Kg)
(i) Training TinyMem models
Polaris
1116
2297
910
TinyMem Mitigation
Polaris
6251
12,871
5097
(iii)
Production-Graid Mitigation
Perlmutter
726
1,647
430
All Experiments (total)
16,815
6,437
A.5 Loss LANDSCAPE VISUALIZATION
We created the loss landscape visualizations in Fig. 4 following the approach introduced by Li et al. (2018). We use the corresponding Python package which can be found at https: //gps:///tt //gi arcellodebernardi/loss Latit/saass-aaacpppsss..
thub. com/m
A.6 COMPUTATION COST OF EXPERIMENTS
The experiments in this paper used approximately 16815 kWh energy, and 6437 Kg of carbon.
We detail the computational resource (i.e., node hours, energy, carbon) used by experiments below.
To calculate node hours, time all of our final experiments and triple that value to account for extra debugging time, faulty runs, and any unaccounted for compute usage.
To calculate energy usage and carbon cost of experiments, follow the methodology detailed by Bouza et al. (2023):
eneryy = NH ((cur Cup) + (ngpu Out gpudp) (0.3725 W/Gb DRAM)) PUE, (4)
where is node hours, Car is the CPU usage factor, Cap the CPU's thermal design power, agpu the number of GPUs node, 9ut the GPU usage factor (we assume 100% utilization), 9up is the GPU's thermal design power, DRAM is dynamic random access memory, and PUE is the power usage efficiency. energy is reported in watt hours. We record system-specific parameter values in Table
carbon = (energy/1000) * CI
(5)
Above in Eq. (5), energy is obtained in watt hours from Eq. (4), and CI is the carbon intensity reported based on the yearly regional average for each computing center from (ElectricityMaps, 2024). carbon is reported in grams.
To offset the carbon cost of these experiments, we publicly release our check pointed trained models (upon publication).
We group experiments into three main phases: (i) Training TinyMem Models; (ii) TinyMem Models LM Mitigation and (iii) Production-Gradd LM Memorization Mitigation. We report the resource usage for each phase of experimentation in Table 7.
A.6.1 ENERGY AND CARBON SAVINGS DUE TinyMem
In Table 8 we estimate the per-experiment node hour, energy, and carbon usage for both phase (ii) which used TinyMem to comprehensively test our mitigation methods and phase (iii) which extended
42

---

Published as a conference paper at ICLR 2025
Table 8: Experiment- Resource Usage. We calculate the number of experiments for phase (ii) and phase (iii) from Section A.3.5 & Section A.4.2 respectively. We then calculate the per experiment Node Hours (NH), Energy, and Carbon usage using estimates from Table 7.
Experiment
# Experiments
NH/Exp.
Energy /Exp. (kWh) Carbon/Exp. (Kg)
(ii) TinyMem (iii Production-Grade
102698
0.06
0.13
0.05
1216
0.60
1.35
0.35
Table 9: Hypothetical Phase (ii) Resource Usage, Resource usage estimates of doing 102, 698
phase (ii) experiments with TinyMem and without TinyMem (i.e, with production-gran LMs) using experiment-wee wise resource usage estimates from Table
Resource Type
TinyMem
w/o TinyMem
% Increase
Node Hours
6,251
61,619
986% (1)
Energy (kWh)
12,871
138,642
1077% (1)
Carbon (Kg)
5,097
36,316
712% (1)
the most promising mitigation methods from TinyMem to production-grade models. Based on the results in Table 8, if we had to conduct the same level of experimentation in phase (ii) without the use of TinyMem (i.e., using production-gen grade models directly). we would use:
0.60 NH/Exp. 102698 Exp. 61619 Node Hours
1.35 KWh/Exp. * 102698 Exp. 138,642 kWh Energy
0.35 Kg/Exp. * 102698 Exp. 36,316 Kg Carbon
We compare the actual experiment costs of phase (ii) with TinyMem with the hypothetical costs of phase (ii) without TinyMem in Table
A.7 EXTENDED RELATED WORK
Knowledge Editing aims change specific facts, associations, or information embedded in an LM outside of the constraints of traditional model training. Model editing requires the ability to localize learned information within subsets the weight space and employs efficient and targeted methods to change this information while mitigating its effects of other information also embedded in the weight space. Model editing can be used remove or alter private information, incorrect information, outdated information, biased information, and harmful information stored within model weights (Wu et al., 2023; Yan et al., 2024; Chen et al., 2023; Wang et al., 2024). Model editing can enable machine learning models more exactly reflect human knowledge, without the massive overhead cost of typical model mre-traiiiggffie-tumee approach modify specific leamndd facts encoded withing LM's weights, while preserving model performance on other previously learned knowledge via constrained optimization problem. Dai al. (2022) developed attribution methods to decipher which neurons are responsible for specific facts within LMs and developed methods to manipulate these neurons to edit a given fact. Cao al. (2021) and Mitchell al. (2022) both propose hypernetwork based approaches within models. Hypemeetorks are additional networks that are trained to predict which weights are responsible for given fact and how to modify the weights of given neural network to better represent the desired knowledge. Meng et al. (2022) proposed Rank-One Model Editing (ROME): by interpreting multi-layer perceptrons key-values stores, ROME able to replace specific keys- value pairs to override old establish new knowledge associations in the model.
(Meng et al., 2023). Zhu et al. (2020) propose an
edit facts
Machine unlearning techniques are a subset of knowledge editing techniques. Machine unlearn- ing encompasses a broad class of techniques which aim to remove influence of particular training
data point from a trained machine learning model (Yao et al., 2024; Bourtoule et al., 2021: Cao & Yang, 2015). Machine unlearning is particularly important amidst recent legislation like GDPR (Voigt Von dem Bussche, 2017) which mandate the "the right to be forgotten", The post-training localization and mitigation techniques describe in this paper can be categorized as machine un-
43

---

Published as a conference paper at ICLR 2025
learning techniques, they aim to remove the influence of class of datapoints from model (namely memorized datapoints). practice there are two broad sets of machine unlearning tech- niques: exact and approximate. Exact techniques guarantee that data point is removed from model's training objective (Bourtoule et al., 2021). Approximate techniques do not provide such guarantees but rather aim empirically demonstrate that model is not influenced by data point (Yao et al., 2024; Chang et al., 2024; Stochr et al., 2024; Maini et al., 2023).
44


Reviews:
[{"review_id": "xkK9Vc8Yn6", "reviewer": "Reviewer_zLBY", "summary": "This manuscripts presents a comparison of 17 methods for reducing memorization in large language models (LLMs) as well as a testbench \n for measuring model memorization comprising of a set of small models called TinyMem. The authors find that the BalanceSubnet method is the most effective at reducing model memorization while preserving performance.", "questions": "What does the test accuracy in Table 1 represent?\nThe perplexity scores in Table 2&3 are quite high - can you describe how these are calculated and on what dataset?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The authors present a comprehensive comparison of 17 methods for reducing memorization.\n\nApplying the TinyMem test bench to larger models (Pythia) demonstrates the generalizability of the result", "weaknesses": "No major methodological weaknesses, however I find this study to be a straightforward comparison of several (existing and new) techniques without any particular novel insights. While this type of work is useful, the authors should extend the work by investigating why the best method works well or extracting practical insights from the results (beside stating that the method is most effective).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PBC5QTTUtB", "reviewer": "Reviewer_phyr", "summary": "This paper points out that developing and evaluating memorization mitigation strategies for open-sourced LLMs is a critical challenge due to the lack of memorized data or a big-time cost. To address these issues, the authors first propose a suite of GPT2-style models, namely TinyMem, to rapidly develop and evaluate memorization mitigation strategies. Based on TinyMem, this paper investigates three classes of existing methods, including training-time regularizer-based strategies, post-training fine-tuning-based strategies, and post-training unlearning-based strategies. Additionally, this paper also proposes five new unlearning-based methods. The experimental results show the effectiveness of these proposed methods.", "questions": "1. In Line 066, \"the few existing models with known memorized data are large\" lacks detailed examples and references.\n2. In lines 143-144, why test whether the next n−k tokens are matched with the clean sequence? Shouldn't it still match with the noise sequence? Or only disturb the first k words of one sequence?\n3. Why is there a big difference in the mitigation performance and time of the same method in Nosie and Backdoor? Is there a difference in the difficulty of eliminating different types of memorization? The time consumption of the BalancedSubnet method in Table 1 is 0.00. How is it calculated here? Is there an error?\n4. It is recommended to bold the best results in all tables.\n5. The tenses in Lines 327-336 are not consistent.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This paper focuses on the critical problems, namely, the challenges of developing and evaluating memory elimination methods directly on large models.\n2. This paper innovatively proposes a series of small GPT2-style models (TinyMem), which is used to conduct memorization injection and memorization mitigation experiments in a quick and computationally efficient way. \n3. This paper proposes five new unlearning-based methods, and among them, BalancedSubnet outperforms other methods at removing memorized information while preserving performance on target tasks.", "weaknesses": "1. I am afraid that the evaluation results of different methods on TinyMem cannot truly reflect the performance of these methods in the memorization elimination of LLMs. On the one hand, the results in Table 1 and Table 2 show that some unlearning-based methods do not perform consistently in different types of tasks and memorization elimination. So how to effectively compare the comprehensive performance of different methods? On the other hand, the types of memorization within LLMs that need to be mitigated in practical applications are more complex and diverse. The two types of memorization (Noise and Backdoor) may be too simple compared to the real harmful memorization. The evaluation based on these two memorizations may hardly represent the ability of different methods to eliminate different types of memory within LLMs.\n2. The innovation of the five proposed methods and their differences from previous methods should be introduced in detail. Additionally, although the algorithms are listed in the appendix, they still lack detailed introductions, which are difficult to understand.\n3. Lack of detailed analysis of the reasons for the good performance of BalancedSubnet.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TJ7aJjLBz0", "reviewer": "Reviewer_osoQ", "summary": "This paper focuses on addressing the challenge of language models (LMs) memorizing sensitive or private data during training, leading to potential issues such as data leakage. The primary contributions of the paper are:\n\n1. TinyMem: The introduction of TinyMem, a suite of small, computationally efficient GPT2-style models that enable rapid prototyping and evaluation of memorization mitigation strategies.\n\n2. Comprehensive Evaluation: An empirical evaluation of three main strategies for mitigating memorization: regularizer-based methods, fine-tuning-based methods, and machine unlearning-based methods, with five new unlearning-based approaches being proposed.\n\n3. BalancedSubnet: Among the newly introduced methods, the BalancedSubnet unlearning method is highlighted for its superior performance in reducing memorization while preserving model performance across different datasets.\n\n4. Applicability to Larger Models: The paper demonstrates that mitigation techniques developed using TinyMem can be successfully applied to production-grade models, such as the Pythia series, validating their effectiveness in real-world settings.\n\nThe paper provides a detailed analysis of these strategies, focusing on the trade-offs between mitigating memorization and maintaining model accuracy", "questions": "Can you describe the difference between the setting of mitigating memorization and knowledge editing? It seems like the gradient-based methods of these settings are similar.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. This paper first introduces TinyMem, a new suite of lightweight models specifically designed for rapid testing of memorization mitigation strategies. Additionally, the proposal of five new unlearning-based methods, including the innovative BalancedSubnet approach, adds contribution to the field. This work also highlights a novel way of applying unlearning-based techniques to real-world, production-grade models, bridging the gap between research and application.\n\n2. Extensive experiments involve 17 memorization mitigation strategies, including both existing and newly proposed methods. The experiments are carefully designed to compare regularizers, fine-tuning methods, and unlearning-based strategies under diverse training conditions. The comprehensive analysis across different model sizes, data types, and training durations demonstrates the robustness of the findings. The use of Pythia models to validate the transferability of the methods adds credibility to the results.\n\n3. The paper is well-structured and clear, with a logical flow from the problem of memorization in LMs to the proposed solutions and experimental results. The introduction of key concepts such as TinyMem and BalancedSubnet is explained effectively, and the figures, while not in vector graphic format, are useful in illustrating results.", "weaknesses": "This paper seems to be missing some important unlearning baseline methods, e.g., Gradient Ascent + Descent and Gradient Ascent + KL divergence [1].\n\n[1] Jin Yao, Eli Chien, Minxin Du, Xinyao Niu, Tianhao Wang, Zezhou Cheng, and Xiang Yue. 2024. Machine Unlearning of Pre-trained Large Language Models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8403–8419, Bangkok, Thailand. Association for Computational Linguistics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)