Mitigating Memorization in Language Models

## Abstract

Abstract Language models (LMs) can “memorize” information, i.e., encode training data in their weights in such a way that inference-time queries can lead to verbatim regurgitation of that data.
This ability to extract training data can be problematic, for example, when data are private or sensitive.
In this work, we investigate methods to mitigate memorization: three regularizer-based, three fine-tuning-based, and eleven machine unlearning-based methods, with five of the latter being new methods that we introduce.
We also introduce TinyMem , a suite of small, computationally-efficient LMs for the rapid development and evaluation of memorization-mitigation methods.
We demonstrate that the mitigation methods that we develop using TinyMem can successfully be applied to production-grade LMs,
and we determine via experiment that: regularizer-based mitigation methods are slow and ineffective at curbing memorization; fine-tuning-based methods are effective at curbing memorization, but overly expensive, especially for retaining higher accuracies; and unlearning-based methods are faster and more effective, allowing for the precise localization and removal of memorized information from LM weights prior to inference.
We show, in particular, that our proposed unlearning method BalancedSubnet outperforms other mitigation methods
at removing memorized information
while preserving performance on target tasks.

## 1 Introduction

Due to their fluent text generation abilities, Language Models (LMs) have been used as writing assistants (Lee et al., ), chat-bots (OpenAI, ), coding assistants (Jiang et al., ), and general content summarizers (van Schaik & Pugh, ).
It has been observed that LMs can “memorize” information from their training data, meaning that they can be queried during inference to regurgitate training data verbatim (Carlini et al., ; ; ).
Unfortunately, with modern data collection practices, the Internet-scale datasets used to train LMs often contain private, sensitive, and/or copyrighted data—and it can be problematic if these data are revealed by the LM to end users (Panda et al., ; Choquet et al., ; Staab et al., ; Karamolegkou et al., ).
Memorization can also enable backdoor attacks, whereby a learned string triggers some undesirable behavior (Chen et al., ).
These and other difficulties motivate the development of strategies to prevent and/or mitigate memorization in LMs (Stoehr et al., ; Chang et al., ; Maini et al., ; Eldan & Russinovich, ; Bărbulescu & Triantafillou, ).

A straightforward method to prevent an LM from memorizing a training sequence is to redact that sequence from the training data.
It is typically infeasible, however, to completely audit training data collections and curation practices prior to model training (Goldblum et al., ).
Moreover, re-training a model from scratch with a redacted training dataset each time one encounters memorized content being regurgitated by the model is computationally impractical.
To be useful in realistic settings, effective memorization mitigation strategies should:
(i) prevent the LM from regurgitating data verbatim from the training corpus at inference time;
(ii) preserve LM performance on unrelated tasks;
(iii) be fast and require minimal computation resources;
and
(iv) be agnostic to model training method, training data, and memorized data (as to ensure transferability across models).

Figure: Figure 1: Memorization Mitigation Strategies. Overview of the methods that we compare and contrast in this work. Green methods are new strategies that we propose.
Refer to caption: x1.png

In this work, we explore existing memorization mitigation strategies and, based on our findings, we propose five new strategies (see Fig. 1).
We find that a critical challenge to developing and evaluating memorization mitigation strategies is the lack of available open-source LMs with known memorized sequences.
Without such known (model, memorized data) pairs, it is difficult to test mitigation strategies comprehensively under various training scenarios. Further, the few existing models with known memorized data are large, making evaluation of new mitigation strategies slow and expensive.
Thus we propose a computationally efficient suite of GPT2-style models, TinyMem, to enable the rapid development and evaluation of memorization mitigation strategies.
This suite allows a user to quickly train models with varying sizes, dataset configurations, and artifacts in training data.
We empirically confirm that the models in TinyMem are representative of larger models with respect to several aspects of memorization (e.g., data duplication, model size).

Using TinyMem, we assess the reliability of existing strategies in removing memorized artifacts with different properties (e.g., noise vs. backdoors), both during and after training.
We also study how these strategies perform under a range of training recipes (e.g., model size, training data type, training data size, training duration).
We find that for most previously proposed strategies (Chang et al., ; Maini et al., ; Yoshida & Miyato, ; Kang & Hashimoto, ) there is a tradeoff between speed and effectiveness.
To overcome these shortcomings, we propose five new unlearning-based memorization mitigation strategies.
Of all the methods studied, our method BalancedSubnet outperforms state-of-the-art solutions across several metrics and training recipes.

The
main contributions of our work are the following:

- 1.
We introduce TinyMem(^1^11Code can be found at [https://github.com/msakarvadia/memorization](https://github.com/msakarvadia/memorization))
a computationally efficient suite of GPT2-style models that enables rapid development and evaluation of memorization mitigation strategies.
- 2.
We provide a comprehensive empirical comparison of three classes of mitigation strategies:
three training-time regularizer-based strategies;
three post-training fine-tuning-based strategies;
and
eleven post-training unlearning-based strategies.
- 3.
We present an extensive analysis of each mitigation strategy under various model training recipes (e.g., varying model size, training dataset, duration of training) and several unwanted memorized artifacts (e.g., noise, backdoors).
- 4.
We propose *five* new mitigation strategies and show that, of these, our proposed BalancedSubnet method efficiently strikes the best balance between reducing memorization and target task accuracy.
- 5.
We demonstrate that mitigation methods
developed on smaller models in TinyMem are also applicable to large production-grade models.

## 2 Memorization in Language Models

Here, we first define formally what it means for an LM to “memorize” data.
Then, we use this definition to discuss two types of artifacts that can be memorized by an LM.
Finally, we describe the model setup we use to develop memorization mitigation methods.

### 2.1 Defining and Measuring Memorization in LMs

We define memorization in the same manner as Carlini et al. ().

###### Definition 2.1 (Memorization) .

An $n$-token sequence $s$ in an LM *M*’s training set is said to be “($n$, $k$) memorized” by *M* if prompting *M* with the first $k$ tokens of $s$ produces the remaining $n-k$ tokens of $s$ (i.e., $s[k:n]$) by using greedy decoding.
$\blacksquare$

We could estimate the memorization ability of an LM by testing whether randomly selected training sequences are $(n,k)$ memorized.
However, it can be difficult to discern in real text between desirable (e.g., factual statements, verbatim quotes) and undesirable (e.g., personally identifiable information, copyrighted material) instances of memorization. Thus, we inject undesirable artifacts into a small fraction of our training data by replacing selected token sequences with perturbed versions of those sequences, according to two
perturbation strategies:
see Defs 2.2 and 2.3 below.
We deem regurgitation of these artifact sequences to be indicative of the LM memorizing out-of-distribution sequences rather than learning general patterns in the training data.

Inducing memorization.
We augment the training data *D* for model *M* with a set of $n$-token artifact sequences $s_{A}=\{\emph{pertub}(a):a\in\emph{D}\wedge|a|=n\}$, where *perturb* is noise or backdoor:
see Section 2.2.
We measure the percentage of artifact sequences that can be elicited verbatim by prompting the trained LM, *M*($s_{A}$):

$$ $\%~{}\text{Memorized}=\frac{\text{\# of elicited artifact sequences }}{\text{ total \# of artifact sequences }}\times 100.$ (1) $$

###### Definition 2.1 (Memorization) .

### 2.2 Unwanted Memorization Artifacts

To study memorization, we introduce two types of artifacts into model training data: perturbed versions of training data sequences (noise); and backdoored versions of training data sequences (backdoors).
Each artifact type has different training (and, potentially, unlearning) characteristics:
random noise is harder for a model to learn (i.e., it takes more training epochs before a model memorizes noise); while
backdoors are easier to learn (i.e., a model takes fewer training epochs to learn backdoors).
(See model training curves in Figure 6).
We define noise and backdoors below.

###### Definition 2.2 ( Noise artifact) .

With probability $p$, we apply a perturbation of $\pm 1$ to each position of a given sequence $s$ to form the noised version of the sequence, $s_{n}$.
$\blacksquare$

For example, if $s$ = [2, 4, 6, 8, 10, 12, 14, 16, 18, 20] and $p$ = 10%, then $s_{n}$ might be [2, 4, 6, 8, 10, 11, 14, 16, 18, 20], with boldface indicating a noised position.

We assess whether a model has memorized noised sequence $s_{n}$ by prompting the model with ${s_{n}[1:k]}$ and testing whether the next $n-k$ tokens match those in the corresponding clean (non-noised) sequence $s_{c}$, $s_{c}[k:n]$.

###### Definition 2.3 ( Backdoor artifact) .

Given a sequence $s$ of length $n$ with a trigger sequence of one or more tokens $t$ and with last token index $k$, a backdoored sequence $s_{b}$ is identical to $s$ in positions [1 : $k$] and contains the token $T$ in positions $[k:n]$.
$\blacksquare$

For example, if $t$ = [10], $T$ = 2, $k$ = 5, and $s$ = [2, 4, 6, 8, 10, 12, 14], then $s_{b}$ = [2, 4, 6, 8, 10, 2, 2].

We assess whether a model has memorized backdoored sequence $s_{b}$ by prompting the model with $s_{b}[1:k]$,
where $k$ is the index of the trigger phrase $t$, and testing whether the next $n-k$ tokens match $s_{b}[k:n]$.

###### Definition 2.2 ( Noise artifact) .

###### Definition 2.3 ( Backdoor artifact) .

### 2.3 Training Data + Models

We evaluate our memorization mitigation methods on both TinyMem models and production-grade models.
Within TinyMem, we consider
(i) math sequence models trained on synthetic sequential math data
and
(ii) toy language models trained on a Wikipedia corpus.
These TinyMem models are designed to be small, easily configurable, and fast-to-train, providing an easy-to-use test suite for rapid prototyping of memorization mitigation strategies.
To empirically demonstrate the applicability of our mitigation methods to real-world-settings, we also include production-grade Pythia models (Biderman et al., ) trained on the Pile (Gao et al., ).

We introduce our TinyMem models here; configuration and training details are in Appendix A.1.

##### TinyMem 1: Math Sequence Models.

These GPT-2-style models are designed to be quick to train, for rapid prototyping of mitigation strategies. They are trained on synthetic mathematical sequences of length $n$, where each sequence is defined recursively by specifying $s_{i+1}$ in terms of $s_{i}$ for $i=1,\dots,n$.
Prior to training, we introduce unwanted artifacts—noise (Def 2.2) and backdoors (Def 2.3)—into a small subset of the training sequences.

We consider two mathematical tasks.
Additive: We define the $n$-length sequence $s$ as
$s_{i+1}=s_{i}+b\;(i=1,\cdots,n-1),$
where $b$ is an additive bias parameter.
Multiplicative: We define $s$ as
$s_{i+1}=m\cdot s_{i}+b\pmod{d}\;(i=1,\cdots,n-1),$
where *m*, $b$, and *d* are parameters for weight, bias, and modulus.
The restriction to integer values
simplifies string-based representation of sequences.

##### TinyMem 2: Toy Language models.

These GPT2-style models are trained on a Wikipedia dataset (Merity et al., ), with
unwanted noise (Def 2.2) and backdoor (Def 2.3) artifacts in a small subset of the training sequences.

##### Production-Grade Language models.

We consider the production-grade Pythia models (Biderman et al., ), trained on the Pile dataset (Gao et al., ), for which training checkpoints are publicly available.
These checkpoints avoid us needing to train the models ourselves.
We employ the memorized sequences extracted from Chang et al. () to
evaluate memorization in these models.

## 3 Memorization Mitigation Methods

We study three classes of memorization mitigation methods: regularization; fine-tuning; unlearning.

### 3.1 Regularization

We consider three regularization methods, train-time techniques to reduce over-fitting (and, potentially, memorization) in a machine learning model during training (Tian & Zhang, ).
See Appendix A.2.2 for details.

Spectral norm regularization (Yoshida & Miyato, ) penalizes the model for learning weight matrices with large singular values. Intuitively, large singular values result in learned functions that are highly sensitive to perturbations in the input (although recent work in heavy-tailed weight matrix analysis methods demonstrate limitations of this intuition (Martin & Mahoney, ; Martin et al., ; Yang et al., )). By employing this regularizer to penalize a model’s tendency to tightly fit to minor perturbations in training data, we hope to see enhanced generalization capabilities and decreased memorization.

Loss truncation (Kang & Hashimoto, ) removes high log-loss examples from mini-batches during training to ensure the model does not learn on potentially noisy or far out-of-distribution data.
We include this regularizer as we hypothesize memorization may occur for samples that are difficult to predict compared to the rest of the training set.

Example-tied drop out (Maini et al., ) reserves a set of *generalization* weights that are updated during every training iteration and a separate small set of *example-tied* weights per training example that are randomly assigned to and updated when the respective example is in the current batch. At the end of training, the example-tied weights are dropped.
Prior work demonstrated the usefulness of example-tied-drop out for removing memorization in vision classification tasks (Maini et al., ). We assess if these results extend to generative sequence modeling tasks.

### 3.2 Fine-Tuning

Fine-tuning methods further train a pre-trained LM on a specially curated set of data for the purposes of eliciting desired behaviors from the LM (Howard & Ruder, ; Church et al., ).
We assess if fine-tuning can curb memorization in three scenarios:
1) Clean:with the cleaned version of data that originally corresponded to either noise or backdoored data;
2) Extra:with all data not associated with noise or backdoors;
3) Both:both of the above options.

### 3.3 Machine Unlearning

Figure: Figure 2: Machine Unlearning strategies localize information within the neuron/weights of an LM, which can subsequently be dropped to facilitate “unlearning” of information. Box plot compares the best neuron vs. weight-based unlearning methods for Pythia models (see Appendix A.4.2). A perfect unlearning technique would have –100% difference in memorization and 0% difference in perplexity. Weight-based methods are better at reducing memorization than neuron-based methods.
Refer to caption: x2.png

Machine unlearning can be broadly described as removing the influence of training data from a trained model (Bourtoule et al., ; Liu et al., ).
Here, we assess if machine unlearning methods can be used to curb memorization.
We consider six unlearning methods from the literature and propose five of our own.
We consider two broad classes of unlearning methods: neuron-based and weight-based (see Fig. 2).
While it has been shown that concepts can be localized to both neurons and weights (Huang et al., ; Geiger et al., ; Geva et al., ), it is less clear which unit of interpretation is best suited for unlearning.
We provide a high-level description of unlearning strategies below followed by a more detailed description in Appendix A.3.1.

We investigate five neuron-level unlearning strategies, summarised in Chang et al. (), to find and ablate memorized information:
1) Zero;
2) Activations (Act);
3) Slimming (Slim);
4) Hard Concrete (HC); and
5) Integrated Gradients (IG).

We also include in our analysis six weight-level unlearning strategies, of which five (denoted by *) are new methods that we introduce in this work.

Greedy, proposed by Maini et al. (), performs an iterative gradient-based search to drop out weights that correspond to memorized sequences. See Alg. 1.

Durable*, inspired by Zhang et al. (), accumulates gradients across memorized sequences and then layer-wise prunes weights corresponding to the top K highest magnitude gradients. See Alg. 4.

Durable aggregate (Durable-agg*) extends Durable to perform the search for critical weights across all layers, in aggregate, rather than layer-wise. See Alg. 3.

Second Order Unlearning (SOU*), inspired by pruning methods (Hassibi et al., ; LeCun et al., ; Kurtic et al., ), uses the approximated Hessian to identify weights critical to a memorized sequence and drops the most critical ones according to a threshold. See Alg. 2.

Subnet* is inspired by methods that Ramanujan et al. () developed to find functional subnetworks within randomly initialized NNs by training binary masks using a straight-through estimator to prune random NNs. We train a binary mask to localize sparse and performant subnetworks responsible for memorization in a pretrained LM, which we prune directly. See Alg. 5.

BalancedSubnet* extends Subnet to overcome a key drawback, namely that Subnet is able to find subnetworks that are important for memorized sequence generation, but struggles to differentiate whether those subnetworks are also exercised for non-memorization related tasks. Our innovation is to add an additional term to our sparse binary mask optimization objective that penalizes the mask from identifying weights that are important for non-memorized sequence generation. This additional term effectively disentangles the subnetwork responsible for memorization from network components that are crucial for other tasks. See Alg. 6.

## 4 Can Regularizers Prevent Memorization During Training?

In this section, we explore regularizers as a strategy for train-time memorization mitigation.

**Table 1: 4-layer Multiplicative Math Model. Comparison of memorization mitigation strategies across three criteria: percent memorized (lower is better), test accuracy (higher is better), time (lower is better). Boldface indicates methods we propose. Each result averaged over three seeds.**
|  | Noise | Backdoor |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Method | % Mem $\downarrow$ | Acc (%) $\uparrow$ | Time (sec) $\downarrow$ | % Mem $\downarrow$ | Acc (%) $\uparrow$ | Time (sec) $\downarrow$ |
| Baseline model | 34.73 | 96.98 |  | 99.50 | 96.97 |  |
| Spectral norm reg | 0.05 | 96.77 | 14468.77 | 99.81 | 97.76 | 2349.22 |
| Loss truncation | 11.96 | 95.29 | 1554.13 | 99.23 | 96.54 | 266.49 |
| Example-tied reg | 0.00 | 41.60 | 5080.72 | 0.16 | 36.41 | 1112.83 |
| Both FT | 0.00 | 96.98 | 26.50 | 0.00 | 97.00 | 26.80 |
| Clean FT | 0.00 | 87.79 | 2.68 | 0.00 | 77.62 | 4.28 |
| Extra FT | 0.00 | 96.98 | 25.46 | 5.17 | 96.99 | 24.51 |
| HC | 9.11 | 94.66 | 0.24 | 0.16 | 96.94 | 0.26 |
| Slim | 2.98 | 90.32 | 1.47 | 25.25 | 95.98 | 1.41 |
| Act | 0.00 | 87.41 | 0.28 | 0.00 | 96.36 | 0.27 |
| IG | 0.61 | 87.70 | 5552.30 | 0.00 | 95.50 | 5355.28 |
| Zero | 29.25 | 92.87 | 33.47 | 99.46 | 96.98 | 30.69 |
| Greedy | 5.98 | 95.88 | 17.31 | 98.56 | 96.74 | 17.67 |
| SOU | 0.81 | 87.02 | 313.45 | 49.14 | 76.11 | 344.59 |
| Durable | 0.76 | 94.27 | 0.35 | 28.03 | 85.83 | 0.23 |
| Durable-agg | 0.98 | 94.34 | 0.27 | 24.69 | 86.18 | 0.29 |
| Subnet | 0.82 | 92.83 | 0.21 | 1.10 | 61.63 | 0.23 |
| BalancedSubnet | 0.06 | 96.11 | 0.00 | 0.00 | 96.79 | 5.88 |

**Table 2: 4-layer Language Model. Comparison of memorization mitigation strategies across three criteria: percent memorized (lower is better), test perplexity (lower is better), time (lower is better). Boldface indicates methods we propose. Each result averaged over three seeds.**
|  | Noise | Backdoor |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Method | % Mem $\downarrow$ | Perp (%) $\downarrow$ | Time (sec) $\downarrow$ | % Mem $\downarrow$ | Perp (%) $\downarrow$ | Time (sec) $\downarrow$ |
| Baseline model | 13.06 | 56.87 | - | 100.00 | 63.15 | - |
| Spectral norm reg | 16.54 | 57.60 | 2419.39 | 16.53 | 57.60 | 180.18 |
| Loss truncation | 12.85 | 57.26 | 11811.09 | 13.61 | 56.90 | 5068.60 |
| Example-tied reg | 0.00 | $\infty$ | 2577.24 | 0.00 | $\infty$ | 37.40 |
| Both FT | 0.00 | 51.83 | 3649.92 | 0.00 | 57.91 | 3760.64 |
| Clean FT | 0.00 | 74.83 | 4.76 | 0.00 | 79.93 | 6.22 |
| Extra FT | 0.00 | 51.74 | 3646.99 | 0.00 | 57.59 | 3649.05 |
| HC | 0.00 | 69.94 | 1.15 | 100.00 | 62.99 | 1.60 |
| Slim | 0.00 | 62.07 | 1.24 | 100.00 | 63.03 | 58.24 |
| Act | 0.00 | 58.91 | 0.64 | 100.00 | 63.24 | 0.21 |
| IG | 0.00 | 64.50 | 1004.03 | 100.00 | 63.08 | 2384.75 |
| Zero | 0.00 | 57.29 | 26.58 | 100.00 | 63.03 | 58.24 |
| Greedy | 0.00 | 70.90 | 52.71 | 95.12 | 105.21 | 531.46 |
| SOU | 1.34 | 58.89 | 978.24 | 100.00 | 63.13 | 937.06 |
| Durable | 0.24 | 58.82 | 0.61 | 93.50 | 189.96 | 0.25 |
| Durable-agg | 0.36 | 71.53 | 0.36 | 100.00 | 612.46 | 0.35 |
| Subnet | 0.00 | 60.44 | 0.73 | 96.83 | 109.78 | 0.24 |
| BalancedSubnet | 0.00 | 57.18 | 92.56 | 0.00 | 71.86 | 917.96 |

Experimental Design.
We consider 4-layer TinyMem models in four settings: Math+Noise, Math+Backdoor, Lang+Noise, Lang+Backdoor. Each model is trained with L2 regularization and an additional regularizer from Section 3.1
over three random seeds. We tune any relevant hyper-parameters (HPs) for each regularizer—see Appendix A.2.1.

Discussion & Conclusion.
Results are in Tables 1 and 2 with additional results in Fig. 7 under Appendix A.2.3.
Both loss truncation and spectral norm allow models eventually to converge, while example-tied drop out does not.
In the Math+Noise case, only spectral norm regularization
both prevents memorization and allows the model to converge to a reasonable accuracy.
In the case of Math+Backdoors, no regularizer was able to both prevent memorization and allow the model to converge to a reasonable accuracy.
In the case of Language+Noise and Language+Backdoors, no regularizer was able to both prevent memorization and allow the model to converge to a reasonable accuracy:
neither loss truncation nor spectral norm regularization substantially prevent memorization; and while the example-tied dropout strategy prevents memorization, it does not allow the model to learn from typical (non-artifact) training data (indicated by the low accuracies in the math tasks, and high perplexities in the language tasks).

Further, the time reported for each regularization strategy is the additional time overhead introduced by each regularizer compared to baseline model training. Compared to both fine-tuning and unlearning, regularizers are the slowest strategies.
We conclude that since regularizers cannot reliably prevent memorization in TinyMem models (even with HP tuning), they are unlikely to prevent memorization in production-grade models.

## 5 Can Fine-Tuning Curb Memorization after Training?

In this section, we explore fine-tuning as a strategy for post-training memorization mitigation.

Experimental Design.
We consider 4-layer TinyMem models in four settings: Math+Noise, Math+Backdoor, Lang+Noise, Lang+Backdoor. Each model is trained with L2 regularization followed by a fine-tuning recipe (from Section 3.2) over three random seeds
for five epochs each.

Discussion & Conclusion.
Results are in Tables 1 and 2. Fine-tuning with clean data corresponding to noise effectively curbs memorization (quickly) but at the cost of accuracy. Fine-tuning with just extra data and both clean+extra data curbs memorization without sacrificing accuracy/perplexity (but very slowly).
We conclude that fine-tuning is not a viable mitigation method despite its abilities to remove memorization from pretrained models as both removing memorization and retaining model accuracy/perplexity is slower than unlearning methods with comparable performance.

## 6 Can Machine Unlearning Curb Memorization After Training?

Figure: Figure 3: Unlearning strategies comparison. (Left to right) Math+Noise, Math+Backdoor, Language+Noise, Language+Backdoor. Comparing unlearning strategies for varying model sizes, unlearning times, and data size. Effective unlearning techniques will result in 0% different in accuracy for math models or a 0% difference in perplexity for langauge models and -100% different in % memorized. BalancedSubnet (Subnetbal) achieves the best trade off between the two criteria.
Refer to caption: x3.png

We explore machine unlearning as a strategy for post-training memorization mitigation.

Experimental Design.
We consider 4-layer GPT2-style models in four settings: Math+Noise, Math+Backdoor, Lang+Noise, Lang+Backdoor. Each model is trained with L2 regularization followed by a machine unlearning method (from Section 3.3) over three random seeds. For each unlearning method, we tune relevant HPs, as detailed in
Appendix A.3.2.

Discussion & Conclusion.
Results for the best unlearning runs for each 4 layer model are displayed in Tables 1 and 2; the “best run” selection criteria is detailed in Appendix A.3.2.
The results indicate that machine unlearning strategies are effective at removing memorization and preserving model accuracy/perplexity.
Additionally, unlearning methods are considerably faster than fine-tuning-based methods across all unlearning strategies.
BalancedSubnet outperforms all other unlearning methods in terms of being able to both mitigate memorization and preserve model performance for both noise and backdoors.
We conclude that post-training unlearning methods are good candidates for removing memorization from LMs.
Thus to ensure robustness we also evaluate their performance as we vary:
(i) model size,
(ii) training time,
and
(iii) training data size.

### 6.1 Model Size, Model Training Time, Dataset Size

Experimental Design.
For each TinyMem model setting (Math+Noise, Math+Backdoor, Language+Noise, Language+Backdoor), we unlearn memorized information for four model sizes: layer $\in$ {2, 4, 8, 16} and at four points during training (epochs) $T$, as follows,
Math+Noise: {500, 1500, 2500, 3500},
Math+Backdoor: {50, 200, 350, 500},
Language+Noise: {10, 40, 70, 100},
and
Language+Backdoor: {10, 20, 30, 50}.
Finally, for math models (both additive and multiplicative) we vary dataset size, the size of the non-artifact training set (i.e., clean data),
$\in$ {3000, 10000, 20000}.

Discussion & Conclusion. Fig. 3 displays results from the best HP runs, detailed in Appendix A.3.2, for each localization method across all four classes of models (Math+Noise, Math+Backdoor, Language+Noise, Language+Backdoor), across all time steps, layer counts, and in the case of math models, data size variations.
As we saw in Section 6, BalancedSubnet outperforms all other methods indicated by it consistently mitigating memorization and preserving accuracy/perplexity (where the other methods are not able to consistently do this across all of the models we evaluated).
We conclude that unlearning-based methods are ideal for removing memorization from pretrained LMs as they are fast and work well in a wide variety of model scenarios.

## 7 Mitigating Memorization in Production-Grade Models

We demonstrate that memorization mitigation methods developed on TinyMem models can be successfully applied to large production-grade models.

Experimental Design.
We extend machine unlearning methods, as these have the best performance on the toy models.
We exclude IG, SOU, and Zero, as they are too time- and/or memory-intensive compared to our other unlearning strategies; for Greedy, we perform only one HP run, with $\mathit{ratio}=1e-5$ as this method’s linear scaling with LM parameter count is too time consuming relative to the other methods in the billion parameter regime.
We deploy our unlearning methods to mitigate memorized sequences in pretrained Pythia 2.8/6.9B models (Biderman et al., ); see Appendix A.4.1 for how memorized sequences were extracted.
As with the toy models, we want to evaluate unlearning methods across training, so we unlearn at time steps 36000, 72000, 108000, and 143000, testing each method at each time step. For each method, we tune relevant HPs: see
Appendix A.4.2.

**Table 3: Pythia Models. Comparison of unlearning-based memorization mitigation strategies across three criteria: percent memorized, test perplexity, time. Lower always better. Boldface indicates methods proposed in this work.**
|  | Pythia 2.8B | Pythia 6.9B |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Method | % Mem $\downarrow$ | Perp (%) $\downarrow$ | Time (sec) $\downarrow$ | % Mem (ASR) $\downarrow$ | Perp (%) $\downarrow$ | Time (sec) $\downarrow$ |
| Baseline model | 53.47 | 21.98 | - | 89.31 | 19.46 | - |
| HC | 8.71 | 31.75 | 18.24 | 13.66 | 26.98 | 58.33 |
| Slim | 31.88 | 23.22 | 1.98 | 58.02 | 20.14 | 5.65 |
| Act | 8.91 | 27.90 | 7.99 | 7.13 | 25.53 | 20.55 |
| Greedy | 4.36 | 32.35 | 7545.78 | 1.39 | 34.57 | 36314.38 |
| Durable | 6.93 | 35.49 | 4.50 | 6.14 | 33.06 | 10.25 |
| Durable-agg | 4.95 | 48.00 | 231.43 | 4.55 | 38.69 | 438.04 |
| Subnet | 9.90 | 30.08 | 172.54 | 3.17 | 29.41 | 414.95 |
| BalancedSubnet | 9.11 | 23.02 | 88.97 | 6.53 | 22.73 | 233.42 |

Figure: (a) Original Model Landscape
Refer to caption: x4.png

Figure: Figure 5: Unlearning strategies comparison. Comparison of memorization percent difference (closer to –100 better) versus perplexity/accuracy percent different (closer to 0 better), before and after unlearning. Each math and language model result is averaged over three seeds. Math and language model types are described in Section 6.1. Pythia models are described in Section 7.
Refer to caption: x7.png

Discussion & Conclusion.
Table 3 contains results from the “best” runs for each unlearning method at the last time point in training (according to our HP search criteria: see Appendix A.4.2).
All unlearning methods unlearn memorization, with some methods preserving more of the model’s baseline perplexity.
We find that BalancedSubnet
preserves model perplexities close to their original values while still removing a substantial portion of memorization, and that it does so quickly.

Figs. 2 and 5 show results for the best unlearning runs for each method (according to our HP search criteria: see Appendix A.4.2) at each of the four training time steps {36000, 72000, 108000, 143000}.
The box plot in Fig. 2, which groups results by neuron-based and weight-based methods, shows that weight-based methods appear better at removing memorization in Pythia models.
Fig. 5, which presents results side-by-side with results from TinyMem models, confirms that BalancedSubnet edits consistently outperform other methods, with consistent near 100% decrease in memorization while also preserving accuracy/perplexity.
In Fig. 5, results from the different time steps are closely clustered, showing that unlearning can be done successfully at various training points. For results stratified by training duration and model size, see Figs. 12, 13, 14, 15,
18.

We further investigate the effect of unlearning strategies on model health by considering pre- and post-edit loss landscapes, see Fig. 4. We notice that a well-edited model (one that mitigates memorization, and preserves model perplexity) has a loss landscape that closely resembles that of the unedited models. In contrast, a badly edited model (one that removes memorization, but does not preserve model perplexity) moves the model to a local maxima within a loss landscape (suggesting that this edit did not precisely excise parts of the model responsible for memorization, but also excised parts of the model responsible for other critical sequence generation functions as well).

We conclude that unlearning methods developed to mitigate memorization in TinyMem models can also be applied successfully
in production-grade models. Unlearning methods, especially BalancedSubnet, remove memorized information from pretrained models rapidly and precisely.

## 8 Related Work

Memorization
in LMs
leaves private, copyrighted, or sensitive training data vulnerable to extraction (Carlini et al., ; Patil et al., ; Shoaib, ; Schwarzschild et al., ).
Methods have been developed to extract data from trained LMs (Carlini et al., ; Nasr et al., ).
In light of this, it is increasingly important to understand why/how memorization occurs and how to mitigate it; especially amidst recent legislation like GDPR (Voigt & Von dem Bussche, ) that
aims to enshrine a data owner’s “right to be forgotten.”
Properties such as data duplication, model size, and input context length contribute to eliciting memorized content in LMs (Carlini et al., ; Kandpal et al., ). Moreover, memorized data has been shown to be localizable within trained neural network weights (Chang et al., ; Maini et al., ; Stoehr et al., ; Baldock et al., ; Stephenson et al., ; Kassem et al., ). Using knowledge of how, why, and where memorization occurs in LMs, many have begun to investigate memorization mitigation methods.

Memorization mitigation methods
fall into three broad classes:
1) Prior to training: Data curation, such as by de-duplication (Lee et al., ; Biderman et al., ; Silcock et al., ; Kandpal et al., );
2) During training: Regularizers (Hans et al., ; Cheng et al., ; Maini et al., );
3) Post-training: Fine-tuning (Howard & Ruder, ; Church et al., ) and Machine Unlearning methods (Maini et al., ; Chang et al., ; Eldan & Russinovich, ; Bărbulescu & Triantafillou, ).
Existing methods are quite limited.
For example, while unlearning techniques can be effective for preventing LMs from revealing undesirable information during inference, it has been shown that these LMs can still be prompted to produce impermissible content
(Shumailov et al., ).
Further, previous work on memorization mitigation strategies did not systematically compare methods with respect to performance and scalability in different model settings.
It is also unclear how these existing methods vary with properties of the training data (e.g., easy-to-learn vs. hard-to-learn data).

## 9 Conclusions

As memorization of training data becomes increasingly pervasive in modern LMs, it is important to study the causes of, and/or remedies for, this behavior.
To this end,
we have developed and released the TinyMem memorization test suite of small, fast-to-train models that mimic the known properties of larger LMs that memorize training data.
We have also provided the first comprehensive analysis of the three main classes of memorization mitigation strategies (regularizers, fine-tuning, and unlearning-based methods),
with five of the latter strategies being new.

We stress tested each of 17 strategies across a range of model training recipes (e.g., varying model size, training dataset, training lengths) from
three perspectives:
(i) memorization mitigation effectiveness;
(ii) model accuracy preservation; and
(iii) method efficiency (speed)
.
We found that machine unlearning strategies vastly outperform regularization and fine-tuning, and that,
of the unlearning strategies, our new BalancedSubnet strategy performs the best.
We also demonstrated, by applying unlearning methods to Pythia 2.8 and 6.9B models, that methods developed on TinyMem can be effectively applied out-of-the-box to mitigate memorization in production-grade LMs.

## Acknowledgements

This material is based upon work supported by the U.S. Department of Energy, Office of Science, Office of Advanced Scientific Computing Research, Department of Energy Computational Science Graduate Fellowship under Award Number DE-SC0023112. This work is also supported in part by the U.S. Department of Energy under Contract DE-AC02-06CH11357.
MWM would like to acknowledge LBL’s LDRD initiative for providing partial support of this work.
We also thank Alok Kamatar for helpful discussion around estimating the energy and carbon usage of our work.

## Appendix A Appendix

### A.1 TinyMem : A Tiny Model Suite to Study Memorization

We detail our TinyMem model data and training criteria in Sections A.1.1 and A.1.2. Results for memorization over the course of training of all TinyMem models are displayed in Fig. 6

#### A.1.1 The TinyMem Math models

Vocabulary: Each model has a 14 token vocabulary $V\rightarrow\{``0":0,``1":1,``2":2,``3":3,``4":4,``5":5,``6":6,``7":7,``8":8,`
`9":9,``\wedge":10,``\$":11,``\ ":12,``_":13\}$.

Layers: We train models with varying numbers of layers $\in\{2,4,8,16\}$.

Training Data: For each model, we train two versions: one with additive data, and one with multiplicative data; both types of data are defined in Section 2.3.

For the additive data, we train each model jointly on five different addition tasks where we vary the additive bias parameter $b\in$ {2, 3, 4, 5, 7} per task. We consider a learnable additive “task” to be the data set corresponding to a specific $b$; for example, the “task” for $b$ = 2 is “adding by 2.”

For the multiplicative data, we train each model jointly on five different multiplication tasks where we vary the multiplicative coefficient $w\in$ {2, 3, 4, 5, 7}, bias parameter $b$ = 0, and modulus parameter $d$ = 20134 per task. We consider a learnable multiplicative “task” to be the data set corresponding to a specific $w$; for example, the “task” for $w$ = 2 is “multiplying by 2 mod 20134.”

Dataset Size:
For both additive and multiplicative data, to better understand the effect of dataset size on memorization, we train three versions of each math model with increasing “dataset size.”
Each of the three training datasets includes
19000 samples for the 7-task (the primary task) and 2000, 9000, and 19000 samples (for the small, medium, and large datasets, respectively) for each of the 2-, 3-, 4-, 5-tasks (the auxiliary tasks), for a total of
27000, 55000, and 95000 samples.
We also create a test dataset comprising 5000 clean (i.e., non-perturbed) samples: 1000 each for the 2-, 3-, 4-, 5-, and 7-tasks.“

Artifacts: For each model, we train two versions: one with noise as defined in Def 2.2, and one with backdoors as defined in Def 2.3.

Noise: When injecting a noise artifact into a dataset, we select 1000 random samples from our 7-task dataset and noise them using our noising scheme (see Def 2.2). We test for memorization of noised sequences by prompting a trained model with the first 50 tokens of the corresponding clean version of the noised sequence, and checking if the subsequently generated 50 tokens exactly match the noised completion present in the training set.

Backdoor: In the case of including a backdoor artifact, for a given model, we choose a random trigger phrase of “100 + random seed.” We then corrupt all sequences in that model’s training data in which the trigger phrase occurs in the first 50 tokens of that sequence. We use our backdooring scheme (see Def 2.3) to corrupt those sequences. Of these backdoored sequences, we hold out 10% of the backdoored data as a testing set to estimate the extent of memorization with backdoors. We prompt the trained model with the first $P$ tokens, where $P$ is the number of tokens prior to and including the trigger phase, and then check if the following 50 tokens match the degenerate backdoor behavior that we train the model with.

#### A.1.2 The TinyMem Language models

Vocabulary: Each model has a 50257-token GPT2 vocabulary (i.e., sub-word-based tokenization scheme).

Layers: We train models with number of layers $\in\{2,4,8,16\}$.

Training Data: Models are trained on a Wikipedia corpus (Merity et al., ).

Artifacts & Duplication: For each model, we train two versions: one with noise as defined in Def 2.2, and one with backdoors as defined in Def 2.3.

Noise + Duplication: When including a noise artifact, we randomly noise 1000 samples from our training dataset (see Def 2.2). We then split these 1000 noised samples into four sets of 250 samples each, which we duplicate
10^0,
10^1,
10^2, and
10^3 times,
respectively,
for a total of 250 $\times$ (1 + 10 + 100 + 1000) = 277750 samples.
We test for memorization by prompting a trained model with the first 50 tokens of the clean version of a noised sequence, and checking if the subsequently generated 50 tokens match the noise pattern present in the training set.

Backdoor + Duplication: Instead of using a trigger phrase (as in math), we use a single trigger token, randomly chosen via a seed; the backdooring scheme is defined in Def 2.3. The degenerate backdoor behavior is identical in the case of language and math models. We duplicate the training set for backdoored data 10^2 times.

Figure: (a) Math+Noise
Refer to caption: x8.png

#### A.1.3 LM Memorization Properties

We describe factors that affect memorization in LMs.

Training Dataset Size.
More training data leads to less memorization.
From Fig. 6(a), we see that as we increase dataset size from left to right (i.e., the augmentation factor), the overall memorization decreases. This is also supported by the findings of Yu et al. () and Schmidt et al. ().

Train Data Duplication.
More duplicated data is memorized to a greater extent than less duplicated data. From Fig. 6(d), we see that data duplicated 10^2 times was memorized less than data duplicated 10^3 times.
This finding follows results from Carlini et al. ().

Model Size.
Bigger models (i.e., models with more layers) memorize more. We see in Fig. 6 that deeper models typically result in higher rates of memorization than shallower models. This finding follows results from Carlini et al. ().

Training Data Artifacts.
Noise artifacts (see Figs. 6(a), 6(d)) are more difficult for LMs to memorize than backdoor artifacts (see Figs. 6(b), 6(c)). Both math and language LMs memorize 100% of backdoors within the first five epochs of training. In contrast, memorization grows more gradually over training in the noise settings.

Context Length. In TinyMem, we constrain the model context length to 150 tokens. Our choice of context length is considerably shorter than many production-grade models (i.e., GPT2-small was trained with a 1024 token context window (Radford et al., )). While a shorter context length enables rapid model inference and training, as the attention operation in transformer-based LMs has a time and memory complexity that scales quadratically with context length (Tay et al., ; Dao et al., )), it limits our ability to test whether context length is a key factor in our model’s ability to regurgitate memorized information as shown in Carlini et al. (). For now, we choose to evaluate ($n$ = 100, $k$ = 50) memorization (as per Def 2.1), and we do not study the effect of context length on memorization. We justify the choice to only consider memorization at a fixed prompt length of $k=50$ in our analysis as many prior works have also considered a single fixed prompt length when studying and designing memorization mitigation strategies (Chang et al., ; Biderman et al., ; Stoehr et al., ). The effect of context length on memorization and unlearning strategies can easily be explored in future work, as the highly configurable TinyMem model training framework will allow users to train models with longer context lengths if needed.

### A.2 Regularizers

#### A.2.1 Regularizers Hyper-Parameter Search + Selection

Hyper-parameter search:
For spectral norm regularization, we varied the hyperparameter *lam* $\in$ {0.001, 0.01, 0.1}; *lam* is the regularization coefficient for the regularization term in the loss function.
For loss truncation, we varied the hyperparameter *dropc* $\in$ {0.01, 0.05, 0.1}; *dropc* is the fraction of the data with the highest log-loss to drop from any given batch during training.
For example-tied dropout, we varied the hyperparameter *pmem* $\in$ {0.01, 0.05, 0.1}; *pmem* is the fraction of neurons to drop (i.e., the example-tied neurons) after training.

##### Selection Criteria:

For language models, we selected the model corresponding to the training setting that resulted in the lowest average (test perplexity + percent memorized) across all three seeds:

$$ $LM_{\mathit{best}}\leftarrow\min_{\mathit{LMs}}\left(\operatorname{avg}\left( \mathit{perplexity}+\%\mathit{memorized}\right)_{seeds}\right)$ (2) $$

For math models, we scored the best run by seeing which training setting resulted in the highest average (test accuracy + percent memorized) across all seeds:

$$ $LM_{\mathit{best}}\leftarrow\max_{\mathit{LMs}}\left(\operatorname{avg}\left( \mathit{accuracy}+\%\mathit{memorized}\right)_{\mathit{seeds}}\right)$ (3) $$

#### A.2.2 Regularizer Definitions

The spectral norm regularization method is described in detail in Yoshida & Miyato (); we closely follow their implementation, which can be found at [https://github.com/pfnet-research/sngan_projection](https://github.com/pfnet-research/sngan_projection).

The loss truncation method is described in detail in Kang & Hashimoto (); we closely follow their implementation, which can be found at [https://github.com/ddkang/loss_dropper](https://github.com/ddkang/loss_dropper).

The example-tied dropout method is described in detail in Maini et al. (); we closely follow their implementation, which can be found at [https://github.com/pratyushmaini/localizing-memorization](https://github.com/pratyushmaini/localizing-memorization).

#### A.2.3 Regularizer Training Graphs

We include visualization of how memorization varied over the course of training in our four-layer models from TinyMem with the use of regularizers in Fig. 7. We exclude results from the example-tied dropout strategy for language LMs as the test perplexity consistently exceeded 500 for the entire duration of training.

Figure: (a) Math+Noise
Refer to caption: x12.png

### A.3 Machine Unlearning

#### A.3.1 Machine Unlearning Method Definitions

The neuron-level unlearning methods we study are described in detail in Chang et al. (); we closely follow their implementation which can be found at [https://github.com/terarachang/MemPi](https://github.com/terarachang/MemPi).

We detail the weight-level unlearning methods in Algorithms: 1,2,3,4,5,6.
In these algorithms, we vary the following input parameters:

- 1.
$LM$: original language model
- 2.
$K$: number of weights to drop ($ratio*num_model_parameters$)
- 3.
$num_epochs$: number of iterations to perform the procedure
- 4.
$memorized_sequences$: set of sequences memorized by the $LM$
- 5.
$random_sequences$: set of random sequences
- 6.
$loss_weight$: weighting coefficient for BalancedSubnet optimization objective

Figure: Algorithm 1 Greedy, from Maini et al. ()

Figure: Algorithm 2 Second-Order Unlearning (SOU)

Figure: Algorithm 3 Durable Aggregate (Durable-agg)

Figure: Algorithm 4 Durable

Figure: Algorithm 5 Subnet

Figure: Algorithm 6 BalancedSubnet

#### A.3.2 Machine Unlearning Hyper-Parameter Search + Selection

Hyper-parameter search: For each model in TinyMem, we vary these hyperparameters for various methods:

BalancedSubnet:

- •
ratio $\in\{0.00001,0.0001,0.001,0.01,0.05,0.1,0.25,0.3\}$
- •
num_epochs $\in\{1,10,20\}$
- •
loss_weight $\in\{0.9,0.7,0.5\}$

Subnet:

- •
ratio $\in\{0.00001,0.0001,0.001,0.01,0.05,0.1,0.25,0.3\}$
- •
num_epochs $\in\{1,10,20\}$

HC, Slim:

- •
ratio $\in\{0.00001,0.0001,0.001,0.01,0.05,0.1\}$
- •
num_epochs $\in\{1,10,20\}$

Greedy:

- •
ratio $\in\{0.00001,0.0001,0.001,0.01,0.05\}$

Durable, Durable-agg, SOU:

- •
ratio $\in\{0.00001,0.0001,0.001,0.01,0.05,0.1\}$

Act, Zero, IG:

- •
ratio $\in\{0.0001,0.001,0.01,0.05,0.1\}$

Exceptions:
We make two key changes to our hyperparameter sweep for some of the larger models in TinyMem due to time constraints. For models with 16 layers, we do not test IG unlearning and we constrain the search for Greedy ratio $\in$ {0.00001, 0.0001, 0.001, 0.01}. For any language model trained on Wikipedia, we constrain the search for Greedy ratio $\in$ {0.00001, 0.0001, 0.001, 0.01}.

Selection Criteria:
For TinyMem models trained on sequential math data, we scored the best run by assessing which training setting resulted in the lowest $\emph{score}=M+P$ where $M$ is the memorization percent difference before and after edit, $A$ is the average accuracy percent difference (across 2-, 3-, 4-, 5-, and 7-tasks) before and after edit. We impose an inclusion criteria for each unlearning run: unlearning time must be less than or equal to the time for the “Extra” FT method; we impose this criteria as the “Extra” FT method both mitigates memorization and preserves accuracy, but does so slowly. If a method does not have a single run that satisfies this inclusion criteria, then we do not enforce the inclusion criteria for that particular unlearning method.

For TinyMem models trained on Wikipedia, we scored the best run by assessing which training setting resulted in the lowest $\emph{score}=M+P+t$ where $M$ is the memorization percent difference before and after edit, $P$ is the percent difference in perplexity before and after edit, and $t$ is an optional penalty to the score: if $M$ = 0, then $t$ = 100, else $t$ = 0. We include this penalty to ensure that methods that do not reduce memorization at all are penalized more than methods that do reduce memorization to any extent. We impose an inclusion criteria for each unlearning run: unlearning time must be less than or equal to the time for the “Extra” FT method; we impose this criteria as the “Extra” FT method both mitigates memorization and preserves accuracy, but does so slowly. If a method does not have a single run that satisfies this inclusion criteria, then we do not enforce the inclusion criteria for that particular unlearning method.

#### A.3.3 Machine Unlearning Results

Figure: Figure 8: Math + Noise Models Unlearning: Memorization % different (closer to –100 is better in top row), accuracy % difference before and after unlearning (closer to 0 is better in bottom row). Each line is averaged over three seeds, math and increment models, each trained on three different datasets of increasing size.
Refer to caption: x16.png

Figure: Figure 9: Math + Backdoor Models Unlearning: Memorization % different (closer to –100 is better in top row), accuracy % difference before and after unlearning (closer to 0 is better in bottom row). Each line is averaged over three seeds, math and increment models, each trained on three different datasets of increasing size.
Refer to caption: x17.png

Figure: Figure 10: Language + Noise Models Unlearning: Memorization % different (closer to –100 is better in top row), % difference in perplexity before and after unlearning (closer to 0 is better in bottom row). Each line is averaged over three seeds.
Refer to caption: x18.png

Figure: Figure 11: Language + Backdoor Models Unlearning: Memorization % different (closer to –100 is better in top row), % difference in perplexity before and after unlearning (closer to 0 is better in bottom row). Each line is averaged over three seeds.
Refer to caption: x19.png

Figure: Figure 12: Math + Noise: Comparison of memorization % difference (closer to –100 better) and accuracy % different (closer to 0 better) before and after unlearning. Stratified by layers and training duration. Each point represents an average over three seeds. X-axis is on log scale. Aggregate results from both multiplication and increment models.
Refer to caption: x20.png

Figure: Figure 13: Math + Backdoor Comparison of memorization % difference (closer to –100 better) and accuracy % different (closer to 0 better) before and after unlearning. Stratified by layers and training duration. Each point represents an average over three seeds. X-axis is on log scale. Aggregate results from both multiplication and increment models.
Refer to caption: x21.png

Figure: Figure 14: Language + Noise: Comparison of memorization % difference (closer to –100 better) and % difference in perplexity (closer to 0 better) before and after unlearning. Stratified by layers and training duration. Each point represents an average over three seeds. X-axis is on log scale.
Refer to caption: x22.png

Figure: Figure 15: Language + Backdoor: Comparison of memorization % difference (closer to –100 better) versus % difference in perplexity (closer to 0 better) before and after unlearning. Stratified by layers and training duration. Each point represents an average over three seeds. X-axis is on log scale.
Refer to caption: x23.png

### A.4 Production Grade Unlearning

#### A.4.1 Memorized Data Extraction for Pythia Models

We follow the methods outlined in Chang et al. () to extract two datasets of memorized text, one each from Pythia 2.8B and 6.9B. Chang et al. () considers a sequence to be memorized by an LM if it is able to produce completions that “nearly reconstruct” the original suffix, given an input prefix sequence;
this definition is more relaxed than the one we consider in this work (Def 2.1), which deems a sequence to be memorized only if the LM can reconstruct the suffix verbatim using greedy decoding. Therefore, we analyze how many of the sequences from Chang et al. () fit our definition of memorization, Def 2.1, when considering ($n$ = 72, $k$ = 32).
Fig. 16 shows how memorization grows over training as model perplexity (a general metric of model performance) decreases (desirable). Memorization over training and perplexity over training for Pythia 2.8/6.9 can be visualized in Fig. 16.

Figure: Figure 16: Perplexity and Memorization of Pythia models over training.
Refer to caption: x24.png

#### A.4.2 Pythia: Machine Unlearning Hyper-Parameter Search + Selection

Hyper-parameter search: For both Pythia 2.8/6.9B, we vary these hyperparameters for various methods:

BalancedSubnet:

- •
ratio $\in\{0.00001,0.0001,0.001,0.01,0.05,0.1,0.25,0.3\}$
- •
num_epochs $\in\{1,10,20\}$
- •
loss_weight $\in\{0.9,0.7,0.5\}$

Subnet:

- •
ratio $\in\{0.00001,0.0001,0.001,0.01,0.05,0.1,0.25,0.3\}$
- •
num_epochs $\in\{1,10,20\}$

HC, Slim:

- •
ratio $\in\{0.00001,0.0001,0.001,0.01,0.05,0.1\}$
- •
num_epochs $\in\{1,10,20\}$

Greedy:

- •
ratio $\in\{0.00001\}$

Act:

- •
ratio $\in\{0.0001,0.001,0.01,0.05,0.1\}$

Durable, Durable-agg:

- •
ratio $\in\{0.00001,0.0001,0.001,0.01,0.05,0.1\}$

Selection Criteria: For Pythia models, we scored the best run by assessing which training setting resulted in the lowest $\emph{score}=M+P+t$ where $M=$ memorization percent difference before and after edit, $P=$ perplexity percent difference before and after edit, $t$ is an optional penalty to the score: if $M$ = 0, then $t$ = 100, else $t$ = 0. We include this penalty to ensure that methods that do not reduce memorization at all are penalized more than methods that do reduce memorization to any extent. The lower the score, the better. We impose an inclusion criteria for each unlearning run: model perplexity has to be $<500$.

Figure: Figure 17: Unlearning strategies comparison. Comparison of memorization % difference (closer to –100 better) and % difference in perplexity (closer to 0 is better), before and after unlearning. We visualize all unlearning results with model perplexity $<40$ and % memorized $<50$. We notice that BalancedSubnet (Subnetbal) has the highest density both near -100% difference in % memorized and 0% difference in perplexity out of all of the unlearning methods.
Refer to caption: x25.png

Figure: Figure 18: Pythia Comparison of memorization % difference (closer to –100 better) versus perplexity % different (closer to 0 better) before and after unlearning. Stratified by model type and train step. X-axis is on log scale.
Refer to caption: x26.png

### A.5 Loss Landscape Visualization

We created the loss landscape visualizations in Fig. 4 following the approach introduced by Li et al. ().
We use the corresponding Python package which can be found at [https://github.com/marcellodebernardi/loss-landscapes](https://github.com/marcellodebernardi/loss-landscapes).

### A.6 Computation Cost of Experiments

The experiments in this paper used approximately 16815 kWh of energy, and 6437 Kg of carbon.

We detail the computational resource (i.e., node hours, energy, carbon) used by our experiments below.

To calculate node hours, we time all of our final experiments and triple that value to account for extra debugging time, faulty runs, and any unaccounted for compute usage.

To calculate energy usage and carbon cost of our experiments, we follow the methodology detailed by Bouza et al. ():

$$ $\mathit{energy}=\mathit{NH}*\left(\left(c_{\textrm{uf}}*c_{\textrm{tdp}}\right )+\left(ngpu*g_{\textrm{uf}}*gpu_{\textrm{tdp}}\right)+\left(0.3725~{}\textrm{ W/Gb}*\mathit{DRAM}\right)\right)*\mathit{PUE},$ (4) $$

where *NH* is node hours, $c_{\textrm{uf}}$ is the CPU usage factor, $c_{\textrm{tdp}}$ is the CPU’s thermal design power, ngpu is the number of GPUs on a node, $g_{\textrm{uf}}$ is the GPU usage factor (we assume 100% utilization), $g_{\textrm{tdp}}$ is the GPU’s thermal design power, DRAM is dynamic random access memory, and PUE is the power usage efficiency. energy is reported in watt hours. We record system-specific parameter values in Table 4.

$$ $\mathit{carbon}=\left(\mathit{energy}/1000\right)*CI$ (5) $$

Above in Eq. (missing) 5, *energy* is obtained in watt hours from Eq. (missing) 4, and *CI* is the carbon intensity reported based on the yearly regional average for each computing center from (ElectricityMaps, ). *carbon* is reported in grams.

To offset the carbon cost of these experiments, we publicly release our check pointed trained models (upon publication).

**Table 4: System Parameters for Polaris (ALCF, ) and Perlmutter (NERSC, ) to estimate energy and carbon usage.**
| Machine | $c_{\mathit{uf}}$ | $c_{\mathit{tdp}}$ | $\mathit{ngpu}$ | $g_{\mathit{uf}}$ | $\mathit{gpu}_{\mathit{tdp}}$ | $\mathit{DRAM}$ | $\mathit{PUE}$ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Polaris | 0.5 | 225 | 4 | 1 | 250 | 512 | 1.58 |
| Perlmutter | 0.5 | 280 | 4 | 1 | 300 | 256 | 1.58 |

We group experiments into three main phases:
(i) Training TinyMem Models;
(ii) TinyMemModels LM Mitigation and
(iii) Production-Grade LM Memorization Mitigation
. We report the resource usage for each phase of experimentation in Table 5.

**Table 5: Energy & Carbon Estimates for our experiments based on Eq. (missing) 4 & Eq. (missing) 5 respectively.**
|  | Experiment | Machine | Node Hours | Energy (kWh) | Carbon (Kg) |
| --- | --- | --- | --- | --- | --- |
| (i) | Training TinyMem models | Polaris | 1116 | 2297 | 910 |
| (ii) | TinyMem Mitigation | Polaris | 6251 | 12,871 | 5097 |
| (iii) | Production-Grade Mitigation | Perlmutter | 726 | 1,647 | 430 |
|  | All Experiments (total) | - | - | 16,815 | 6,437 |

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: What does the test accuracy in Table 1 represent?
The perplexity scores in Table 2&3 are quite high - can you describe how these are calculated and on what dataset?
soundness: 3
strengths: The authors present a comprehensive comparison of 17 methods for reducing memorization.

Applying the TinyMem test bench to larger models (Pythia) demonstrates the generalizability of the result
summary: This manuscripts presents a comparison of 17 methods for reducing memorization in large language models (LLMs) as well as a testbench 
 for measuring model memorization comprising of a set of small models called TinyMem. The authors find that the BalanceSubnet method is the most effective at reducing model memorization while preserving performance.
weaknesses: No major methodological weaknesses, however I find this study to be a straightforward comparison of several (existing and new) techniques without any particular novel insights. While this type of work is useful, the authors should extend the work by investigating why the best method works well or extracting practical insights from the results (beside stating that the method is most effective).

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. In Line 066, "the few existing models with known memorized data are large" lacks detailed examples and references.
2. In lines 143-144, why test whether the next n−k tokens are matched with the clean sequence? Shouldn't it still match with the noise sequence? Or only disturb the first k words of one sequence?
3. Why is there a big difference in the mitigation performance and time of the same method in Nosie and Backdoor? Is there a difference in the difficulty of eliminating different types of memorization? The time consumption of the BalancedSubnet method in Table 1 is 0.00. How is it calculated here? Is there an error?
4. It is recommended to bold the best results in all tables.
5. The tenses in Lines 327-336 are not consistent.
soundness: 2
strengths: 1. This paper focuses on the critical problems, namely, the challenges of developing and evaluating memory elimination methods directly on large models.
2. This paper innovatively proposes a series of small GPT2-style models (TinyMem), which is used to conduct memorization injection and memorization mitigation experiments in a quick and computationally efficient way. 
3. This paper proposes five new unlearning-based methods, and among them, BalancedSubnet outperforms other methods at removing memorized information while preserving performance on target tasks.
summary: This paper points out that developing and evaluating memorization mitigation strategies for open-sourced LLMs is a critical challenge due to the lack of memorized data or a big-time cost. To address these issues, the authors first propose a suite of GPT2-style models, namely TinyMem, to rapidly develop and evaluate memorization mitigation strategies. Based on TinyMem, this paper investigates three classes of existing methods, including training-time regularizer-based strategies, post-training fine-tuning-based strategies, and post-training unlearning-based strategies. Additionally, this paper also proposes five new unlearning-based methods. The experimental results show the effectiveness of these proposed methods.
weaknesses: 1. I am afraid that the evaluation results of different methods on TinyMem cannot truly reflect the performance of these methods in the memorization elimination of LLMs. On the one hand, the results in Table 1 and Table 2 show that some unlearning-based methods do not perform consistently in different types of tasks and memorization elimination. So how to effectively compare the comprehensive performance of different methods? On the other hand, the types of memorization within LLMs that need to be mitigated in practical applications are more complex and diverse. The two types of memorization (Noise and Backdoor) may be too simple compared to the real harmful memorization. The evaluation based on these two memorizations may hardly represent the ability of different methods to eliminate different types of memory within LLMs.
2. The innovation of the five proposed methods and their differences from previous methods should be introduced in detail. Additionally, although the algorithms are listed in the appendix, they still lack detailed introductions, which are difficult to understand.
3. Lack of detailed analysis of the reasons for the good performance of BalancedSubnet.

---

confidence: 2
contribution: 3
initial_rating: 8
presentation: 3
questions: Can you describe the difference between the setting of mitigating memorization and knowledge editing? It seems like the gradient-based methods of these settings are similar.
soundness: 4
strengths: 1. This paper first introduces TinyMem, a new suite of lightweight models specifically designed for rapid testing of memorization mitigation strategies. Additionally, the proposal of five new unlearning-based methods, including the innovative BalancedSubnet approach, adds contribution to the field. This work also highlights a novel way of applying unlearning-based techniques to real-world, production-grade models, bridging the gap between research and application.

2. Extensive experiments involve 17 memorization mitigation strategies, including both existing and newly proposed methods. The experiments are carefully designed to compare regularizers, fine-tuning methods, and unlearning-based strategies under diverse training conditions. The comprehensive analysis across different model sizes, data types, and training durations demonstrates the robustness of the findings. The use of Pythia models to validate the transferability of the methods adds credibility to the results.

3. The paper is well-structured and clear, with a logical flow from the problem of memorization in LMs to the proposed solutions and experimental results. The introduction of key concepts such as TinyMem and BalancedSubnet is explained effectively, and the figures, while not in vector graphic format, are useful in illustrating results.
summary: This paper focuses on addressing the challenge of language models (LMs) memorizing sensitive or private data during training, leading to potential issues such as data leakage. The primary contributions of the paper are:

1. TinyMem: The introduction of TinyMem, a suite of small, computationally efficient GPT2-style models that enable rapid prototyping and evaluation of memorization mitigation strategies.

2. Comprehensive Evaluation: An empirical evaluation of three main strategies for mitigating memorization: regularizer-based methods, fine-tuning-based methods, and machine unlearning-based methods, with five new unlearning-based approaches being proposed.

3. BalancedSubnet: Among the newly introduced methods, the BalancedSubnet unlearning method is highlighted for its superior performance in reducing memorization while preserving model performance across different datasets.

4. Applicability to Larger Models: The paper demonstrates that mitigation techniques developed using TinyMem can be successfully applied to production-grade models, such as the Pythia series, validating their effectiveness in real-world settings.

The paper provides a detailed analysis of these strategies, focusing on the trade-offs between mitigating memorization and maintaining model accuracy
weaknesses: This paper seems to be missing some important unlearning baseline methods, e.g., Gradient Ascent + Descent and Gradient Ascent + KL divergence [1].

[1] Jin Yao, Eli Chien, Minxin Du, Xinyao Niu, Tianhao Wang, Zezhou Cheng, and Xiang Yue. 2024. Machine Unlearning of Pre-trained Large Language Models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8403–8419, Bangkok, Thailand. Association for Computational Linguistics.

Meta-review: This work investigates methods to mitigate memorization in language models. It introduces *TinyMem*, a framework designed to rapidly develop and evaluate memorization mitigation strategies. The study finds that regularizers are slow and ineffective, fine-tuning is costly but effective, and unlearning methods are fast and precise at removing memorized data while preserving overall model performance. Consequently, the authors propose five other unlearning methods, among which *BalancedSubnet* performs the best. 

The comparisons are comprehensive, and the results demonstrate the generalizability of the proposed methods.  It might be beneficial for the authors to incorporate key revisions from the Appendix into the main body during the rebuttal, as the current version may distract readers.

My final recommendation is acceptance.

Decision: Accept (Spotlight)