Preserving Privacy Through Dememorization:  An Unlearning Technique For Mitigating Memorization Risks In Language Models
Authors: Aly M. Kassem
Omar Mahmoud
Sherif Saad
Venue: EMNLP 2023 Main (2023)
Keywords: Large langauge models
privacy
memorization
TL;DR: We propose a Dememorization framework to learn a paraphrasing policy to mitigate memorization risks in language models outperforming strong baselines.

Abstract: Large Language models (LLMs) are trained on vast amounts of data, including sensitive information that poses a risk to personal privacy if exposed. LLMs have shown the ability to memorize and reproduce portions of their training data when prompted by adversaries. Prior research has focused on addressing this memorization issue and preventing verbatim replication through techniques like knowledge unlearning and data pre-processing. However, these methods have limitations regarding the number of protected samples, limited privacy types, and potentially lower-quality generative models. To tackle this challenge more effectively, we propose “DeMem,” a novel unlearning approach that utilizes an efficient reinforcement learning feedback loop via proximal policy optimization. By fine-tuning the language model with a negative similarity score as a reward signal, we incentivize the LLMs to learn a paraphrasing policy to unlearn the pre-training data. Our experiments demonstrate that DeMem surpasses strong baselines and state-of-the-art methods in terms of its ability to generalize and strike a balance between maintaining privacy and LLM performance.

Preserving Privacy Through DeMemorization: An Unlearning Technique For Mitigating Memorization Risks In Language Models
Aly M. Kassem
Omar Mahmoud Sherif Saad +
c School of Computer Science, University of Windsor Applied Artificial Intelligence Institute, Deakin University (kassem6, sherif. sawddduwiddsor... ca mwwmooddreeeearreroooo deakin. edu. au
Abstract
Pretraining LM
Pre-Training Corpora
Data Preprocessing
LLM
Large Language models (LLMs) are trained on vast amounts of data, including sensitive information that poses risk personal pri- vacy if exposed. LLMs have shown the ability memorize and reproduce portions their training data when prompted by adversaries. Prior rescarch has focused on addressing this memorization issue and preventing verbatim replication through techniques like knowledge unlearning and data pre- pr-ppecessing.. However, these methods have limitations regarding the number of protected samples, limited privacy types, and potentially lower-quality generative models. To tackle this challenge tively, we propose "DeMem," a novel unlearn- ing approach that utilizes an efficient reinforce- ment learning feedback loop via proximal pol- icy optimization. By fine-tuning the language model with negative similarity score as a re- ward signal, we incentivize the LLMs to learn paraphrasing policy to unlearn the pre-training data. Our experiments demonstrate that De- Mem surpasses strong baselines and state-of- the-art methods in terms of its ability to gener- alize and strike balance between maintaining privacy and LLM performance.
l'ine-tuning
Subset
PoMee--Plliy-LLi.
Generated
Reward Fn
Negative Similarity
Figure I: First, LLM is pre-trained on large corpora in which Deduplication is applied. Then, a subset of training corpora employed to learn the LM DeMem Policy via negative similarity feedback.
effec-
Studies have shown that language model with 6 billion parameters (GPT-J) can memorize at least I 1% of its training data (Carlini et al., 2022). One potential cause of this memorization is the train- ing strategy of the language model, as its objec- tive is to identify the relationships between tokens, either an auto-regressive LM setup or through masked language modelling (MLM) (Devlin et al.. 2018), where the model predicts the masked to- kens based on their surrounding context (Radford et al., 2018). Additionally, repeated instances in the training corpus can contribute memorization, as more frequent examples are more likely be memorized (Lee et al., 2021). To address the is- sue of memorization in LLMs, several approaches have been proposed, including data sanitization (Lison et al., 2021), the application of differen- tial privacy algorithms(Abart et al., 2016; Anil et al., 2021; Li al., 2021; Tramèr et al., 2022; Basu et al., 2021), data deduplication (Kandpal al., 2022), and knowledge unlearning (Jang al., 2022). These techniques aim to prevent the gener- ation of memorized content. However, they also come with certain drawbacks. Data sanitization
1 Introduction
Large language models (LLMs) have experienced exponential growth in recent years, scaling up from millions to billions trillions of parameters (Rad- ford et al., 2019; Brown et al., 2020; Chowdhery et al., 2022: Fedus et al., 2021). As their scale increases, the training sets for these models also expand to billions of tokens (Gao et al., 2020), lead- ing to overall performance improvements, even in few-shot learning scenarios (Brown et al., 2020). However, this growth in model size and training data has raised practical concerns regarding privacy risks associated with memorizing the training data. Adversaries can extract individual sequences from a pre- mre-trinned model, even if the training dataset is publicly available (Carlini et al., 2021).

---

assumess that private information can be easily iden- tified and is not contex--dependent. Differential privacy can lead to lower-quality generative models (Anil et al., 2021). On the other hand, knowledge unlearning restricts the number of samples that can be forgotten at once to avoid degrading the overall capability of the language model, which may limit its effectiveness in real-world scenarios.
sumptions limitations about the data's structure or size be protected. Also, unlike the DP meth- ods, the proposed framework does not apply any partition mechanism to split the data into public data and private data; as language data cannot be partitioned(Brown et al., 2022), we apply the policy on all training data as defining, partitioning data into private and public, and limiting the number of samples inadequate in the real-world scenarios. To summarize, our main findings are the follow- ing:
In this study, we propose DeMemorization (De- Mem), reward-based (un)learning framework for language models. DeMem leverages a paraphras- ing policy to address memorization, using a nega- tive similarity metric as a reward to encourage the language model (LM) to unlearn.
Using a reinforcement learning feedback ap- proach results in little to no performance degradation of general capabilittes while be- ing practical, consistent, and independent of increasing the number of protected samples. At the same time, maintaining the fluency and coherence of the generated samples.
Given samples of prefixes and suffixes from the original pre-training data of the language model, we use a prefix as input for the language model to generate the suffix; then, we compute the negative BERTScore (Zhang et al., 2019) to measure the dissimilarity between the true suffix and generated suffix, the dissimilarity scores are then regarded as reward signal to maximize in the training process, which guarantees that the approximate memoriza- tion will be mitigated.
As the language model size increases, the con- vergence rate improves. Convergence refers to the model-generated suffixes diverging sig- nificantly from the original ones while the perplexity difference between generated and original examples decreases.
For instance, given training sample like "Alice Green lives at 187 Bob Street," where the prefix is " "Alice Green lives at" and the suffix is "187 Bob Street", our goal is to have the fine-tuned LM para- phrase the suffix as "12 Red Street." This para- phrasing approach minimizes the memorization relationship between the prefix and suffix without erasing the training sample from the LM's parame- ters or replacing it with meaningless content, which negatively impact the LM's performance
As the size language model increases, the dissimilarity score increases. This suggests that larger models may tend to "forget" the memorized data faster.
Combining Deduplication with DeMemo- rization enhances privacy with insignificant degradation(--O.56) in the Language model performance.
We conducted experiments using GPT-Neo and OPTLMs (with models ranging from 125M to 2.7B parameters) (Black et al., 2021; Zhang et al., 2022). DeMem achieved little no performance degra- dation on the initial LM capabilities measured via nine common NLP classification benchmarks (Hel- laswag (Zellers et al., 2019), Lambada (Paperno et al., 2016), Winogrande (Sakaguchi et al., 2021), I), CОРA (Roemmele et al., 20,1), ARC-Easy, ARC- Challenge (Clark et al., 2018), Piqa(Bisk et al., 2020), MathQA (Amini et al., 2019), and Pub- medQA (Jin al., 2019)).
2
Background
2.1 Memorization Definitions
In the context of memorization in large language models, we follow the definition proposed by (Lee et al., 2021), which introduced approximate mem- orization. Given a string S, splitted into prefix (P) and suffix (Sr). We fed the prefix to the LM to get the generated suffix (Sc). The memorization is measured with the chosen edit distance between the true and generated suffix. In our study, we choose the edit distance to be a similarity measure (SacreBLEU (Post, 2018)) as proposed in (ippolito et al., 2022), to be able to capture the approximate memorization, not just the "Eidetic memorization" (Carlini et al., 2021) as the definition of verbatim
We also evaluate DeMem increasing the con- text of the prefix, as many studies show that a longer context is provided, the memorization ratio increases (Carlini et al., 2021, 2022). The pro- posed framework makes no explicit, implicit as-

---

memorization fails include more subtle forms of memorization (Ippolito et al., 2022).
boundaries (Brown et al., 2022).
Knowledge Unlearning (UL): is an effective method that reverses the training objective of mini- mizing the negative log-likelihood for forgotten tokens. minimally affects language modeling performance in larger models for small number of samples. UL has two approaches: batch unlearning for multiple samples and sequential unlearning for smaller chunks. However, unlearning large number of samples at once significantly degrades average language model performance. While UL effectively addresses memorization, it has not been tested on sample sizes larger than 128. Also, It does not preserve fluency or coherency for generated suffixes, which are crucial for practical applications.
2.2 RL In Language Models
Unlearning undesirable behaviors is more compati- ble with the reinforcement learning (RL) paradigm. In the realm of NLP, RL has been employed to enhance scalar metrics through reward optimiza- tion (Ramamurthy et al., 2022; Ziegler et al., 2019; Ouyang al., 2022). Lately, RL has gained promi- nence for addressing undesirable behavior, includ- ing toxicity. social biases, and offensive speech. This accomplished by using Proximal Policy Op- timization (PPO) (Schulman et al., 2017) to opti- mize a Language Model (LLM) based on a reward model. In this paper, we investigate using RL with a language model to mitigate privacy risks associ- ated with memorization.
this work, we compare our proposed method with a data- lata-preprocessing approach proposed by (Kandpal et al., 2022), which shows that deduplicat- ing helps minimize data memorization. While this method is effective, we show that memorization is still high in the LMs pre-trained with this approach; thus, show that combining pre-processing with our approach, "Deaae "DeMemorizatiinn" effectively miti- gates memorization. We also compare our method with UL and show not inadequate or impracti- cal in real-world scenarios due to limited number of samples to forget at once.
3 Related Work
In this section, we delve into recent studies to miti- gate memorization in language models, which can be categorized into three main approaches: data pre/post-processing, differential privacy methods, and knowledge unlearning.
Data Pre/Post- PrrrossttProeoggggg This approach re- duces memorization in training data by applying filters before or after feeding it into the language model. One method is data deduplication (Kandpal et al.. 2022), which removes duplicates and im- proves model performance. However, it only par- tially protects against memorization the model can still memorize non-duplicate sequences. An- other approach "MemFREE decoding" (1ppolito et al., 2022), which efficiently checks the memo- rization in the LM generation by an n-gram in the training dataset.
4 Methodology
4.1 DeMemorization Via Dissimilarity Policy
DeMemorization framework operates by learning a paraphrasing policy to mitigate memorization risks. We divide each sample into prefixes and suffixes using an LM and a subset of pre-training data. The unlearning process is as follows: we select prefix P and true suffix Sy, then input the prefix into the pre- pre-trained LM to produce a suffix Sç. Using negative similarity metric, evaluate how the generated suffix is dissimilar to true. We use that
Differential Privacy (DP): is widely-used technique for training models prevent memoriza- tion of individual training examples (Abadi al., 2016). While effective for fine-tuning language models (Yu et al., 2021; Li et al., 2021), DP often reduces performance compared to non-private models (Anil et al., 2021). State-of-the-a art language models are typically trained without DP, using large amounts of data and computational resources. DP algorithms are computationally expensive, slower converge, and have lower utility compared to non-private methods (Anil et al., 2021). Applying DP to language data is challenging due to defining private information
as a reward signal to encourage the LM develop a paraphrasing policy, generating dissimilar tokens to minimize memorization. These steps can be summarized as follows:
P.ST ~ Dr
(1)
SG -
IPP))
(2)
Dis = -BBERT Score(So.
(3)

---

4.1.1 Reward Function
all models; we selected a specific number of steps for cach model as the convergence rale for each model different. We mean by convergence in this context that the model-generated suffixes become significantly different from the original suffixes but without a considerable loss in the perplexity as the difference between the perplexity of the generated examples and original examples becomes smaller, so we selected the appropriate number of steps that balance between these goals.
To yield the desired outcome of paraphrasing to mitigate memorization risk, need to employ similarity function to achieve this goal. The pro- posed reward function should allow changes in words or even the entire sentence while preserving the semantic meaning, Also, while learning the paraphrasing technique, we aim to ensure that the fine-tuned or Dememorized LM stays within the original LM to avoid potentially less coherent and relevant generation.
Learning Dissimilarity with BERTScore. To achieve the dissimilarity goal, we employ BERTScore. One advantage of BERTScore over other contextual embedding methods is the ability to operate on pairwise tokens using contextual em- beddings, providing a more flexible definition of dissimilarity in our context. This flexibility means that BERTScore can yield a high similarity score for different words that share the same entity, en- couraging the language model learn a paraphras- ing policy effectively. We employed the F-score metric produced using BERTScore.
4.2 Measuring Memorization In Language Models
As mentioned in subsection 2.1, we adopt the con- cept of approximate memorization, as it provides a more precise and adaptable approach to capturing subtle forms of memorization compared to the limi- tations of exact memorization. We employ widely accepted text similarity measure from standard Nat- ural Language Processing (NLP) evaluation tech- niques to quantify approximate memorization accu- rately: the SacreBLEU metric. SacreBLEU is an improved version of BLEU, known for its stability in measuring the quality of machinc-generated text.
Achieving Stability Via KL Penalty. To achieve the stability goal, we introduce a KI. di- vergence penalty term to quantify the dissimilarity between these two policies. This step helps en- sure that our optimization process remains within trustworthy region. The KL divergence, calculated for the policies, is expressed as:
To measure forgetting, we consider the negative of SacreBLEU, By utilizing SacreBLEU as met- ric for estimating approximate memorization, we define DeMemorization forgetting the process of minimizing the relationship between the given prefix P and the suffix S.
To (ail(ss)) To, (a:)s))
(L(0))0.) = Σ (a(a)(s)) log
(4)
This relationship represents the information that the adversary seeks to extract based on the given prefix. The metric we mentioned quantifies this relationship. In an example scenario, an adversary has the personal email address "bob @ aadam.com' and seeks to obtain the password. If the LM has memorized this association, it can provide the pass- word "12345" when given the email, however, by minimizing or altering their relationship. LM can generate a different suffix as the password "0912,",
AE[L.)]
Here, we denote as the pre-trained policy, repre- senting a model that has undergone initial training without fine-tuning. Additionally, we introduce A. the updated policy, which signifies the policy after fine-tuning further training.". We deduct KL divergence with default value weight 3 0.2 as penalty term.
4.1.2 Policy Optimization Via PPO
To optimize the policy, we employ a Proximal Pol- icy Optimization (PPO) methodology, incorporat- ing a top-p sampling rate of 01.5, a technique com- monly referred to as Natural Language Policy Opti- mization (NLPO), as elaborated in-depth in (Rama- murthy al., 2022) (please refer to Appendix A for comprehensive elucidation). A value network V is included beside the language modeling head to estimate the value function. The batch size is 32 for
As a result, the generated suffixes are valid and meaningful output without memorizing sensitive information. This approach achieves the dual ob- jectives of preserving the LM's general capability and the fluency of generated suffixes while ensur- ing privacy. Also, the solution is more practical in real-world situations than completely removing all information, which can negatively impact the capabilities of the language model (LM).

---

evaluation phase. In the first setting, we evaluated the model's ability to predict the suffix when pro- vided with only the prefix. the second setting, we evaluated the model's capability to predict the suffix when given the pre-prefix and prefix. This evaluation assessed the model's capacity to protect against acquiring additional information or knowl- edge. A longer context in language model can be considered form of attack (Carlini al., 2022). The sequence splitting is illustrated Figure 2.
Evaluation
Train & Evaluation
Pre-Prefix
Prefix
Suffix
100 Token
50 Token
Token
Sequence 200 tokens
Figure 2: Illustration of sequence splitting in the training & evaluation data.
5.1.2 Downstream Tasks
To ensure stronger privacy protections for language models (LMs) without compromising their original capabilities, we undertake a comprehensive evalu- ation that encompasses both privacy risks and the inherent strengths of LMs. This evaluation involves quantifying the LMs' performance across various classification tasks to assess their general capabil- ities. The tasks include Hellaswag (Zellers et al., 2019) and Lambada (Paperno al., 2016) bench- marks, which gauge linguístic reasoning abilities, as well as Winogrande (Sakaguchi et al., 2021) and COPA (Roemmele et al., 2011), which mea- sure commonsense reasoming abilities. Addition- ally, we utilize ARC-Easy, ARC-Challenge (Clark et al., 2018), Piqa (Bisk et al., 2020), MathQA (Amini et al., 2019), and PubmedQA (Jin et al., 2019) benchmarks assess scientific reasoning abilities. In addition to these classification tasks. We also measure the perplexity on the Wikitext (Merity et al., 2016) and Lambada (Paperno et al., 2016) datasets to gain insights into the LMs" lan- guage understanding and modeling. Whenever pos- sible, we use the test sets for these evaluations; otherwise,
5 Experiments
In this section, we begin by introducing the dataset used for training and assessing the paraphrasing policy. Subsequently, we assess the overall per- formance of the dememorized LM general per- formance on nine benchmarks. We then estab- lish the baseline methods for comparison. Finally, we define the evaluation metrics that enable us to measure the memorization and the performance in downstream tasks.
5.1 Experimental Settings
5.1.1 Memorization Dataset
We employed subset of the Pile dataset, released benchmark for training data extraction attacks large Language Models. Generally, the Pile dataset contains data from 16 different sources (e.g.. books, Web scrapes, open source code). We used this version of the subset 1, designed to be easy to extract to assess targeted attack performance. The dataset contains only 15,000 samples since the full version has not been released yet. Each sample consists of 200 tokens sampled randomly from the Pile training set. The topies included in the subset are code, news, logs, conversations, copyrights, links, etc. Most of them are in the English language. The dataset is splitted into 13,500 samples for training and 1,500 samples for testing.
resort to the validation sets. Also, did not report Lambada's perplexity & and ac- curacy as shows high values for perplexity & low values for accuracy for the UL baseline. To discard the anomaly and better assess the performance, we report it in Appendix E.
5.1.3 Baseline Methods
Our experiments used the GPT-NEO family (125M, 1.3B, 2.7B), pre-trained on the publicly available 825GB Pile dataset. Additionally, we employed the OPT family (125M, 1.3B. 2.7B) (Zhang et al., 2022), which was pre-trained on subset of the deduplicated version of the Pile, along with other corpora from diverse domains. OPT served as our bascline method for deduplication, as per (Jang et al., 2022), since the deduplicated version GPT.
Training & Evaluation Data. Each sample consists of 200-token sequence divided into 100 pre-prefix tokens, 50 prefix tokens, and 50 suffix tokens. During the training phase, we exclusively utilized the prefix and suffix tokens. However, we tested the model in two different settings during the
tht ttw.bttttttttttee///////eee..e com/ hamm/goolle/e ressearch/ 1m wwwaaacctononnnnnneee

---

NS
LMUACOT
GEN OPPLLL
Egecholign
Medel
N.SaveBLEU"
CENI
7/66
13
00.19
20.33
10.33
18
33
09.25
37385
F1R171 SO
12.15
+CL
125
16387716
E.S
54817065
10.44
23340.00
1.L
51.55
10.5
-DoMien
125
43.45
30.13
155
175
05 30
13.55
13
10.75
07.05
1.9
226
$2
10.29
T1.96
51.65
256
15.05
7.15
71.7
20.11
24.35
4.30
15.16
M6T1
41.33
2,00
93.51
5.76
-TeMeen
178
51311
29.03
16.70
3.07
2
DeMon
91.56
$1.0
1.50
9447
250
32.55
71.00
155
27.55
57.67
1543
190
67.56
53.74
14.31
6.88
250
182
16.22
0.3
121/6
1.9
<741
41.30
99:48
11.27
33
20.34
233
54.33
K.26
135
3681
52.45
14.15
1336
DcMan
$2.20
2.38
Table 1: Main Results: GPT-NEO averaged random samples (s = 32, 128, and 256) for UL. NEO initial GPT-NEO LM. UL+ knowledge unlearning, DeMEM = DeMemorization. LM ACC. = average accuracy of 8 classification datasets, LM PPL = perplexity of Wiki- text dataset, GEN PPI. = perplexity of generated suffix. Steps for DeMEM & Epochs for UL
Table 2: Main Results: OPT averaged random sam- ples 32, 128, and 256) for UL. UL. knowledge unlearning, DeMEM = DeMemorization. LM ACC average accuracy of 8 classification datasets, LM PPL perplexity of Wikitext dataset, GEN PPL perplexity of generated suffix. means that the value so high, Reaching infinity. Epochs for UL Steps for DeMeM.
value of learning rate of 1.41 x 10 was applied to all models. We use the default value of KL Beta of 0.2 and a clip range of 0.2. The GPT-Neo & OPT LMs were employed using the official release in the Hugging Face library. For UL training and mem- orization evaluation, we utilized the official code provided by the authors. For the selection of hyper- parameters, see Appendix F. In downstream tasks, we employed the nn-evaluation-hamee framework (Gao et al., 2021) for all baseline methods.
NEO LMs by (Kandpal et al., 2022) were not pub- licly accessible. We also applied DeMemorization to the OPT LMs, which can be seen as combina- tion of the deduplication approach and DeMemo- rization, resulting in a significant enhancement in the privacy of these models. Furthermore, we in- cluded UL (Jang et al., 2022) as a second baseline method to highlight weaknesses and distinctions.
5.1.4 Implementation Details
For training, we utilized the training subset and fine- tuned the GPT-Neo OPT LMs fine-tuned them for multiple iterations depending on the model size. To compare our proposed method with UL & dedu- plication, we followed the configuration proposed by (Jang et al., 2022) to ensure an adequate com- parison, we randomly sample samples from the test subset and evaluate the models those samples for UL since it forgets samples only at once, we make the LM forget the s samples and then evaluated. To follow the same configuration, we show the average results of 5 random samplings of s samples for all of our experimental settings.
5.1.5 Evaluation Metrics
We conducted comprehensive evaluation of De- Memorization and baseline methods, employing multi- perspeetreeeee approach to assess their effective- ness in three key areas:
(1) Measuring Forgetting: As mentioned in subsection 4.2, we employed negative Sacre-BLBU to quantify memorization.
To assess text fluency, we utilized the perplexity score of the un- derlying original model before forgetting. This metric enabled to assess the grammatical cor- rectness and coherence of the generated suffixes. Performance on Downstream Tasks: We sessed the performance of the unlearned models across nine classification tasks, employing accu- racy scores and perplexity measurements on Wiki-
(2) Evaluating Generated Suffixes:
To explore the impact of increasing the sample size to be forgotten, we performed five random sam- plings of 32, 128, and 256. DeMemorization was carried out using a batch size of 32, and default

---

123M
1.38
2,70
52
50
Avg. Perforance
48
46
44
42
40
of Samples Forgoten Once
DoMam
Figure 3: Average LM performance on the benchmarks when varying the total number of samples forgotten for NEO (125M, 1.3B, 2.7B).
text and Lambada.
plicated) exhibit higher N-sacreBLEUU scores than NEO (non-duplicate version) models while achicv- ing similar or better performance in downstream tasks. However, even in these models, memoriza- tion remains high, as only a portion of the memo- rized samples are duplicates.
5.2 Experimental Results & Discussion
We conducted comprehensive experiments as- sess the performance of DeMemorization against the baseline methods. Our main observations are as follows:
Therefore, we explored the UL approach and De- Memorization. The models that utilized both frame- works benefited significantly and became more ro- bust privacy LMs. While UL reduced memoriza- tion by approximately 99% of N-sacreBLEU. it also negatively impacted the general capability of the LM, resulting in 1 % difference from the original LM actoss various configurations. On the other hand, DeMemorizztion achieved compara- ble results UL, with reduction of ~94% in memorization, without the need to completely re- move training data points from the LM parameters. In comparison, the loss general LM capability was insignificant, at around ~0.5%, in the case of 125M and NEO 1.3B DeMemorization. even en- hanced performance. These findings suggest that employing a combination of deduplication and De- Memorization effectively mitigates memorization while maintaining the general capability of the LM. Since data deduplication is applied in most of the re- cent & large language models (Penedo et al., 2023; Touvron et al., 2023; Biderman et al., 2023; Taylor et al., 2022; Scao et al., 2022; Black et al., 2022), we believe our approach combined with deduplica- tion will effectively mitigate memorization.
5.2.1 Overview of The DeMemorization Performance
We comprehensively evaluated the DeMemoriza- tion approach nine classification tasks, wikitext for perplexity, and the generated samples. The eval- uation results, shown in Table demonstrate that the DeMemorization approach effectively pro- vides privacy and decreases the memorization for GPT-NEO while maintaining the LM general ca- pability, measured by evaluating the classification tasks. It also maintains the fluency of the general LM and generated suffixes. On the other hand, the UL approach provides more robust protection since it removes the data points completely from the train- ing data, which lowers the general LM capability by a large margin. This effective privacy-wise needs to be more practical from the performance perspective. Thus, we tried to balance this tradeoff by employing the DeMemorization approach. We provide the results for each dataset in Appendix E for reference.
5.2.2 Deduplication With DeMemorization & UL
We included OPT LMs baseline for the pre- processing technique, which applies deduplication decrease memorization. Deduplicating the train- ing data has effectively mitigated memorization, as Table Table demonstrate. OPT models (dedu-
5.2.3 Number of Samples, Stability, & Universal Policy
We investigated the impact of increasing the num- ber of samples on the performance of both UL and DeMemorization, line with the findings from

---

Before DeMem
(b) After DeMem
Figure 4: Threshold of 75% SacreBLEU of The Generated Samples Before & After DeMemorization For Neo 2.7B Longer Context.
(Jang et al., 2022), UL is sensitive to the number of samples being unlearned simultaneously. Our experimental results validate this observation in Ta- ble 1, Table 2. As the number of samples increases, we observe decrease in the LM's performance. On the other hand, DeMemorization demonstrates a different behavior as is unaffected by the num- ber of samples as shown Figure 3. In DeMemo- rization, the LM is fine-tuned one-time using nega- tive similarity as a reward during training, followed separate test set. This allows
Moddl
AParamelers
SauwBLEUT
BEFORE
PPL
N
SMINBLEU
AFTER
PPL
135M
6:12
03104
N6OI
1.38
39.58
0.64
88.31
7:58
2.78
10.55
1.41
$2.65
1.54
125M
Table 3: Comparsion of Negative SacreBLEU & Per- plexity Means Before & After Applying The Framework On Longer Context; 100 Extra Tokens Combined With The Prefix
by evaluation the model to learn universal policy to forget unlimited number of samples. Here, the term "un- limited" signifies the absence of any restrictions, assumptions, or re-training of the LM regarding the number samples to be unlearned.
plexity for all models. UL showed significantly higher perplexity some cases, even reaching finity. UL's high perplexity attributed to its gradi- ent ascent approach, which softens the probability distribution and leads to a more uniform distribu- tion and higher perplexity. However, this softening procedure degrades LM performance the model becomes less confident in generating tokens. We also evaluated the perplexity of unlearned samples, which crucial in practical applications where the unlearned data domain used. DeMemorizztion caused an average degradation of approximately 0.5% in NEO models and around 1.5% in OPT models. UL exhibited higher degradation in both models due to the complete removal of correspond- ing data points from the model parameters.
In UL, however, the model is fine-tuned and evaluated the same samples to forget them at time. To unlearn forget multiple samples, the model needs undergo fine-tuning multiple times through sequential or batch unlearning. In each iteration, the model is fine-tuned with a specific number of samples (typically 32, as suggested by the authors) to prevent a decrease in the LM's over- all capability. This can be regarded an assump- tion about the number of samples to be protected incomplete solution.
simultaneously, leading See Appendix G to highlight more UL framework assumptions.
5.2.5 Protection Against Discoverability Phenomenon
5.2.4 Perplexity of WikiText & Generated Suffix
Discoverability phenomenon refers to the obser- vation that some memorization only becomes ap- parent under certain conditions, such when a model is prompted with a sufficiently long con- text. (Carlini al., 2022) found that the fraction of extractable sequences increases in a log- log-linear fashion with the number of tokens in the context.
Perplexity serves as crucial metric for assessing the overall performance of Language Model (LM) in terms of its ability generate fluent and coherent text. We computed perplexity for Wikitext and presented the results in Table 1, Table 2.
DeMemorization had a minimal impact on per-

---

Generaeed
Gereneed
(OTT Cradiin Oals 2017 02-06 3000
TraneEE FILL VAME
Towelaa FUll NAME SEMAN SATORESSRSA
STO NAMEE COA.DRRROO
"Lanou Toart LANDUIGE
12.97
62.98
1.88
1:98
1: argra" Largange "AVV-tenss
EANGUANG 00 argnor
Lisnow, or qptier any baee vardoon weadd
fulllo cada out
Eailbo part but
18:41
99.95
1.93
2:11
witoout indied
FITNESSS
tie disieis os to the ROU, RSA, thack D8S5,
Cosrigh Clap aay code renowvl) product abold
induddd the dirtbudin orpylight tame sucept budder
Df 2.0
Figure 5: Generated & True Suffixes given the prefixes before & after applying DeMem. Green indicates that this part is memorized according to the true suftix, while red indicates that it's dissimilar.
For example, with context of 50 tokens, approxi- mately 33% of training sequences can be extracted from the NEO-6B model. However, with context 450 tokens, this percentage rises to 65%.
memorization examples decreased from 910 to 497 and 1036 321, respectively (refer to Appendix B for other models). The red region in Figure 4 rep- resents samples with scores equal or above 75%. After DeMeemrizatiin,, the distribution of samples spreads more evenly across different values instead of being concentrated beyond the 75% threshold.
We evaluated our DcMemorization approach by increasing the prefix context from 50 to 150 tokens. The results in Table 1, Table 2 show that extending the context does not significantly impact the 125M model in NEO, with forgetting rate decrease from 58.44% to 45.47% and has no effect in OPT-125M. However, for larger models like 1.3B and 2.7B, longer context considerably reduces the forgetting rate by approximately 49% in NEO and around 10% in OPT. Nevertheless, DeMemorization effec- tively counters this type of attack, increasing the forgetting rate by approximately 10% for the 125M model and approximately 30% for larger sizes in OPT & NEO shown in Table 3. This demon- strates the universality and generalizability of the learned policy across various scenarios.
Box plots (see Appendix D) confirm the efficiency of the DcMemorization approach, evidenced by the median of the sample's 's distribution before and after DeMemorization.
5.2.7 Qualitative Results
Figure 5 demonstrates that the framework is capa- ble of learning policy that reduces or eliminates the amount of memorized personal data, such email addresses. However, it should be noted that in certain instances, this can increase perplexity. More samples demonstrating Dememorization can be found in Appendix C.
5.2.6 Approximate Memorization Threhold
Conclusion
Based on (Ippolito et al., 2022), BLEU score of 75% for the generated suffix is considered a suit- able threshold for determining approximate mem- orization. However, our investigation found that even threshold as low as 50% after applying the framework can mitigate this issue. Nevertheless, we chose the widely accepted threshold of 75% to demonstrate the effectiveness of our frame- work. Applying DeMemorization to the LM re- sulted in a significant decrease in memorized sam- ples. For GPT-Neo 1.3B and 2.7B, approximate
In this paper, we present a novel framework that tackles the problem of training data memorization in LLMs. We achieve this by employing an RL paraphrasing policy. Through extensive evalua- tions conducted in diverse settings, demonstrate the effectiveness of our approach. Our framework successfully reduces memorization by significantly decreasing the SacreBLEEU score while preserving the overall capabilities of the LM as measured by nine classification benchmarks.

---

Limitations
ference on computer and communications security. pages 308-318.
One of the limitations of our work is that it relies on a single scalar reward for optimization, as the problem has dual objectives: dissimilarity and per- plexity. To overcome this limitation, suggest exploring other techniques, such as Multi-objective Reinforcement Learning, which can potentially en- hance performance and optimize both objectives simultaneously.
Aida Amini, Saadia Gabriel, Peter Lin, Rik Koncel- Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. 2019. Mathqa: Towards interpretable math word problem solving with operation- based formalisms. arXiv preprint 19XX:::905.133.1
Rohan Anil, Badih Ghazi, Vineet Gupta, Ravi Kumar, and Pasin Manurangsi. 2021. Large-scale differen- tially private bert. arXiv preprint 1000::2108.000 01624.
Priyam Basu, Tiasa Singha Roy, Rakshit Naidu, Zum- rut Muftuoglu, Sahib Singh, and Fatemehsadat Mireshghallah. 2021. Benchmarking differential pri- vacy and federated learning for bert models. arXiv preprint arXiv:21066 13973.
Ethics Statement
Improving the large language model to be privacy- preserving is crucial since the language models have become more prominent and involved in many of life. Ensuring the
Stella Biderman, Hailey Schoelkopf. Quentin Anthony, Herbie Bradley, Kyle O' O'Brien, Eric Hallahan, Mo- hammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. 2023. Pythia: A suite for analyzing large language models across training and scaling. arXiv preprint 23X.0:2304.01333
applications in multi- data privacy of those models is vital since some ad- versary may be able to reach that information. To make those models widely used, have to guaran- tee they cannot emit private data. In this paper, we hope our work will serve as a foundation for devel- oping new and innovative solutions to the problem of approximate memorization in large language models since verbatim memorization can give a false sense of privacy, as earlier work suggested. Our proposed framework provides a promising ap- proach to addressing this issue. Further research and experimentation in this area can lead to even more effective methods for reducing memoriza- tion in these models. Our work also highlights the importance of considering both the computational cost and the performance trade-off when develop- ing new techniques for addressing memorization in large language models.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. 2020. Piqu: Reasoning about physical com- monsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432-7439.
Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leaby, Kyle McDonell, Jason Phang, et al. 2022. Gpt-neox-20b: An open-source autoregressive language model. arXiv preprint arXiv:2204.000 06745.
Sid Black, Leo Gao, Phil Wang. Connor Leahy, and Stella Biderman. 2021. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh- Tensorflow. If you use this software, please cite it using these metadata.
Hannah Brown, Katherine Lee, Fatemehsadat Mireshghallah, Reza Shokri, and Florian Tramer. 2022. What does it mean for a language model to preserve privacy? arXiv preprint arXiv.2202.05520.
Acknowledgements
The authors would like to thank Niloofar Mireshghallah for helpful feedback.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 13:7717119
This research is supported by the Vector Schol- arship in Artificial Intelligence, provided through the Vector Institute and Natural Sciences and En- gincering Research Council of Canada (NSERC) by NSERC Discovery Grant. This research was en- abled in part by support provided by Compute On- tario and the Digital Research Alliance of Canada.
Nicholas Carlini, Daphne Ippolito. Matthew Jagiclski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. 2022. Quantifying memorization across neural lan- guage models. arXiv preprint arXiv:2202. 07646.
References
Nicholas Carlini, Florian Tramer, Eric Wallace. Matthew Jagielski, Ariel Herbert- Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. 2021. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pages 2633-2650.
Martin Ahadi, Andy Chu, Ian Goodfellow, H Bren- dan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. 2016. Deep learning with differential pri- vacy. In Proceedings of the 2016 ACM SIGSAC con-

---

Aakanksha Chowdhery. Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.03311.
Alexandra Levine. 2021, Suicide hotline shares data with for for-profit spinoff, raising ethical questions. politico.
Xuechen Li, Florian Tramer, Percy Liang, and Tatsunori Hashimoto. 2021. Large language models can be strong differentially private learners. arXiv preprint (xXiv:210000669)
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question an- swering? try arc, the ai2 reasoning challenge. arXiv preprint 1803 05447.
Pierre Lison, lldikó Pilán, David Sánchez, Montser- rat Batet, and Lilja Øvrelid. 2021. Anonymisation models for text data: State of the art, challenges and future directions. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joins Conference Natural Language Processing (Volume 1: Long Papers). pages 4188-4203
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arxiv preprint arXiv:1810.04805.
William Fedus, Barret Zoph, and Noam Shazeer. 2021, Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture mod- els. arXiv preprint arXiv: (4l9:1600.77844
Leo Gao, Stella Biderman, Sid Black, Laurence Gold- ing, Travis Hoppe, Charles Foster, Jason Phang. Ho- race He, Anish Thite, Noa Nabeshima, et al. 2020. The pile: An 800gb dataset of diverse text for lan- guage modeling. arXiv preprint arXiv:2101.00022
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroli Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instruc- tions with human feedback. Advances in Neural Information Processing Systems,
Leo Gao, Jonathan Tow, Stella Biderman, Sid Black. Authony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, al. 2021. A framework for few-shot language model evaluation. Version vO. O.I. Sept.
Denis Paperno, Germán Kruszewski, Angeliki Lazari- dou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. 2016. The lambada dataset: Word pre- diction requiring broad discourse context. arXiv preprint arXiv: 06031.
Ari Holtzman, Jan Buys, Maxwell Forbes, Antoine Bosselut, David Golub, and Yejin Choi. 2018. Learn- ing to write with cooperative discriminators. arXiv preprint Ttt::1805.0066
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary Vitoto, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation pytorch.
Daphne Ippolito, Florian Tramèr, Milad Nasr, Chiyuan Zhang, Matthew Jagielski, Katherine Lee, Christo- pher Choquette- Choo, and Nicholas Carlini. 2022. Preventing verbatim memorization in language mod- els gives false sense of privacy. arXiv preprint arXiv:2210.17546.
Cchoo,
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. 2023. The refinedweb dataset for falcon Ilm: Outperforming curated corpora with web data, and web data only. arXiv preprint 16X1v:2306.01116
Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Seo. 2022. Knowledge unlearning for mitigating privacy risks in language models. arXiv preprint arXiv:2210.01504.
Keith Porcaro. 2022. The real harm of crisis text line's data sharing. wired.
Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W Cohen, and Xinghua Lu. 2019. Pubmedqa: A dataset for biomedical research question answering. arXiv preprint arXiv:1909 1909.06146.
Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference Machine Translation: Research Papers. pages 186- 191, Brussels, Belgium. Association for Computa- tional Linguistics.
Nikhil Kandpal, Eric Wallace, and Colin Raffel. 2022.
Deduplicating training data mitigates privacy risks in language models. arXiv preprint 0r2iv22202.06539.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. 2018. Improving language under- standing by generative pre-training.
Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. 2021. Deduplicating training data makes language models better, arXiv preprint arXiv:2107.06499.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog, 1(8)99

---

Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannaneh Hajishirzi, and Yejin Choi. 2022. Is reinforcement learning (not) for natural language processing?: Benchmarks, baselines, and building blocks for natural language policy optimiza- tion. arXiv preprint arXiv:22 10.00141..
Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi, Huseyin A Inan, Gautam Kamath, Janardhan Kulka- mi, Yin Tat Lee, Andre Manoel, Lukas Wutschitz, et al. 2021. Differentially private fine-tuning of lan- guage models. arXiv preprini
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can machine really finish your sentence? arXiv preprint arXiv:1905.07830
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020. Deepspeed: System optimiza- tions enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference Knowl- edge Discovery & Data Mining, pages 3505-3506.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li, Xi Victoria Lin, 2022, Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.0 2205.0106.
Melissa Roemmele, Cosmin Adrian Bejan, and An- drew Gordon. 2011. Choice of plausible alter- natives: An evaluation of commonsense causal rea- soning. In AAAI spring symposium: logical formal- izations of commonsense reasoning, pages 90-95.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Eval- uating text generation with bert. arXiv preprint arXiv:190.00967.
Keisuke Sakaguchi. Ronan Le Bras, Chandra Bhagavat- ula, and Yejin Choi. 2021. Winogrande: adver- sarial winograd schemna challenge at scale. Commu- nications the ACM, 64(9):99-110 106.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Chris- tiano, and Geoffrey Irving. 2019. Fine-tuning lan- guage models from human preferences. arXiv preprint 190.029909.08593.
Teven Le Scao, Angela Fan, Christopher Akiki, EI- lie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni. François Yvon, Matthias Gallé, et al. 2022. Bloom: A 176b- parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100
John Schulman, Filip Wolski, Prafulla Dhariwal, Alee Radford, and Oleg Klimov. 2017. Proxi- mal policy optimization algorithms. arXiv preprint arXiv:
Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic, 2022. Galactica: A large language model for science. arXiv preprint 2ete2221900900
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux. Timothée Lacroix, Baptiste Rozière, Naman Goyal, Fric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302.1397 13971.
Florian Tramèr, Gautam Kamath, and Nicholas Car- lini. 2022. Considerations for differentially private learning with large-scale public pretraining. arXiv preprint 2rXiv:2212.064777
Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, and Nathan Lam- bert. 2020. Trl: Transformer reinforcement learning. https: /ttps::/gtt //gi thub. com/l 1ww.rr//tr//
Thomas Wolf. Lysandre Debut, Victor Sanh. Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, al. 2019. Huggingface's transformers: State-of- the-art natural language processing. arXiv preprint 1rXiv=1910.03771.

---

A Natural Language Policy Optimization vs PPO
To tackle the challenge posed by large action spaces in language generation tasks, the NLPO (Natural Language Policy Optimization) framework was proposed. Previous research by (Ramamurthy et al., 2022) highlighted the difficulties faced by existing RL algorithms when dealing with models like GPT-2/3 and T5, which have extensive vocabularies of 50K and 32K tokens, respectively, and this issue becomes even more pronounced with newer models. NLPO introduces a masking policy that is periodically updated and incorporates a top-p sampling technique during training. This technique helps address the dilemma of balancing the inclusion of task-relevant information while mitigating the risk of reward hacking, By extending the PPO (Proximal Policy Optimization) algorithm, NLPO aims enhance the stability and effectiveness of training language models. NLPO achieves this by employing top-p sampling through generating, which restricts the selection of tokens to smaller setting where the cumulative probability surpasses a given threshold parameter, (Holtzman et al., 2018).
B Displaying Approximate Memorization Threshold
Recent studies suggested that approximate memorization occurs the BLEU score of 75%; we follow this suggestion and demonstrate the effectiveness the proposed framework in this section by comparing the number of samples that exceed this threshold before and after applying the framework.
SacreBLEUssufiic, suffixy) >> 0.75
(5)
(a) True Suffixes Standard Setting
(b) Generated suffixes Standard Setting
(c) True Suffixes Longer Context Setting
(d) Generated Suffixes Longer Context Setting
Figure 6: Threshold of 75% Of The True & Generated Samples SacreBLEU For GPT-Neo 125M Standard Setting
As shown in Figure 6, the memorization ratio for the GPT-Neo 125M model is relatively low. However, when using standard and longer context settings, there are many instances where the samples are distributed on and beyond the 75% threshold. Despite this, after implementing the proposed framework, the distribution of samples is more evenly spread across various values rather than being concentrated solely in the region beyond the 75% threshold. contrast to the other variation, GPT-Neo 1.3B & 2.7B have a large memorization ratio, especially in case of longer context; the framework effect can be seen obviously as many samples exceed the threshold in case of those variations as shown in Figure 7 and Figure 8.

---

True Suffxes Standard Setting
Generated suffxees Standard Setting
(c) True Suffixes Longer Context Setting
(d) Generated Suffixes Longer Context Setting
Figure 7: Threshold of 75% Of The True & Generated Samples SacreBLEU For GPT-Neo 1.3B Standard Setting
(a) True Suffixes Standard Setting
(b) Generated suffixes Standard Setting
The Sufle CacntLel Serm
Denoriad Cullee SCHNCLEU Stote
(c) True Suffixes Longer Context Setting
Generated Suffixes Longer Context Setting
Figure 8: Threshold of 75% Of The True & Generated Samples LacreBLEU For GPT-Neo 2.7B Standard Setting

---

C Qualitative Results
this section, we demonstrate the effectiveness of our proposed framework by presenting a thorough analysis of samples generated before and after its application. To provide a comprehensive evaluation, we have chosen samples from various model sizes, including 125M, 1.3B, and 2.7B, and included examples from both standard and longer contexts. Additionally, we present samples from different training phases to showcase the learned policy's evolution over time. As previously mentioned, the policy initially focuses replacing individual words numbers to decrease the similarity between samples. As the training process progresses, the policy becomes more aggressive and replaces entire phrases, as shown Figure 9.
Phete
Thes Sunts
Ganersted Surbo-Caton
Generaied Gullviterr
PML
stay EXEOUITIN
nEcLAge Lero BAR, Hos BSER PREVOUULY
botty liee apperl pweinn Rule to Rf Pradiss
pooploolly oalyyooe cocdroad for perioi rucll eecess caryy
ant Rulk Sac Buanemn Chis. Rdaiterll ritha Suarmn CIn
SupKore Countol Ohio
day
continued prodooudy
prme Couner
applitionn day sumoo
Acpellan
Addilorats, th Supremn
ontini DRISUE
contrued by tis cntyy, tammiaae the curter of Appellan bust to roliowan appeat wit the Segeaim Courrol Oflo
espeal oxgasiion
say dlys,,
dEpoal peir Leciotinn
MeckaD redoroa itue
borray www.rtoawwaaarrroooooooo.
<dbosiindllt apa divalied Ensweer cliic ceasciipo cheater roberbanne
START Tom Kittraal gortantaorigg
START TOR MMTAAR
3.10
88:27
3.04
3.20
Selene Prots Soodler imenedions Ss the Yhur
product independert research
to provide acturate up monerchi it movet in you
utiluied vitn miStrall Puoremucoo conns cational loar. & service aros cars am not reppore the for content
conaiia one from
sstig owns onderaioodd
Iscaact momait to
persoral
Rocrodd solooi
recommmnnt
reveers.Ce Cenzol
or
couled commenti loppes
accountee, other
Mie lakedio Taunea incladly and con rontt Contect (u Decisimoo nox We
IMPTLE INCLUDING VARRMANTIES OF VENG ANTAARITTY
NPPLED: INCLDING BUT
BEYLES INCLUDING HARRSANTIES OIT MERG INNTABUTIE
NOT TE2
tha poin outen nutioo shull be induadee" oill copica PROViti YuR (P, WITHOOUTT WARRRATTY AWY
WARRRANT orin
MOROOAA LIT
PARTIOI AR P. IIROSE AND IN EVENT
BART CILLICHLA PUSROSR
PART GUE AR PUBRRSSE
AND BORGNTERNNEDE
Figure 9: Suffixes that are memorized by the employed language models and the generated suffixes given the same prefix. Green indicates that this part is memorized according to the true suffix, while red indicates that it's dissimilar.

---

D
Median Comparison
Origall
(a) GPT-Neo 125M On Standard Setting
(b) GPT-Nco 1.3B On Standard Setting
(c) GPT-Neo 2.7B On Standard Setting
Frencook
(d) GPT-Neo 125M On Long Setting
(e) GPT-Neo 1.3B On Long Setting
(1) GPT-Neo 2.7B On Long Setting
Figure 10: Displaying The Negative SacreBLEU Distribution of The Models On Standard & Long Settings Before (blue) & After (orange) Applying The Framework

---

E Results of Each Dataset
Hiell..
Lamba.
Wine.
COPA
ARC-E
ARC-C
Piqu
MathQ
PubQ
Avg
Model
#Samples
(ACC)
(ACC)
(ACC)
(ACC)
(ACC)
(ACC)
(ACC)
(ACC)
(ACC)
(ACC)
NEE
28.66
37.35
50.43
61.0
43.72
19.11
63.05
22.78
55.10
43.36
+DeMEM
28.54
34.09
50.74
64.00
43,89
19.96
63.00
22:44
55.10
43.46
28.47
1.3
52.09
60.00
36,90
20.13
58.48
21.10
39.50
39.56
27.03
0.05
50.59
59,00
31.18
18.94
54.00 54.95
20.40
33,80
36.98
28.40
50,90
61.00
37.87
36.20
26,66
0.00
52,95
56.00
31.90
19.28
55.22
19,93
33,80
36.97
26.71
54.00
31.10
22,91
26.31
51.61
54,00
29.67
18.85
53.09
19.43
33,80
35.92
+UL
128
26.91
0.67
51.30
59.00
32.91
18.06
55.76
20,97
33,90
37.35
26,
26, 40
$1.06
53.00
29 16
18.60
52:55
19.83
53,80
35.55
26.61
26.56
53.51
36.00
31.94
20.64
59.03
22.37
55.20
36.14
26.78
0,05
36.58
26.56
50.82
55,00
30.93
18.60
54.24
20.43
33,80
36.30
26.73
31,06
55.00
31.39
18.43
54.00
20.63
33,80
36.38
NED
38.65
57.20
54.93
69.00
56.18
23.12
71.10
24.05
54,40
48.93
+DMMEM
38.73
51.71
55,48
73.00
55.17
23.63
70.72
23.65
54.80
49.40
39.00
24.23
54.69
74.00
54.25
25.25
69.58
23.11
52,00
48.98
38.50
72.00
69.47
23.31
+L
32
35.51
65.70
53.82
75.00
51.13
22.86
68.11
24.22
55.00
48.21
37.41
64.48
56,66
73.00
52.94
23.63
69.11 1.5
23.85
53,70
48.79
8.51
52.88
30.00
$4.50
33.32
59.58
$7.30.
66.00
47.34
21.58
65.72
25.19
55.10
46.14
26.89
34
36.15
90.11
40.52
53.90
65.00
42.29
20.81
61.91
23.68
55.00
44.09
31.14
6.63
55.72
63.00
42.29
19:53
63.05
22.17
34,00
41.36
28. 79
5.93
52.17
55.00
33.45
18.85
56.80
21.60
56.70
40.42
29.43
39.53
53.35
60.00
36.57
23.24
55,20
42.09
+UL
256
28.02
28.
53.90
62.00
34,97
19.11
58.75
23.31
54,60
41.83
29.19
31.34
52,40
55.00
35.47
19.36
58.75
22.61
55,20
41.00
56.00
37.07
21.94
NE0
79.00
DeMEM
42,30
39.42
58.01
79.00
00.14
23.47
71.68
24.58
56.70
52.48
39.33
77.00
58.75
24.12
28.34
23 23.48
53.11
75.00
31.90
22:44
55.00
21.84
52.70
42.61
76.00
57.40
43.69
44.65
$8.16
73.00
59.34
27.04
71.87
25.22
60.30
$2.33
55.30
31.86
55.88
54.38
69.00
43.60
20.39
66.25
23.21
55.30
45.50
40.36
44.85
UL
128
41.52
50.36
59.64
71.00
59.37
25.51
71.32
23.91
61.10
$1.42
37.16
58.43
58.32
71.00
50.25
23.72
69.01
21.38
56.90
45.84
39.20
17.54
59.27
73.00
52.14
25.85
68.60
22.51
37,80
47.29
31.86
55.88
54.38
69.00
43.60
20.39
66.26
23.21
57.00
45.71
25.00
0.15
52.72
58.00
26.09
19:45
53.91
19.96
55.20
35.00
+UL
256
26.52
0.03
51.69
58.00
27.44 44
17.91
54.89
19.63
56.60
39.08
25.97
58.00
26.
29.4
6.30
50.67
62.00
30.59
20.47
56.03
20.56
34,70
38.00
Table 4: Main Results:
NEO averaged 5 random samples (s = 32, 128, and 256) for UL. UL = knowledge unlcarning,
DeMEM = DeMemorization. LM ACC average accuracy of 8 classification datasets. Lambada Accuracy excluded from the average due to anomalies

---

Hella.
Lamba.
Wino.
COPA
ARC-E
ARC-C
Piqa
MathQ
PubQ
Avg
Model
#Samples
(ACC)
(ACC)
(ACC)
(ACC)
(ACC)
(ACC)
(ACC)
(ACC)
(ACC)
(ACC)
OPT
29.21
37.92
50.28
66.00
43.52
19.11
63.00
22.04
37.10
41.28
+D+MEM
28.90
36.08
50.43
66.00
40.99
19.70
62.73
21.64
47.60
42.25
26.84
50.03
60.00
28.87
19.79
57.01
21.50
33.80
37.23
26.61
0.02
52.40
60.00
28,
19.88
56.47
20.77
33.80
37.30
+UL
32
26.87
2.27
49.25
58.00
29.40
19.70
57.67
21.17
33.80
36.98
26.67
0,42
49.48
61.00
28.61
20.30
56.42
20.67
33.80
37.12
26.68
0.34
51.46
56,00
28.95
20.39
55.71
20.46
33.80
36.68
26.61
0.03
48.22
57.00
28.40
21.16
54.89
20.23
33.80
36.29
26.70
0.03
49.40
57.00
28.57
21.16
55.33
20.70
33.80
36.58
+UL
128
26.62
50.11
58.00
28.32
21.07
54.57
19.83
33.80
36.54
26.68
0.03
51.14
57.00
28.32
21.50
55.05
20.70
33, 33,80
36.77
26.50
0.01
49.32
57.00
28.28
20.64
54.62
19.69
33.80
36.23
26.73
49.64
60.00
28.57
21.16
57.12
21.27
33.80
37.29
26.91
0.7
50.82
61.00
28.74
20.64
56.40
21.34
33.80
37.46
+UL.
256
26.99
0.5
50.27
59.00
28.36
21.50
56.52
20.77
33.80
37.15
26.84
0.4
50.82
58.00
28.07
21.84
56.25
21.23
33.80
37.11
26.93
49.48
58
28.15
21.33
56.63
21.13
33.80
36.93
L3T
41.48
57.91
59.35
79.00
57.07
23.42
71.76
23.29
57.90
51.65
+DeMEM
41.57
53.74
60.45
78.00
55.13
24.91
70.83
23.85
56.50
51.40
30.37
0.64
52.17
59.00
28.15
23.72
56.69
20.50
55.50
40.76
30.60
$1.30
29.71
23.29
57.61
20.93
40.40
39.10
+UL
32
30.07
0.81
51.93
60.00
27.73
23.03
56.47
20.90
51.20
40.16
28.13
51.77
54.00
27.18
22.35
55.27
19.83
33.80
36.54
30.28
1.88
50,82
63.00
29.04
22.18
57.07
21.23
40.40
39.25
27.10
51.14
53.00
24.07
21.75
55.93
19.26
55,20
40.76
27.49
51.38
24.32
22.26
56.20
19.29
55.20
39.10
+UT.
128
27.34
50.90
54.00
24.66
23.20
55.76
19.09
55.20
40.16
28.53
$1.06
60.00
27.86
22.44
55.60
20.67
48.
36.54
27.16
53.27
49.00
24.74
22.01
56.42
19.26
55.20
39.25
27.87
$1.46
56.00
27.27
21.75
56,20
20.77
34.70
37.00
28.32
50.82
56.00
28.03
22.18
56.63
20.77
34.10
37.10
+UL
256
27.93
50.74
54.00
27.
21.24
55.05
20.16
33,90
36.37
27.98
50.82
54.00
28.03
21.58
55.87
20.80
33.80
36.37
28.03
51.14
54.00
27.06
22.44
55.60
20.56
38.70
37.19
2P
45.84
63.57
61.01
77.00
60.77
26.88
73.83
23.85
60.80
53.74
+DMMEM
41.57
53.73
60.22
76.00
58.08
24.74
72.41
23.71
60.90
52.20
30.88
0.75
52.56
58.00
29.50
23.20
57.61
20.67
55.60
41.00
25.32
50.35
50.00
24.20
22.26
54.46
19.59
55.20
37.67
+UL
32
28.26
51.14
51.00
25.42
23.37
54,89
20.36
55.20
38.70
25.28
51.69
49. 00
21.84
19.19
$5.20
37.56
25.37
50.19
53.00
24.45
22.86
54.57
19.09
55.20
38.09
27.31
$1.93
47.00
24.70
22.01
57.39
19.56
55.20
38.14
37.09
37.55
49.88
65.00
39.39
22.61
64.09
20.83
54.50
44.17
+UL
128
27.67
$1.85
47.00
24.70
22.18
56.36
19.09
55.20
38.00
36.16
34.32
49.64
64.00
38.88
21.75
62.84
21.84
54.40
43.69
31.67
9.64
49.48
62.00
31.14
23.12
59.57
22.17
$1.40
41,32
25.89
$1.06
49.00
24.36
22.61
56.63
19.36
$5.20
38.01
27.21
$2.56
43.00
24.70
22.35
56.96
19.32
55.20
37.66
+UT.
256
31.10
50.27
31,52
22.61
60.44
22.11
41.90
39.99
26.86
50.43
44.00
24.24
21.67
55.76
19.43
55.20
37.20
25.32
50,59
54.00
24.62
22.44
54.18
18.79
55.20
38.14
Table Main Results: OPT averaged random samples (s 32, 128, and 256) for UL. UL knowledge unlearning. DeMEM = DeMemorization. LM ACC = average accuracy of 8 classification datasets. Lambada Accuracy is excluded from the average due to anomalies

---

Lamba. (PPL).
Wikitext.
Model
#Samples
(PPL)
NE0
30.26
32.28
+DeMEM
33.58
33.13
19.67
357.79
1818857.11
3961.67
+UL
32
7405.89
335.80
3385138. 138.7
6732.11
25647.89
144102.93
2655013335093.0
9621014
124900785
36560950
+UL
128
182274.08 255
1935.31
139501 8915.85
163375.73
747238 74142.82
2072110.35
128824105.70
40390.67
17736.35
41620,78
+UL
256
5446764.99 91
9477.52
47724404.48
22130.00
9320659.24
12115,23
NEO
7.49
16.16
+DcMEM
9.01
16.70
31.33
26.77
20.60
24.67
+UL.
32
6.61
22.39
7.53
20.90
7.087
27.16
747.21
53.23
14.52
+UL
128
4920762.54
770.51
41.42
41.93
342,03
41.51
13.72
61.20
189789
227.06
+UL
256
189367.90 90
91.03
681965.60 60
171.54
705.34
42.59
NE0
5.62
13.93
+D:MEM
6.51
14.15
6.13
19.87
2992343.20
1531.31
+UL
32
10.44
28.07
10.28
17.41
6.23
17.11
17.83
61.20
41.04
63.78
+UL
8.91
16.15
10.58
33.69
53.39
116.89
25,89
0
27.21
+UL
31.10
7.97
26.86
25.32
Table 6: Perplexity Results On Lambada & Wikitext: NEO averaged 5 random samples (s 32, 128, and 256) for
UL. UL = knowledge unlearning, DeMEM DeMemorization.

---

Lamba. (PPL)
Wikitext.
Model
#Samples
(PPL)
OPT 125M
26.02
31.94
+DeMEM
31.14
35.35
6.64
16.41
+DeMEM
7.61
17.39
2.7B
5.11
14.31
+DMMEM
7.61
15.25
Table 7: Perplexity Results On Lambada & Wikitext: OPT For Original LM & DeMEM Since UL produced Infinity.
Baseline Method Hyperparameters
We selected the hyperparameters for UL based (Jang et al., 2022) for NEO models, using the number of epochs required for unlearning until the target sequences meet the forgetting criteria. For OPT models, we used half the number of epochs compared NEO models in specific sizes, as OPT models achieved the same loss as NEO models but fewer epochs.
Memorization's Assumptions
As previously discussed, presenting assumptions to address the memorization problem often leads incomplete solutions. This is evident in the case of differential privacy, which assumes whether the data is private or not. Similarly, UL assumes that the training and evaluation data are memorized. which impractical in real-world applications considering that language models are trained on vast corpora with billions of tokens. Furthermore, fine-tuning an LM in an application involving potentially sensitive/private data poses challenges in splitting the data into sensitive/private and non-sensitive/privatt portions for the purpose of forgetting (Levine, 2021; Porcaro, 2022; Brown et al., 2022). On the other hand, DeMemorization does not rely on assumptions about the training data that need to be unlearned. Instead, we fine-tune the LM to learn universal policy that reduces the relationship between the prefix and suffix. This policy achieves its objective by replacing the token with similar entity that is semantically correct but not directly linked the same prefix, as illustrated in Figure 3. Another assumption is the limited number of samples be unlearned once, which we discussed before.
context
Hardware & Software Dependencies
In order to fine-tune GPT-Neo models of sizes 1 125M and 1.3B, we utilized a cluster of two V100 GPUs, each equipped with 32GB of VRAM. The 125M model required approximately 0.38 minutes per PPO epoch, resulting in total computation time of 3.04 minutes for six epochs. The 1.3B model required slightly longer computation time of 1.68 minutes per PPO epoch, for a total of 13.44 minutes over eight epochs. For the largest variant, GPT-Neo 2.7B, we utilized cluster of four V100 GPUs, each with 32GB VRAM, and employed a sharding strategy with zero 3 (Rasley et al., 2020). Each PPO epoch for this model required 5.125 minutes, resulting in a total computation time of approximately 20 minutes over four epochs. For finetuning those models, we employed the HuggingFace library (Wolf et al., 2019) for training and Pytorch (Paszke et al., 2017) for parallelizing the model. For RL fine-tuning, we employed TRL (Transformer Reinforcement Learning) library(von Werra et al., 2020).


Reviews:
[{"review_id": "kZok3tP4t9", "reviewer": "Reviewer_mR2L", "summary": "", "questions": "- Results for performance under DeMEM are reported as one number across all values for #Samples but\n  different values for memorization score. This is a bit confusing as it is implying there is only\n  one fine-tuning process independent of the number of samples. Was DeMEM not fine-tuned with the\n  same samples as UL?\n\n- My list of reasons to reject includes some important questions whose answer will have a big\n  impact on my evaluation. Please answer those.\n\n- What is \"the KL penalty\" ?\n\n- Line 427 discusses not reporting anomalous results but the reason of why the should be considered\n  anomalous is not provided. Please explain.\n\n- Infinite results suggest numerical instability. One place I could see this happening is if the\n  usual small epsilon was not correctly employed to control numerical singularities. Did the\n  evaluation not include such techniques? Related, at what level of perplexity does the resulting\n  model produce unrecognizable gibberish? I presume it is at a point far below the levels being\n  reported, even the non-infinite ones.", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you, reviewer mR2L, for reviewing our paper. Please consider raising your score if we are able to address some of your concerns.\nPlease find the responses for your feedback below:\n\n> The paper experimentally compares itself against one \"knowledge unlearning\" (UL) baseline that targets specific sets of tokens to discount in fine-tuning. \n\nWe would like to direct your attention to two sections of the paper: Related work from 249 to 260 and Section 5.1.3 Baseline Methods from 433 to 450, where we compared our proposed method with two other baselines: the first relied on preprocessing the data using deduplication, and the second focused on knowledge unlearning (UL method), which is the only one present in the literature. Additionally, we included a detailed comparison discussion of the proposed and deduplication methods' performance in section 5.2.2 of the paper.\n\n> Experiments show that the proposed method offers a better accuracy vs. privacy tradeoff in that while it may not be as effective at unlearning, it significantly better preserves the performance of the model.\n\nThat is correct; however, providing stronger privacy protections for LMs may become meaningless if it requires sacrificing their original capabilities. Thus, it’s a tradeoff. We cannot only rely on achieving high performance in terms of privacy metrics and neglect the quality of the generated suffix, the performance of the LM, or how related it is to the original prefix. However, our method combined with the deduplication method achieves comparable results to the UL method with slight differences while offering better accuracy. Since data deduplication is applied in most of the recent & large language models (Penedo et al., 2023; Touvron et al., 2023; Biderman et al., 2023; Taylor et al., 2022; Scaoet al., 2022; Black et al., 2022), we believe our approach combined with deduplication will effectively mitigate memorization while achieving insignificant degradation(∼0.5%) in the Language model performance. So, we ensure that the generated suffixes are related to the context of the prefix and, at the same time, achieve high forgetting rates. The previous statements and results can be found in Section 5.2.2 and Table 2 of the paper.\n\n>It seems that the methodology aims at unlearning particular instances and evaluates that unlearning on those instances that were unlearned. That is, no sort of generalization of the idea of \"paraphrase\" is tested. The setting is thus: I have a pre-trained model and I know it generates a set of instances S based on their prefix. I don't want it to generate those exact instances but something similar. In this case there may be simpler things to do than fine-tune. For example: detect instances of S and output completions generated by a different model (or just refuse to complete). Alternatively, if S are known ahead of training, remove them from the training data.\nSuggestion: First: is my reading of the evaluation and goals accurate? If so, include some discussion as to why fine-tuning should be used to address specific instance as opposed instance-specific handling (example above). If my reading is not correct, please describe and evaluate (if not already) in the work how the unlearning process generalizes to non unlearned instances.\n\nThe methodology and evaluation setup that you mentioned are correct descriptions, but for the UL method, because they attempt to forget a specific number of samples (S) by training and then evaluating the model's unlearning on the exact samples (S). However, in our approach, we train the model on separate training data (mentioned in Section 5.1.1 from 389 to 404) and then evaluate the unlearning on the samples (S) used in the UL method to compare our results with theirs. Therefore, the test set is fixed in both methods, but the training data in our method is entirely different. Thus, our model generalizes because we train and test it on different sets of data. As a result, this is a strength point that proves the effectiveness of our method's generalizability against UL, not a weakness point. The previously mentioned details of the training/evaluation setup of the proposed method are described in sections 5.2.3 (573 to 585) and 5.1.4.\n\n>What is \"the KL penalty\" ?\n\nFirst, let us define the function of the KL penalty; it computes the KL divergence between the probability distributions of the generations of initialized LM and the finetuned LM. An important point to note is that the KL penalty is not a separate component from the core design of our reward function, as we designed the reward function to include two components: the dissimilarity score and the KL penalty. Each of the reward components has an important role; the dissimilarity score encourages the model to learn a policy that generates different samples from the original prefix, but we have to make sure that the policy does not significantly deviate the generations from the original; to achieve this, we included the KL penalty in the reward; thus, the generated and the original samples will have the same meaning. Employing the KL penalty as a component of the reward function is commonly used in most Reinforcement Learning tasks [1, 2, 3].\n\n>This likely prevents the model degradation that UL exhibits. Could the same be used in UL? Related, I feel like the use of this \"KL penalty\" (at least based on its stated goals) is significant enough that it should be discussed in more detail.\nSuggestion: Discuss whether similar approach to KL penalty can be enabled for UL and if so, what would the experimental results look like?\n\nIf we recall the core detail of the UL approach, it is a method that relies on supervised learning, which means that the training of the model is done in a supervised manner as they maximize the loss of the language model, adding the KL penalty to their approach would completely change their method setup or objective and might impossible to compose it since we propose an RL method that uses a concept of reward function, not loss function. Their methods suffer from the degradation problem, so one of the solutions in our approach was to use the KL penalty along with the BERTScore in the reward In order to overcome it. Proposing the use of the KL penalty in their approach should be a further investigation for the paper's authors; our objective was to compare our results with theirs and overcome the limitations of their work. Again, this should be regarded as a strength point for our proposed method as opposed to the UL method. The details of using the KL penalty are mentioned in lines 299 to 304; however, thank you for your suggestion. We will include more details about it in our revision since it overcomes a considerable limitation of the previous works in the literature.\n\n>Second, there is sufficient room between in memorization results for UL as compared to the proposed method to tune parameters for alternate tradeoffs. That is, tune UL to achieve the same memorization score and compare the resulting performance score. I suspect the number of samples conceptually cannot be tuned as they are also the ones being tested but this still leaves the ability to only unlearn a smaller than the number of samples instances and take the resulting penalty in the memorization score for the others.\nSuggestion: Attempt to fine-tune UL for a comparison of one metric on the same level of the other. Alternatively discuss why this is not possible.\n\nIf we correctly understand your question, we used the official hyperparameters proposed by the authors and are already tuned by them to achieve their objective. Since changing the values of the official hyperparameters will change the policy that their models learned. However, we tried to tune some of these hyperparameters; for example, reducing the number of epochs drastically decreased the dissimilarity score and increased the performance. Additionally, a core component of their method is called \"forgetting threshold,\" which is their empirical definition of forgetting as they consider a token sequence to be forgotten and unsusceptible from extraction attacks when it meets their suggested threshold. Thus, we must keep these tuned hyperparameters, or we will oversight their whole approach. Further details of their approach can be found at [4].\n\n>Results for performance under DeMEM are reported as one number across all values for #Samples but different values for memorization score. This is a bit confusing as it is implying there is only one fine-tuning process independent of the number of samples. Was DeMEM not fine-tuned with the same samples as UL?\n\nAs mentioned in the preceding responses, we finetune our model with separate training data and then evaluate it on various test sets. So, we only finetune the model once and then use it for evaluation on each test set. This proves the generalization of the learned policy, as it is independent of the number of protected samples. The details of this part can be found in lines 140 to 141 and 573 to 581.\n\n>Line 427 discusses not reporting anomalous results but the reason of why the should be considered anomalous is not provided. Please explain.\n\nThe reason why they should be considered anomalous is provided in line 428: “as it shows such high values for perplexity & low value for accuracy for UL baseline.”  Including these values in the average of accuracy & perplexity will make the average score so low for accuracy & high for perplexity of the UL baseline but not for our approach, as it achieves good performance on this dataset, as reported in Appendix E. So for fair comparison we did not add them into the average performance of UL. However, all of the results are included in Appendix E.\n\n>Infinite results suggest numerical instability. One place I could see this happening is if the usual small epsilon was not correctly employed to control numerical singularities. Did the evaluation not include such techniques? Related, at what level of perplexity does the resulting model produce unrecognizable gibberish? I presume it is at a point far below the levels being reported, even the non-infinite ones.\n\nCan you please make the question more clear? Since I did not get what you mean by epsilon. As far as I know, there is no epsilon in equations in our approach and the same for UL. Also, there is no epsilon in the Perplexity equation or accuracy. Infinite and large values for perplexity happen in the UL baseline due to softens the probability distribution and leads to a more uniform distribution. The previous clarification is in lines 601 to 607. Additionally, it is provided by the authors of the paper.\n\n>Why is \"GENERATION AS A TOKEN-LEVEL MDP\" capitalized?\n\nTo highlight the employed environment. \n\n>\"a longer context in a language model can be considered a form of attack\"\nYou mean a longer context can be used to carry out a more effective attack?\n\nYes, by this statement, we mean that a longer context can be used to carry out a more effective attack, as observed by [5] that employing a longer context increases the memorization percentage. We show the results of increasing the context length and how it can affect memorization in section 5.2.5.\n\n>\"Typos Grammar Style And Presentation Improvements\"\n\nThank you for the detailed suggestions. We will incorporate them in our revision.\n\nReferences:\n\n[1]. Ramamurthy, Rajkumar, et al. \"Is reinforcement learning (not) for natural language processing?: Benchmarks, baselines, and building blocks for natural language policy optimization.\" arXiv preprint arXiv:2210.01241 (2022).\n\n[2]. Ziegler, Daniel M., et al. \"Fine-tuning language models from human preferences.\" arXiv preprint arXiv:1909.08593 (2019).\n\n[3]. Ouyang, Long, et al. \"Training language models to follow instructions with human feedback.\" Advances in Neural Information Processing Systems 35 (2022): 27730-27744.\n\n[4]. Jang, Joel, et al. \"Knowledge unlearning for mitigating privacy risks in language models.\" arXiv preprint arXiv:2210.01504 (2022).\n\n[5]. Carlini, Nicholas, et al. \"Quantifying memorization across neural language models.\" arXiv preprint arXiv:2202.07646 (2022).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "+ \"Paraphrasing\" as a goal is an intuitive goal that avoids memorization and perhaps even other\n  undesired outcomes in NLP ML.\n\n+ Reasonable results on multiple LLMs and unlike UL, offering usable models while performing some\n  amount of unlearning.", "reasons_to_reject": "- It seems that the methodology aims at unlearning particular instances and evaluates that\n  unlearning on those instances that were unlearned. That is, no sort of generalization of the idea\n  of \"paraphrase\" is tested. The setting is thus: I have a pre-trained model and I know it\n  generates a set of instances S based on their prefix. I don't want it to generate those exact\n  instances but something similar. In this case there may be simpler things to do than fine-tune.\n  For example: detect instances of S and output completions generated by a different model (or just\n  refuse to complete). Alternatively, if S are known ahead of training, remove them from the\n  training data.\n\n  Suggestion: First: is my reading of the evaluation and goals accurate? If so, include some\n  discussion as to why fine-tuning should be used to address specific instance as opposed\n  instance-specific handling (example above). If my reading is not correct, please describe and\n  evaluate (if not already) in the work how the unlearning process generalizes to non unlearned\n  instances.\n\nUPDATE: See also comment added post-rebuttal.\n\n- The comparison to UL may be unfair.\n\n  Firstly, the presented method includes:\n\n      \"KL penalty is applied per token using a reference model (the pre-trained model before\n       finetuning.) This prevents the fine-tuned model from generating a suffix that deviates too\n       much from the reference language model\"\n\n  This likely prevents the model degradation that UL exhibits. Could the same be used in UL?\n  Related, I feel like the use of this \"KL penalty\" (at least based on its stated goals) is\n  significant enough that it should be discussed in more detail.\n\n  Suggestion: Discuss whether similar approach to KL penalty can be enabled for UL and if so, what\n  would the experimental results look like?\n\nUPDATE: See also comment added post-rebuttal.\n\n  Second, there is sufficient room between in memorization results for UL as compared to the\n  proposed method to tune parameters for alternate tradeoffs. That is, tune UL to achieve the same\n  memorization score and compare the resulting performance score. I suspect the number of samples\n  conceptually cannot be tuned as they are also the ones being tested but this still leaves the\n  ability to only unlearn a smaller than the number of samples instances and take the resulting\n  penalty in the memorization score for the others.\n\n  Suggestion: Attempt to fine-tune UL for a comparison of one metric on the same level of the\n  other. Alternatively discuss why this is not possible.\n\nSmaller points:\n\n- Intro includes the term \"PPO\" but it is not explained until later in the paper. Please include a\n  brief definition at first use.\n\n- Some of the qualitative results in Figure 6 would be nice to include in the main paper.", "questions_to_authors": "- Results for performance under DeMEM are reported as one number across all values for #Samples but\n  different values for memorization score. This is a bit confusing as it is implying there is only\n  one fine-tuning process independent of the number of samples. Was DeMEM not fine-tuned with the\n  same samples as UL?\n\n- My list of reasons to reject includes some important questions whose answer will have a big\n  impact on my evaluation. Please answer those.\n\n- What is \"the KL penalty\" ?\n\n- Line 427 discusses not reporting anomalous results but the reason of why the should be considered\n  anomalous is not provided. Please explain.\n\n- Infinite results suggest numerical instability. One place I could see this happening is if the\n  usual small epsilon was not correctly employed to control numerical singularities. Did the\n  evaluation not include such techniques? Related, at what level of perplexity does the resulting\n  model produce unrecognizable gibberish? I presume it is at a point far below the levels being\n  reported, even the non-infinite ones.", "paper_topic_and_main_contributions": "The paper describes a finetuning method based on reinforcement learning in order to de-memorize\npotentially privacy-sensitive information; that is, prevent a generating model from completing text\nlike \"most most cherished secret is \" with the actual completion of such an instance that was used\nin training. At the core of the technique to use a reward for completing text sequences that\n*differ* from the actual training text instances as measured by a (di)similarity score (in this\ncase as determined by BertSCORE, a similarity metric based on token embeddings in a BERT model).\nThe method also includes a penalty (details not certain) to prevent the fine-tuning from deviating\ntoo much from the original. Together these goals are nicely summarized as training a model not to\ngenerate but to \"paraphrase\".\n\nThe paper experimentally compares itself against one \"knowledge unlearning\" (UL) baseline that\ntargets specific sets of tokens to discount in finetuning. Experiments show that the proposed\nmethod offers a better accuracy vs. privacy tradeoff in that while it may not be as effective at\nunlearning, it significantly better preserves the performance of the model. The UL technique is\nshown to breakdown for sufficiently large sets of samples.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Equation 4 has S_G twice. Related, the definition is a bit vague around the \"precision\" part. If\n  the definition is important, include more details. If not important, consider removing.\n\n- Grammar issue near \"to not hurts\".\n\n- Grammar issue near \"OPT LMs fine-tuned them\".\n\n- Grammar issue near \"became more robust privacy LMs\".\n\n- This sentence:\n\n    \"We employed a subset of the Pile dataset, which was released as a benchmark for training data\n     extraction attacks on large Language Models.\"\n\n  Make it sound like Pile was the benchmark for data extraction attacks. Please rephrase.\n\n- Figure 2 has text which is far too small to be readable. Figure 3 is better but still too small\n  to read.\n\n- Why is \"GENERATION AS A TOKEN-LEVEL MDP\" capitalized?\n\n- Syntax/grammar issue near \"knowledge. as a\". Also, the statement made there is not clear:\n\n    \"a longer context in a language model can be considered a form of attack\"\n\n  You mean a longer context can be used to carry out a more effective attack?"}}, {"review_id": "3J8bszaI1e", "reviewer": "Reviewer_N84C", "summary": "", "questions": "* What data is used for the proposed dememorization approach? This was not made clear.\n* Eq 4: BLEU or SacreBLEU?\n", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you, reviewer N84C, for reviewing our paper. Please consider raising your score if we are able to address some of your concerns. Please find the responses for your feedback below:\n\n>Scope of the work seems limited. Most of the experiments offer comparisons against only a single baseline approach (UL).\n\nWe want to direct your attention to two sections of the paper: Related Work from 249 to 260 and Section 5.1.3 Baseline Methods from 433 to 450, where we compared our proposed method with two other baselines: the first relied on preprocessing the data using deduplication, and the second focused on knowledge unlearning (UL method), which is the only method present in the literature for addressing memorization using the unlearning approach. Additionally, we included a detailed comparison discussion of the proposed and deduplication methods' performance in section 5.2.2 of the paper.\n\n>The intuition and motivations for the proposed approach are not clearly presented. The ‘12 red street’ example that repeatedly appears in the paper is not clear/convincing.\n\nWe aim to address the memorization risk of language models. The recently used approach relies on removing the data points entirely from the model’s parameters. For example, if we have a sentence in the training data, “John lives in Canada.” the suffix “John lives” and the prefix “in Canada.” The memorization problem arises when we give the model the prefix “John lives” and the model outputs “in Canada.” The recent solution tries to replace the true suffix with any other sequence of tokens, even if it is not related to the context of the sentence, which affects the capabilities of the pre-trained model. However, in our method, we tried to tackle this problem using a paraphrasing approach to avoid memorization and, simultaneously output a meaningful sequence related to the context of the sentence. To further illustrate, if we give the input prefix “John lives,” the dememorized model should output “in USA.” or “in Australia.” or any other suffix that is related to the input prefix. Therefore, we define memorization as a relationship between the prefix and the suffix and try to minimize the relationship between them. Consequently, we quantify this relationship using the sacre-bleu metric. The previous statements are mentioned in lines 105–112 and 337–360 in the paper.\n\n>paragraph in line 267 talks about an ‘environment’ which is vague and unclear.\n\nThe core component of our approach is using reinforcement learning to fine-tune the language model to a policy that does not memorize sentences. In the context of reinforcement learning (RL), the \"environment\" is essentially the place or context where an RL agent operates and learns. Think of it as a virtual or simulated world where the agent interacts to learn how to perform tasks. This environment contains all the elements and rules that the agent needs to understand in order to achieve its goals. In our case, the agent is the generative model that takes some actions that can affect the environment; therefore, the learned policy. For the model to converge to the desired optimized policy that meets our goals and objectives, we use the reward function (sacrebleu), which serves as a way to measure the quality of the agent's actions and decisions. It quantifies the desirability of the agent's behavior. The agent's goal is to find a policy (a strategy for selecting actions in different states) that maximizes the expected cumulative reward over time. The preceding statements are mentioned in lines 268-288 of the paper.\n\n>Mathematical descriptions of the method are missing (e.g., KL penalty, value network)\n\nKL penalty, or value network, are the basis of reinforcement learning (RL) training setups. Therefore, we did not feel the need to include the mathematical equations because they are commonly used in every RL task. However, That is a good suggestion. We will say more about this in our revision.\n\n>What is the loss function?\n\nIn terms of reinforcement learning (RL), there is no loss function per se. However, it does have a concept similar to a loss function, although it is typically referred to as the \"reward function.\" In reinforcement learning, the goal is to teach an agent to make sequential decisions in an environment to maximize some notion of cumulative reward. The incorporated reward function in our method is defined in lines 281-288 and equation 3 of the paper, Which is the negative BERTScore combined with the KL penalty.\n\n>Missing Ablation: There was no empirical justification for using the SacreBLEU metric for computing dissimilarity.\n\nThe justification for employing the SacreBLEU metric is based on following the work proposed by [1], which used SacreBLEU to measure the approximate memorization and proved its efficiency. Therefore, we employed it as we adopted the concept of approximate memorization. Thus, we used the negative value of the same metric to measure the dissimilarity because it initially measures the similarity. The previous statements are present in section 2.1, lines 175-182, and section 4.2, lines 319-335. \n\n>While the proposed method harms the pre-trained model less, it is also worse in terms of the privacy metric (SacreBLEU). As a result, it is not clear to me if the proposed method is more favorable. \n\nProviding stronger privacy protections for LMs may become meaningless if it requires sacrificing their original capabilities. Thus, it’s a tradeoff. We cannot only rely on achieving high performance in terms of privacy metrics and neglect the quality of the generated suffix, the performance of the LM, or how related it is to the original prefix. However, our method combined with the deduplication method achieves comparable results to the UL method with slight differences while offering better accuracy. \nSince data deduplication is applied in most of the recent & large language models [2,3,4,5,6,7], we believe our approach combined with deduplication will effectively mitigate memorization while achieving insignificant degradation(∼0.5%) in the Language model performance. So, we ensure that the generated suffixes are related to the context of the prefix and, at the same time, achieve high forgetting rates. The previous statements and results can be found in Section 5.2.2 and Table 2 of the paper.\n\n>Given that most comparisons are focused on UL, I would have expected to see a proper technical description of UL as a preliminary. Further, UL is the only baseline considered and there are no comparisons against other baselines.\n\nRecalling our previous responses, we compare our method with two baselines, and UL is one of them. Additionally, we included a brief description of the approach, its main components, advantages, and limitations. However, we will try to incorporate further information about its details in the revision.\n\n>What data is used for the proposed dememorization approach? This was not made clear.\n\nWe employed a subset of the pile dataset for training. Then, we evaluated and tested the model on other subsets of the pile dataset. So, the training and testing are done on completely different sets of data. Each sample consists of a 50-token prefix and a 50-token suffix; the model takes the prefix as input and then generates a suffix, which is compared to the original suffix using BERTScore. The training set is 13,500 samples, while the test set is 1,500 samples. All the details regarding the data employed in our approach are presented in Section 5.1.1. Can you please further clarify which part of the dataset was not clear in order to answer your question?\n\n>Eq 4: BLEU or SacreBLEU?\n\nMathematically, both BLEU and SacreBLEU are the same. The only difference is from a technical perspective, as SacreBLEU provides a more robust implementation, offering better handling of tokenization, corpus-level processing, and support for multiple languages.\n\n>Writing/clarity needs significant improvement.\n\nThank you for your suggestion, but can you please point out the sections other than the ones that you mentioned before in order to revise them in the revised version?\n\n>Line 125: “explicit, implicit assumptions’ -> why not just say ‘assumptions’?\n\nTo precisely address what assumptions we mean, we use “explicit” or “implicit” assumptions since they are different.\n\n>The paragraph that starts at line 249 seems out of place. The paper is plagued by such structure issues.\n\nThis paragraph summarizes the limitations we found in the literature and how our proposed method addresses them. That’s why it’s included at the end of the related work section.\n\n>line 378: ‘designed to be easy to extract to assess’ - what does this mean?\n\nThis means that the designed dataset is constructed in such a way that the Langauge model can memorize it, so we can assess how the unlearning techniques perform. Since if we used samples that LM does not memorize, we wouldn’t be able to assess the efficiency of the unlearning approaches. \n\n>line 390: ‘100 prefix tokens, 50 prefix tokens’ -> unclear\n\nIn the paper, this line is written as follows:\n“Each sample consists of a 200-token sequence divided into 100 pre-prefix tokens, 50 prefix tokens, and 50 suffix tokens”. So, it’s pre-prefix, not prefix only.\n\n>line 399: ‘pre-prefix’ -> what is this\n\nThe base components of the sample in the data are “prefix” and “suffix.” We further investigated using a longer context to evaluate how the model would perform. Therefore, we added another component to the sample, which is “pre-prefix,” which denotes the 100 tokens preceding the original prefix. Thus, the components of each training sample in the longer context setups are (pre-prefix, prefix, and suffix).\n\n>\"Typos Grammar Style And Presentation Improvements\"\n\nThank you for the detailed suggestions. We will incorporate them in our revision.\n\nReferences:\n\n[1]. Ippolito, Daphne, et al. \"Preventing verbatim memorization in language models gives a false sense of privacy.\" arXiv preprint arXiv:2210.17546 (2022).\n\n[2]. Penedo, Guilherme, et al. \"The RefinedWeb dataset for Falcon LLM: outperforming curated corpora with web data, and web data only.\" arXiv preprint arXiv:2306.01116 (2023).\n\n[3]. Touvron, Hugo, et al. \"Llama: Open and efficient foundation language models.\" arXiv preprint arXiv:2302.13971 (2023).\n\n[4]. Biderman, Stella, et al. \"Pythia: A suite for analyzing large language models across training and scaling.\" International Conference on Machine Learning. PMLR, 2023.\n\n[5]. Taylor, Ross, et al. \"Galactica: A large language model for science.\" arXiv preprint arXiv:2211.09085 (2022).\n\n[6]. Scao, Teven Le, et al. \"Bloom: A 176b-parameter open-access multilingual language model.\" arXiv preprint arXiv:2211.05100 (2022).\n\n[7]. Black, Sid, et al. \"Gpt-neox-20b: An open-source autoregressive language model.\" arXiv preprint arXiv:2204.06745 (2022).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Improving privacy properties of language models is an important problem\n* Method works better than the baseline UL method on some metrics\n* Method is more robust to number of samples to be unlearned and increase in prompting context length\n", "reasons_to_reject": "* Poor clarity (see further comments in Presentation suggestions)\n* Scope of the work seems limited. Most of the experiments offer comparisons against only a single baseline approach (UL). \n* The intuition and motivations for the proposed approach are not clearly presented. The ‘12 red street’ example that repeatedly appears in the paper is not clear/convincing. \n* Description of the proposed approach (sec 4.1) is superficial and missing technical details. I could only get a surface level understanding of the method. \n  - paragraph in line 267 talks about an ‘environment’ which is vague and unclear.\n  - Mathematical descriptions of the method are missing (e.g., KL penalty, value network)\n  - What is the loss function?\n* Missing Ablation: There was no empirical justification for using the SacreBLEU metric for computing dissimilarity. \n* While the proposed method harms the pre-trained model less, it is also worse in terms of the privacy metric (SacreBLEU). As a result, it is not clear to me if the proposed method is more favorable. Table 1 is also hard to interpret. There is no highlighting of best numbers for each metric.\n* Given that most comparisons are focused on UL, I would have expected to see a proper technical description of UL as a preliminary. Further, UL is the only baseline considered and there are no comparisons against other baselines. \n", "questions_to_authors": "* What data is used for the proposed dememorization approach? This was not made clear.\n* Eq 4: BLEU or SacreBLEU?\n", "paper_topic_and_main_contributions": "This paper proposes a method to mitigate privacy risks in language models. Given a pre-trained language model, a Reinforcement learning approach is proposed where the model is encouraged to produce continuations dissimilar to ground-truth continuations. The proposed method is shown to be more effective at addressing privacy risks while minimally affecting the capabilities of the pre-trained model. \n", "excitement": 2, "reproducibility": 1, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "* Writing/clarity needs significant improvement. \n* Line 125: “explicit, implicit assumptions’ -> why not just say ‘assumptions’?\n* Sentence in line 171 is unclear\n* The paragraph that starts at line 249 seems out of place. The paper is plagued by such structure issues. \n* SacreBLEU is claimed as a widely used metric but was not cited\n* line 378: ‘designed to be easy to extract to assess’ - what does this mean?\n* line 390: ‘100 prefix tokens, 50 prefix tokens’ -> unclear\n* line 399: ‘pre-prefix’ -> what is this\n* All the tables are very small and hardly readable\n* 5.1.3: Make it clear that OPT was trained on deduplicated data\n* line 456 has a long sentence which is hard to comprehend\n* 5.1.5: MA was never defined\n* Table 1 is referenced in page 7 but the table appears in page 5\n"}}, {"review_id": "H35M3L45NO", "reviewer": "Reviewer_yMwJ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you, reviewer yMwJ, for reviewing our paper. Please consider raising your score if we are able to address some of your concerns. Please find the responses for your feedback below:\n\n>The title mentioned the Unlearning Technique, but the method part did not give too many relevant descriptions or highlight your own innovation points.\n\nStarting from lines 137 to 143, we define the innovations and findings of this research paper:\n1. Our method achieves comparable results to the baselines in the literature from a privacy wise while offering better accuracy in the performance of general capabilities of the language model.\n\n2. The recent solutions try to replace the true suffix with any other sequence of tokens, even if it is not related to the context of the sentence, which affects the capabilities of the pre-trained model. However, in our method, we tried to tackle this problem using a paraphrasing approach to avoid memorization and simultaneously output a meaningful sequence related to the context of the sentence.\n\n3. Our method is unaffected by changing the number of samples to be unlearned, as the previous method hurts the general LM capabilities when increasing the number of protected samples.\n\n4. The proposed method can generalize to unseen samples, unlike the recent unlearning methods that are required to train and evaluate on the same samples. As a result, it requires training multiple times for new samples to be unlearned. \n\n> The methods section is too simple and doesn't focus on what the title is supposed to show.\n\nThe methodology section starts by describing how we achieve forgetting or dememorization using the dissimilarity policy. In the first subsection, 4.1, we introduce the RL settings and basis for our approach, then demonstrate the chosen reward function to achieve our goal of unlearning. Finally, we present the hyperparameters and the convergence. In the next subsection, 4.2, we define our concept of memorization, which is the approximate memorization, and then what metric we choose to quantify it. Finally, we define the definition of forgetting from our perspective as minimizing the relationship between the prefix and suffix. Can you please point out the part that needs more clarification so we can include it in the revision?\n\n>The authors do not give a comprehensive study of the results and the comparison with other papers or baselines. \n\nWe want to direct your attention to two sections of the paper: Related Work from 249 to 260 and Section 5.1.3 Baseline Methods from 433 to 450, where we compared our proposed method with two other baselines: the first relied on preprocessing the data using deduplication, and the second focused on knowledge unlearning (UL method), which is the only method present in the literature for addressing memorization using the unlearning approach. Additionally, we included a detailed comparison discussion of the proposed and deduplication methods' performance in section 5.2.2 of the paper.\n\n>There is some confusion in tables. For example, in the header of table 1, LM(PPL) goes down, but the actual value goes up. I'm not sure what the down arrow means, is it performance or number?\n\nWe are sorry for any confusion caused by the clarity in the tables; the arrow goes down means lower values are better, while the arrow goes up means higher values are better. Thanks for the suggestion, We will add this clarification in our revision.\n\n>Should pay more attention to design the layout of tables.\n\nThank you for your suggestion, but can you please point out the problems in the layout other than the ones that you mentioned before in order to edit them in the revised version?\n\n>No theoretical proof or guarantee to prove that the data is really be forgotten.\n\nThe forgetting is measured by the similarity metric “SacreBLEU,” which we used based on empirical justification proposed by [1] to capture the approximate memorization. As shown in Tables 1 and 2, negative sacrebleu(the metric that measures forgetting) is lower in the pre-trained language model before applying the dememorization or UL approaches, indicating higher memorization and lower forgetting. While after employing the dememorization or UL, it shows higher negative sacrebleu, indicating lower memorization and higher forgetting.\n\nReferences:\n\n[1]. Ippolito, Daphne, et al. \"Preventing verbatim memorization in language models gives a false sense of privacy.\" arXiv preprint arXiv:2210.17546 (2022).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well-written for the most part and easy to follow. \n2. The experiments are organized well in effectiveness of the Dememorization, the impact of increasing the context length and general language model capabilities of Dememorized model on downstream tasks .", "reasons_to_reject": "1. The title mentioned the Unlearning Technique, but the method part did not give too many relevant descriptions, or highlight your own innovation points.  The methods section is too simple and doesn't focus on what the title is supposed to show. \n2. The authors do not give a comprehensive study of the results and the comparison with other papers or baselines. There is some confusion in tables. For example, in the header of table 1, LM(PPL) goes down, but the actual value goes up. I'm not sure what the down arrow means, is it performance or number? \n3. Should pay more attention to design the layout of tables.\n4. No theoretical proof or guarantee to prove that the data is really be forgotten.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper focuses on the problem of memorization issue in LLM. In this paper, the author proposes a Dememorization framework for mitigating memorization by using a reinforcement learning feedback approach with just a few parameter updates. In detail, this paper uses a prefix as input for the language model, and computes the negative BERTScore to dissimilarity score between the true suffix and generated suffix, which is then used as a reward signal to maximize in training.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Most reviewers agree that the current manuscript proposes a RL-based approach that leverages dissimilarity score between the authentic and generated completions as a reward signal. To enhance the paper attractive, the authors are encouraged to address several points: a) clearly articulate that the primary objective is to reduce memorization during pre-training phase in fine-tuning phase. 2) add more quantitative details in KL-divergence and UnLearning methods in a self-contained manner; 3) incorporate additional unlearning baselines if feasible. Otherwise, justifying why differential privacy methods cannot be applicable to pre-training of LLMs would provide more clarity.
Decision: Accept-Main