Knowledge Unlearning for Mitigating Privacy Risks in Language Models
Authors: Joel Jang
Dongkeun Yoon
Sohee Yang
Sungmin Cha
Moontae Lee
Lajanugen Logeswaran
Minjoon Seo
Venue: Submitted to ICLR 2023 (2023)
Keywords: privacy
large language models
knowledge unlearning
natural language processing

Abstract: Pretrained Language Models (LMs) memorize a vast amount of knowledge during initial pretraining, including information that may violate the privacy of personal lives and identities. Previous work addressing privacy issues for language models has mostly focused on data preprocessing and differential privacy methods, both requiring re-training the underlying LM. We propose knowledge unlearning as an alternative method to reduce privacy risks for LMs post hoc. We show that simply applying the unlikelihood training objective to target token sequences is effective at forgetting them with little to no degradation of general language modeling performances; it sometimes even substantially improves the underlying LM with just a few iterations. We also find that sequential unlearning is better than trying to unlearn all the data at once and that unlearning is highly dependent on which kind of data (domain) is forgotten. By showing comparisons with a previous data preprocessing method known to mitigate privacy risks for LMs, we show that unlearning can give a stronger empirical privacy guarantee in scenarios where the data vulnerable to extraction attacks are known a priori while being orders of magnitude more computationally efficient. We release the code and dataset needed to replicate our results at http://www.omitted.link/.

Under review as a conference paper at ICLR 2023
KNOWLEDGE UNLEARNING FOR MITIGATING PRIVACY RISKS IN LANGUAGE MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
Pretrained Language Models (LMs) memorize a vast amount of knowledge during initial pretraining, including information that may violate the privacy of personal lives and identities. Previous work addressing privacy issues for language mod- els has mostly focused on data preprocessing and differential privacy methods, both requiring re-training the underlying L.M. We propose knowledge unlearning as an alternative method to reduce privacy risks for LMs post hoc. We show that simply applying the unlikelihood training objective to target token sequences is effective at forgetting them with little to no degradation of general language mod- eling performances for larger LMs; it sometimes even substantially improves the underlying LM with just a few iterations. We also find that sequential unlearning is better than trying to unlearn all the data at once and that unlearning is highly dependent on which kind of data (domain) is forgotten. By showing comparisons with previous data preprocessing method and decoding method known miti- gate privacy risks for LMs, we show that unlearning can give a strong empirical privacy guarantee in scenarios where the data vulnerable extraction attacks are known priori while being orders of magnitude more computationally efficient and robust. We release the code and dataset needed to replicate our results at http://www.omitee.lnnk//
INTRODUCTION
Recent work has shown that an adversary can extract training data from Pretrained Language Mod- els (LMs) including Personally Identifiable Information (PII) such as names, phone numbers, and email addresses, and other information such licensed code, private clinical notes, and 128- bit UUIDs (Carlini et al., [021] [ee al., 2022 Huang et al., 2022 Lehman al. 202T). In 2021, an Al chatbot Iruda became the first Al system to be sued for violating the Personal Information Protec- tion Act after generating the exact home addresses and bank account numbers actual individuals
unintentionally (Park] (02I). Heikkilä (022) has also shown that GPT-3 (Brown et al.) 2020), one of the most well known LM currently in commercial use, offered detailed private information about the Fditor-in-Chief of MIT Technology Review including his family members, work address, and phone number. Considering findings that show extracting training data gets easier as LMs scale to larger sizes (Carlini et al.) (022a)) and that it is common practice for practitioners to release billion parameter pretrained LMs for public use (Gao et al. 2020 Black et al. 2021 Zhang et al. 2022). it has become important to provide privacy guarantees for large LMS.
Practitioners are required to delete personal information from the LMs by individuals' request be- cause each individual has the "Right To Be Forgotten (RTBF)" (Mantelero 2013 Graves et al., 2021) and can limit the direct and indirect commercial use of their personal information (Villaronga al. 2018). Previous methods addressing privacy risks for language models attempt to remove all private information from the training data (data preprocessing) (Aura et al. 2006 Dernoncourt et al.)
2017 Lison et al. 202 Kandpal et al. (022) or attempt to design algorithms that ensure differen- tial privacy (DP) (Dwork) 2008 Dwork et al. 2006 Abadi et al. 2016: Anil et al. 2021 Li et al.) 2022
Yu et al. 2022). Both approaches require retraining the underlying LM every time individuals
want to practice their RTBF, which makess them inadequate for large LMs that are extremely costly to retrain. Furthermore, as pointed out by Brown et al. (2022). data preprocessing methods assume private information to be easily identifiable, specified, and removed and DP algorithms can only

---

Under review as a conference paper at ICLR 2023
Proposed Approach
LM
He tran after austication
Re-tran LM with OP Aigorifin A188 expr))
Pertorn a tiee latenn updlutie
Data Preprocassing
Differential Privacy
Taken
Knowledge Unlearning
Corara
Find Remove
27
my Forgetter (RTBF))
Marial Stoee Siogle
5S%, 123--407- 2913
Donito dioecct conit
tazlo Not Wock $5.00
Ilsh
Figure I: Comparison of previous approaches and knowledge unlearning when an individual practices his/her Right-To-Be-Fonn Forgo-Fooooo (RTBF).
guarantee protection for information that has clear privacy borders, which make them inadequate in the real-world scenarios where the standard of privacy might differ by each individuals.
To this end, we propose knowledge unlearning (Figure 1 as an efficient solution that can be applied with just few parameter updates instead of pretraining the underlying LM again. We perform ex- periments on GPT-Neo LMs (125M, 1.3B, 2.7B) (Black et al. 202T) and show that simply changing the gradient descent the opposite direction during language modeling (which can also be seen as maximizing instead of minimizing the loss function) is effective at protecting target sequences from extraction attacks with little no performance degradation on the initial LM capabilities measured via 9 common NLP classification benchmarks (Hellaswag (Zellers et al.)
2019). Lambada (Paperno
al.
2016). Winogrande (Sakaguchi et al. (02T), COPA (Gordon et al. 2012), ARC-Easy (Clark 2018), ARC-Challenge (Clark et al. 2018). Piqa (Bisk et al. (0020). MathQA (Amini et al. and PubmedQA (Jin et al., 2019)) and 4 dialogue tasks (Wizard of Wikipedia (Dinan et al.
et al.
2019
2019
Empathetic Dialogues (Rashkin et al. 2019). Blended Skill Talk (Smith et al. 2020). and
Wizard of Internet (Komeili et al.) 2022)) D. For some cases, knowledge unlearning unexpectedly shows significant improvements in LM performance for some of the benchmarks.
We compare our approach with data deduplication method (Kandpal et al., 2022) and diffcrential privacy decoding method (Majmudar et al., 2022) which are both known mitigate privacy risks, and show the effectiveness of knowledge unlearning by providing a strong privacy protection while being much more efficient and robust. We also provide a general guideline that can be used to quantify the memorization and extraction likelihood of target token sequences and suggest when we can empirically consider them to have been "forgotten". Specifically, we introduce novel metric that measures the extraction likelihood by varying the prefix length of the target token sequence and quantifying how much of the suffix is actually extracted from the LMM
Surprisingly, for knowledge unlearning, we find that it is easier to forget chunk of instances se- quentially rather than trying to forget them all at once. We provide further analysis and show that the difficulty of knowledge unlearning depends heavily on the target data being forgotten, especially the domain of the target data. We also provide empirical examples of performing extraction attacks and how exactly knowledge unlearning provides privacy protection for the LMM
To summarize, our main contributions are fourfold:
We compare knowledge unlearning with two approaches from literature known to mitigate privacy risks: a data preprocessing approach and a Differential Privacy (DP) Decoding ap- proach. We show that our approach results little to no performance degradation gen- eral capabilities (sometimes resulting in improvement) while providing a strong privacy protections in situations individuals practice their RTBF whereas the data preprocessing approach provides weaker privacy protection while being orders of magnitude compu- tationally demanding and the DP Decoding approach results in a severe degradation of modeling performance.
2

---

Under review as a conference paper at ICLR 2023
We perform additional experiments to determine which factors contribute to the difficulty of knowledge unlearning and find that (1) trying to forget many samples at once results in substantial LM performance degradation which can be mitigated by sequentially forgetting chunks of data and that (2) the domain of the target data (Code, License, Wikipedia, etc.) plays critical role in determining how hard they are to forget.
We provide novel metric and general guideline for quantifying the privacy risks for LMS and determine when they should be considered to have "forgotten" given target sequence.
Knowledge unlearning surprisingly seems to make LMs stronger where the extreme cases bring +8.0% (37.6% - 45.6%). +10.1% 57.4% + 67.5%), and +7.9% (62.2% + 70.1%) improvements on Lambada for GPT-NEO 125M, 1.3B, and 2.7B, respectively.
2
RELATED WORK
2.1 PRIVACY METHODS FOR LANGUAGE MODELS
Prior work that tries to mitigate privacy risks for LMs can be divided mainly into data pre/post- processing methods and differential privacy methods.
Data preprocessing aims to sanitize the training data; it aims to get rid of all data that might violate any kind of privacy from the training data prior to training. These methods mostly utilize measures such parsers and classification models that try identify and predict patterns that constitute private information. This is effective at identifying well-formatted private information such social security numbers or special forms of medical notes (Aura al., 2006 Dernoncourt et al., 2017 Lison et al. [021] Kandpal et al. 2022). However, as pointed out by Brown et (2002), considering that private information is mostly context-dependent and sometimes in non-specific format, data preprocessing methods cannot fully claim that they provide privacy guarantees, especially guarantees that match each individual's standards. Methods that attempt to utilize post-processing methods such as applying censorship to the LM outputs still face the same limitations.
(Data) Pre/Post-Processing
In this work, compare our proposed method with data preprocessing approach proposed by Kandpal et al. 2022)) which shows that deduplicating the training corpora before pretraining helps pretrain LMs that show stronger robustness against extraction attacks than an LM pretrained under the same circumstances without deduplicating the pretraining corpora. However, we highlight that this approach, which may still be effective at mitigating the overall privacy risks, is not the most suitable approach when considering a realistic scenario of individuals requesting the removal of their information from the implicit parameters of the LMs.
Differential Privacy Differential Privacy (DP) aims to guarantee that the effect of an individual input on the output of specific function is bounded [Dwork] 2008 Dwork et al.] (006). In the context of deep neural networks, DP, which needs be applied during the training phase, aims to construct models that can provide general guarantees that the individual information within the training data cannot be inferred (Abadi et al. 2016). While DP has shown to be surprisingly ef- fective at fine-tuning LMs (Li et al. 2022 Yu et al. (022), pretraining LMs with DP still suffers from substantial performance gap. expensive computation, and slow convergence (Anil al. (02T). Furthermore, as pointed out by BBoon et al. (2022). DP can only provide limited guarantees for LMs because DP requires unified definition for privacy boundaries, which is inherently impossible for natural language data. Most importantly, in realistic scenario where individuals may practice their Right-To TT--e--Forgee Forgotten (RTBF) dynamically after model deployment, is nontrivial to apply ex- isting descent-based DP algorithms such as DP-SGD to only protection against targeted extraction attacks.
2.2 MACHINE UNLEARNING
Machine unlearning has received attention as an alternative approach to overcome data privacy issues in machine learning (Cao & Yang 2015; Ginart et al.) 2019 Bourtoule et al.) 2021 Graves et al.) 2021). Several studies attempt to explore machine unlearning for deep neural networks (Golatkar
al. 2020 Mehta al. 2022). However, they mostly focus proposing algorithms for image
3

---

Under review as a conference paper at ICLR 2023
classification models where they aim to forget whole class; that is, achieve random performance for specific image classes such as "cats" or "ships". We are the first, to the best of our knowledge, to explore unlearning a specific sequence of tokens for LMs which is quite different set-up from traditional image classification models (~ttens of image classes vs. a sequence of tokens that can each be classified into V E (=0500000). In this work, coin this approach as knowledge unlearning since we are more focused on forgetting specific knowledge represented by sequences of tokens.
Zhou et al. (2022) focus on how forgetting can be leveraged to improve the performance of the un- derlying model. They propose "forget-and-releare that unifies existing iterative training algorithms by selectively removing undesirable information and re-learning good features, helping boost per- formance for the task of image classification and multi-agent emergence communication. The un- derlying assumption that it often easier to define and stop unwanted behavior than teach good behavior. We also show this phenomenon in Section 4 where we unintentionally find unlearning just a few sequences tokens sometimes boosts general LM capabilities.
2.3 MEMORIZATION IN LANGUAGE MODELS
Previous work that explores to which extent LMs have memorized their training data approach the phenomenon with two different viewpoints. Some work view memorization of LMs simply threat to individual privacy (Carlini et al.] 202T; 2022a; Jagielski et al., 2022) and utilize metrics that quantify how much the LMs are susceptible to adversarial attacks. These metrics are mostly dependent on the specific types of attacks such as the membership inference attack (Shokri et al.) 2017) and measure the privacy risks of LMs by quantifying the success rate of these attacks. In our work, we instead focus on more targeted extraction attacks.
Another line of work simply quantifies how much knowledge is accumulated and forgotten during pretraining by extracting relational knowledge about the world (Petroni et al., 2019 Lazaridou et al,, 2021 ang al. 2022b a). This line of work does not view memorization a negative trait, but as a positive one that can be leveraged to extract world knowledge from its implicit parameters and per- form knowledge-intensivee tasks such as question answering training knowledgeable conversation
agents.
Our work is highly related to Jagielski al. (022 's work where they also assert that forgetting can be a relaxed version of differential privacy. However, there are two main differences between our work and theirs. First, they only analyze forgetting as passive form of mitigating privacy, asserting that data seen early in large-scale training obtain privacy benefits, whereas we suggest more active form of forgetting. Second, they only show analysis results with image classification and audio generation models while we specifically focus on large LMs.
KNOWLEDGE UNLEARNING FOR LANGUAGE MODELS
3.1 METHODOLOGY
We propose simply negating the original training objective of minimizing the negative log-likelihood of the token sequences as our main method of knowledge unlearning LMs. Specifically, given a sequence of tokens = (TT),., ), our unlcarning training objective is simply maximizing the following loss function:
T
(f) = Σ (xt((<)))))
(1)
where F't denotes the token sequence x = (.1,,.., 1) and po (Ze (()))) denotes the conditional
probability of predicting the next token to be 2, when given J<t to an LM f with parameters 0.
Prior work refer to this training objective as unlikelihood training and combines it together with the
original loss of minimizing the negative log-likelihood for the final objective of enhancing language modeling quality (Welleck et al., 2020) and few-shot learning for downstream NLP tasks (Tam) et al. 2021). In contrast, simply optimize the unlikelihood training objective since are only concerned with forgetting. While this method seems simple, it is highly effective at forgetting specific token sequences without affecting the overall LM capabilities shown in Section 4
4

---

Under review as a conference paper at ICLR 2023
3.2 QUANTIFYING PRIVACY RISKS OF LANGUAGE MODELS
In this subsection, we introduce two metrics we use to quantify the privacy risks given a specific token sequence and how empirically define the token sequence to be forgotten. In this work, do not utilize metrics such as membership inference attack recall (Shokri et al.) (017) since we are not interested in quantifying the general privacy risks of LMs, but instead the privacy risks on the specific target token sequences.
Extraction Likelihood (EL) We first introduce new metric, EL. Given sequence of tokens (21 m. TT) and an LM f with pre-trained parameters 0, we define EL to be as follows:
(x) EET ( - n OV EcEn-gramata) 1fc E n-grams(b))
(2)
[n-grams(a))
where n-grams() denotes the list of n-grams the given token sequence and fo( denotes the output token sequences from the LM fo when given Zet as input that can have max lengths |aza but may be shorter when the EOS (end-of-sequence) token generated beforehand.
The process of varying the prefix length |æree can be seen varying the strength of adversarial attacks. This based on the assumption that the more prior information is provided about the target token sequence, the easier the LM will be able to extract it. Overall, EL can be seen as estimating the general extraction likelihood since we are measuring the average success rate of varying extraction attacks quantified via getting the n-gram overlap of generated and target token sequences. While previous metrics quantifying the privacy risks of LMs are dependent on specific adversarial attacks, this characteristic of EL allows it to quantify the general likelihood of extraction without any dependency on specific extraction attacks.
We regard be hyper- pppee-parameter that be varied depending the stringency of privacy standards. The higher n is set, the stricter we set the standard for a successful extraction attack.
Memorization Accuracy (MA) We define Memorization Accuracy (MA) follows:
ΣF 1f argmax (x(e))) xz}
MA(x)
(4)
T
MA quantifies how much fa has memorized the given token sequences and was proposed by Tiru- mala al. (2022) analyze the training dynamics of large LMs.
Empirical Definition of Forgetting By utilizing both EL,, and MA, we empirically define spe- cific token sequence æ to be forgotten and is no longer susceptible to extraction attacks when the following conditions are met:
1 Σ MA(x') Σ (x' and MA(x)= S D a'ED'
EL,
(5)
D'
I'ED'
where D' represents a validation corpora not seen during training. In other words, we define be forgotten when the (L)( and MA(æ) reach a value that is lower than the average EL,, and MA on token sequences that were not seen during training.
4 EXPERIMENTS
4.1 MODELS, DATASETS, AND CONFIGURATIONS
Baselines For the experiments, use the GPT-NEO (125M, 1.3B, 2.7B) LMs (Black al. 2021 initially pretrained on the Pile corpora (825GB) (Goo et al. 2020), and the OPT (125M, 1.3B, 2.7B) LMs (Zhang al. 2022), pretrained subset of the deduplicated version of the Pile well
5

---

Under review as a conference paper at ICLR 2023
as other corpora from different domains. For the experiments, we perform unlearning the GPT-NEo LMs and quantify the privacy risks of the target data compared to the OPT LMs to measure how effective our proposed approach is in contrast to deduplicating the training corpora before pretraining the underlying LM Kandpal et al. (2022). We do not use the exact LMs from Kandpal et al. (2022) because the LMs were not open-sourced, and thus the OPT LMS instead. We also consider the Differential Privacy (DP) Decoding (Majmudar et al. (022) as one of the baselines; This approach proposes a decoding strategy that performs linear interpolation of the original logits with the uniform distribution and performs nucleus sampling, which they theoretically show provides DP guarantees. is set as the linear interpolation weight where À 0 performs nucleus sampling from the uniform distribution and À = 1 performs regular nucleus sampling, using the logits as weights during random sampling.
Target Data For the actual target data used to quantify the privacy risks of the LMs, we sample instances from the Training Data Extraction Challenge Challenge[[where where 15,000 examples (each are 200 token sequences long) from 16 different domains of the Pile corpora that are identified to be somewhat easy-to-extire are provided. For our experiments, we randomly sample s samples from the 15,000 examples and make the underlying LM forget the s samples at once. As a default, we show the average results of 5 random samplings of samples for all of our experimental settings. We only provide the average of the 5 samplings and do not separately report the standard deviation. Instead, we provide the results of each individual run in Appendix A
Evaluation Datasets Provding stronger privacy protections for LMs may become meaningless if it requires sacrificing their original capabilities. Thus, while quantifying the privacy risks of LMs, we also quantify the original LM capabilities by evaluating the LMS on 9 different clas- sification tasks quantifying the general capabilities: Hellaswag (Zellers et al. 2019) and Lam- bada (Paperno et al. 2016) benchmarks to measure linguistic reasoning abilities, Winogrande (Sak-) aguchi et al.] 2021) and COPA (Gordon et al., (012) to measure commonsense reasoning abilities and ARC-Easy (Clark et al. 2018). ARC-Challenge (Clark et al. 2018), Piqa (Bisk et al.) 2020).
MathQA (Amini et al. 2019). PubmedQA (Jin et al. 2019) benchmarks to measure the scientific reasoning abilities. We also evaluate on 4 dialogue tasks (Wizard of Wikipedia (Dinan et al. 2019), Empathetic Dialogues (Rashkin et al.) 2019), Blended Skill Talk (Smith et al. 2020 and Wizard of Internet (Komeili et al. (2022) to evaluate the generation capabilitiess of the LMs. We use the test set for Lambada and the validation set for the rest of the datasets. We also show the results of measuring the perplexity on the validation corpora of Pile and Wikitext in Appendix [B] We do not include measuring perplexity as one of the main evaluations because perplexity might not be the most suitable metric for quantifying general LM performance, especially in the case of unlearning (further explanation given in Appendix B We evaluate DP Decoding only on the 4 dialogue tasks because the decoding strategy cannot be applied for performing the classification tasks which is evaluated by utilizing a verbalizer.
Configurations For the learning rate, we set it to 5e-5. We show the effect of varying learning rates in Appendix [ We use a constant learning rate scheduling throughout the run. We fix the global batch size to be the same as s (how many samples global batch sizes smaller than s proved to degrade general LM capabilities ? For EL,, we set n= 10 which means EL measures the extraction likelihood of extracting n consecutive tokens of varying extraction attack | For calculating EL and MA, we use naïve greedy decoding strategy. We set both the dropout and weight decay rates to O. Lastly, while we provide guideline of empirically deciding a single token sequence to be forgotten in Section 3.2, for considering a chunk of token sequences to be forgotten, we use the average EL and MA as an approximation of the individual EL and MA.
forgotten once) because having
4.2 MAIN EXPERIMENTS
Forgetting Threshold First, we show how we get the Forgetting Threshold for EL and MA, the values where we consider the token sequence to be forgotten and unsusceptible from extraction
http:://gihhhh..mooogg.ee. wwwpee/eeeeeeeemmeettriooo...eee
"In Section 4.3 We show that plays critical role in determining how much the unlearning will degrade in general capabilities of the LM since 128 shows result much degradation. Method to mitigate this is proposed in Soction 4.3 well.
"e set the n value to 10 since we empirically consider an extraction to he successful when 10 consecutive token sequences are successfuly generated by the I.M. We show varying the n with values from [5. 10,20,0000 Appendix(1]

---

Under review as a conference paper at ICLR 2023
Table 1: Forgetting Threshold for GPT-NEO LMs
El Thresbold
() Threshold
Model (Size)
GPT-NEO (125M)
29.91
GPT-NBO (1.3B)
5.68
33.27
NEO (2.7B)
5.53
34.02
Main Results showing the average of 5 random sampling of a 32 (forgetting 32 samples once). OPT represents the LM with deduplication applicd. NEO denotes the initial GPT-NEO LMM, NEo DPD* represents applying DP Decoding strategy by varing the to match the forgetting criteria, NEo UL represents performing unlearning on the initial NEO until it provides a stronger security for the target sequences than OPT, NEO UL represents performing unlearning on GPT-NEO until target sequences match the forgetting criteria, LM Avg. denotes the average accuracy of the 9 classification datascts, and Dialogue Avg. denotes the average score of the dialogue datasets, Best comparable performances are bolded and second best underlined.
Table 2:
Model
#
EL (%r
MA
LM Avg. (ACC)
Dialogue Avg. (FI)
Epoch
Params
(%) 4
OPT NEO
125M
$2.9
42.4
10.2
125M
30.9
77.4
43.4
NEO DPD
125M
0.0
27.4
NIA
7.3
NEO + UL
125M
50.1
42.6
8.0
11.0
+ UI.
125M
27.4
OPT
1.3B
23.3
67.1
50.6
12.4
NEO
1,3B
67.6
92.2
49.8
11.5
NBO + DPD
1.3B
0.0
21.4
NIA
7.1
NEo + UL
1.38
11.0
62.2
49.7
11.6
UI
OPT
2.7B
25.6
69.2
52.7
12.9
NEO
2.7B
70.4
93.4
52.3
11.5
NBO + DPD
2.7B
0.0
24.2
NIA
6.9
NEO + UL
2.78
13.0
66.0
52.3
12.5
5,4
UL*
attacks, for model sizes of GPT-NEO LMs in Table
[ For D', we perform weighted sampling (same domain distribution as the Pile training corpora) of 10,000 instances each with token lengths 200 from Pile validation corpora and measure average 1L and MA (Equation 5). which are empirically set as the Forgetting Threshold values.
Main Results Table 2 shows the main results of performing unlearning on LMs of varying sizes and the baselines. While we provide the average performances of the random samplings in Table 2. we provide each individual runs in Appendix A for reference.
We highlight five main observations regarding the results. (1) OPT LMS show much lower EL and MA than GPT-NEO LMs, confirming that deduplicating the pretraining corpora is indeed help- ful for mitigating privacy risks. (2) NEO + DPD+ enables effective protection against extraction attacks demonstrated via the lowest EL and MA score; however, it brings severe degradation of generation capabilities measured via the Average Fl score of the 4 dialogue generation tasks, (3) NEo UL* results in severe degradation of both classification and dialogue tasks for the 125M, only severe degradation of dialogue tasks for 1.3B LM while for the 2.7B LMs, enables re- taining most of its previous capabilities. (4) While the LMs scale to larger sizes, it takes fewer epochs for the target sequences to be forgotten. Together with (3), this implies that larger LMs are strong wnlearners. (5) While NEO + UL provides stronger privacy protection than OPT without sacrificing its performance from NEO for the 2.7B LM, it is much more computationally efficient (3,500.000x 000x) than re-training the underlying LM, which is required for all data preprocessing ap- proaches
Computational efliciency
which calculated Total Training Tokens Parameter Size) Brown
8020
FLOPSs for OPT LMs were estimated using information from Zhung (0022) We provide the FLOPS for the methods in Appendix

---

Under review as a conference paper at ICLR 2023
Any. ww..frrraaoo
Seeplo Regette et Oree
('Serptoo Sopenially Regetio
(a) Batch Unlearning
(b) Sequential Unlearning
Figure 2: Average LM performance on the classification benchmarks when varying the total number of samples forgotten at once is shown in (a) and the average LM performances when the 128 samples are divided into chunks and are forgotten sequentially is shown in (b). The lines denote the average performances of 5 random samplings and the standard deviation is shown the shaded regions. The dotted lines in (b) denotes the 128 performance in (a) for comparison purposes.
Overall, results show unlearning to be an effective approach to providing a strong privacy protection while retaining and sometimes even improving general LM capabilities.
Sequential Unlearning is more Stable than Batch Unlearning. We show the effect of varying s (the of data instances to be forgotten at once) in Figure 2h across model scales. We denote this approach batch unlearning. As shown by the 128 results, it is harder to forget more samples at once, resulting in substantial degradation of average LMM performance regardless of how large the LM is. Since I 32 does not show much degradation, we explore if sequentially unlearning can be a solution. In Figure 2b, we show the result of dividing the 128 samples into 4 chunks of 32 and performing sequential unlearning; we unlearn each chunk time until the chunk reaches the forgetting threshold. Surprisingly, as shown by the performance gap at s = 128 between the dotted lines (the 128 performance of Figure ( and straight lines, the end result vastly different even though exactly the same instances were forgotten. Sequential unlearning shows almost no degradation of average LM performance. In Appendix G we show that chunks once forgotten stay forgotten and that later chunks are forgotten much faster compared to the initial chunk. This result hints at the generalization of unlearning, which we do not further explore in the scope of this work. The result also suggests that knowledge unlearning can be continually applied to LMs when needed.
4.3 ANALYSIS OF KNOWLEDGE UNLEARNING
Providing Better Intuition of What Exactly Happens During Knowledge Unlearning. To show exactly what happens to the LM during knowledge unlearning, we show how the performance of each of the LM benchmarks changes (1.3B) model (each run with 8 1) in Figure 3 As shown in the figure, the LM performance for each benchmark varies tremendously on which sample is chosen be forgotten. Furthermore, the ending time of each run is different, indicating that some samples are forgotten faster than others. We also show empirical examples of performing actual extraction attacks with prefix length of 100 in Appendix
we perform 10 runs of unlearning to the GPT-NEO
Towards Understanding Why Some Instances are Harder to Forget To measure why some instances are harder to forget, we perform 5 random samplings of s = 8 from 8 different domains from the Training Data Extraction Challenge D and perform unlearning
the GPT-NEO 1.3B LM. We also show the results each individual run in Appendix A As shown in Table despite undergoing the same number of token updates (10 epochs of unlearning), different domains result in vastly different outcomes; ENRON FMAILS results in the average LM performance degradation of only -0.4% while USPTO BACKGROUNDS results in -4.5% degradation. Furthermore, the final EL10 varies depending on the domain, suggesting that some domains (c.g.. FREELAW) are harder to forget than others. Lastly, domains that are more structured, which means the data consists of some kind of patterns such as a list of emails (ENRON EMAILS) or code (GITHUB (CODE)). seem to result in less degradation of LM performance contrast to domains that are more unstructured, which means the data consist of mostly raw English text such review for journal submission (PUBMED
8

---

Under review as a conference paper at ICLR 2023
37.2%%
60%
15.5%
17,000
57%
Anrr
335.0%
34.5%
36.50%
36.25%
54.0%
12
Epacdo
(a) Hellaswag
(b) Lambada
(c) Winogrande
14.5%
60.0%
27.50%
51.5%
64.9%
25.00%
65.3%
23.75%
Prochs
Ppoche
Plochs
(d) COPA
(e) ARC-Easy
ARC-CRRCallee
70.
15.0%
10.0%
82.1%
50.0%
69.0%
47.5%
Ppochs
Ppeche
(g)
Piqa
(h) MathQA
(i) PuhMedQA
Figure 3: Performance on the 9 classification benchmarks as we perform 10 different unlearning runs on GPT-
NEO 1.3B where = 1.
Table 3: Unlearning GPT-NEO 1.3B on token sequences sampled from 8 different domains. We fix the epoch 10, set 8 and show the result of the average of random samplings. Italicized denotes the A from INITIAL.
Initial EL10
Final EL1c
Helln. (ACC)
Lamha. (ACC)
Wine. (ACC)
COPA (ACC)
ARC-E (ACC)
ARC-C (ACC)
Piqa (ACC)
MathQ (ACC)
PubQ (ACC)
Avg. (ACC)
Domains
INITIAL
37.0
57.4
54,9
70.0
56.6
25.8
70.4
53.8
49.8 (0.0)
FREELAW
60.4
12.1
37.2
52.2
53.9
68.4
55.5
26.2
70.1
21.7
53.5
48.7 (-1.1)
GIT. (CODE)
63.9
0.6
37.3
53.4
54.4
69.2
56.3
26.0
69.9
21.5
49.8
48.7 (-1.7)
GIT. (LICENSE)
75.8
0.0
37.1
$2.0
54.2
69.0
26.4
70.1
21.8
51.8
48.8 (-1.0)
ENRON EMAILS
77.3
0.0
57.2
54.8
26.3
21.8
53.1
49.4 (-0.4)
(-1.9)
CC
(-2.0)
59.4
0.0
33.7
53.5
21.5
50.3
45.3 (-4.5)
PUBMED CENT.
71.8
0.0
36.5
44.5
54.1
69.6
55.6
24.8
70.0
21.9
46.4
47.0 (:2.8
CENTRAL). We provide examples from each domain in Appendix However, further analysis of understanding exactly which components make unlearning work should be made in future work.
CLOSING
In this paper, propose knowledge unlearning method for mitigating privacy risks in LMs that provides a strong privacy protection with littl to no degradation of general LM capabilities measured by evaluating on common LM classification benchmarks and 4 dialogue benchmarks for the larger sized LMs. As large LMs expand their use cases, potentially affecting the daily lives of people, the research community should make sure that the privacy of individuals is not violated intentionally or unintentionally by the knowledge stored in the implicit parameters of these models. Since is inherently impossible to prevent and predict all future privacy concerns prior to pretrain- ing the LM,, we suggest the community consider knowledge unlearning for ensuring privacy upon individuals" requests post hoc pretraining..
We provide some limitations our Appendix

---

Under review as a conference paper at ICLR 2023
REFERENCES
Martin Abadi, Andy Chu, lan Goodfellow, Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pp. 308-318, 2016.
Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik
Yejin Choi, and Hannaneh Hajishirzi. MathQA: Towards interpretable math word problem solving with operation- bassed formalisms. In Proceedings of the 2019 Conference of the North American Chapter of the Associ- ation for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 2357-2367, Minneapolis, Minnesota, June 2019. Association for Computational Lin- org/N19 1245
guistics. doi: 10.18653/v1/N199 1245. /ttpaa/ttt/a////aaaaatttiiii /tti:::s///cc.anhoooogy...
Rohan Anil, Badih Ghazi, Vineet Gupta, Ravi Kumar, and Pasin Manurangsi. Large-scale differen- tially private bert. arXiv preprint
2021.
Tuomas Aura, Thomas A Kuhn, and Michael Roe. Scanning electronic documents for personally identifiable information. Proceedings the ety. pp. 41-50, 2006.
the 5th ACM workshop on Privacy electronic soci-
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, al. Piqa: Reasoning about physical com- monsense natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. Gpt-neo: Large scale autore- gressive language modeling with mesh-tensorflow. If you use this software, please cite it using these metadata, 58, 2021.
Lucas Bourtoule, Varun Chandrasekaran, Christopher A Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang. David Lie, and Nicolas Papernot. Machine unlearning. In 2021 IEEE Symposium Security and Privacy (SP), pp. 141-159. IEEE, 2021.
Hannah Brown, Katherine Lee, Fatemehsadat Mireshghallah, Reza Shokri, and Florian Tramèr. What does it mean for a language model to preserve privacy? arXiv preprint arXiv:2202.05520, 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Yinzhi Cao and Junfeng Yang. Towards making systems forget with machine unlearning. In 2015 IEEE Symposium on Security and Privacy, pp. 463-480. IEEE, 2015.
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 27), pp- 2633-2650 2021.
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. arXiv preprint arXiv:2202.07666, 2022a.
Nicholas Carlini, Matthew Jagielski, Chiyuan Zhang, Nicolas Papernot, Andreas Terzis, and Florian Tramer. The privacy onion effect: Memorization is relative. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Sys- tems, 2022b. URL http:://///peeeeee/eeem net http://ooummmddddreeee Bid=ErUILrGaVEE
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. ArXiv, 1bs/1803.55 2018.
Franck Dernoncourt Ji Young Lee, Ozlem Uzuner, and Peter Szolovits. De-identification of patient notes with recurrent neural networks. Journal of the American Medical Informatics Association, 24(3):59--606, 2017.
10

---

Under review as a conference paper at ICLR 2023
Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. Wizard of wikipedia: Knowledge-powent conversational agents. In Internationaa Conference on Learning Representations, 2019.
Cynthia Dwork. Differential privacy: A survey of results. In International conference theory and applications of models of computation, pp. 1-19. Springer, 2008
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory of cryptography conference, pp. 265-284. Springer, 2006.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint
2020.
Antonio Ginart, Melody Guan, Gregory Valiant, and James Y Zou. Making ai forget you: Data deletion in machine learning. Advances in neural information processing systems, 32, 2019.
Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Eternal sunshine of the spotless net: Selective forgetting in deep networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7304-9312, 2020.
Andrew Gordon, Zornitsa Kozareva, and Melissa Roemmele. SemEval-2012 task 7: Choice of plau- sible alternatives: An evaluation of commonsense causal reasoning. In *SEM 2012: The First Joint Conference on Lexical and Computational Semantics Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012), pp. 394-398, Montréal, Canada, 7-8 June 2012. Asso- ciation for Computational Linguistics. URL StRphttss:// httptttsss///aaaattaaae acl httpa/cannnntooo...ooooooooooot. org/ S12-1052
Laura Graves, Vineel Nagisetty, and Vijay Ganesh. Amnesiac machine learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 1 11516-11524, 2021.
Melissa URL
Heikkilä.
What
does gpt-3 "know" about me?nbsp: Aug 2022.
Jie Huang, Hanyin Shao, and Kevin Chen-Chuan Chang. Are large pre-trained language models leaking your personal information? arXiv preprint arXiv:2205. 12628, 2022.
Matthew Jagielski, Om Thakkar, Florian Tramèr, Daphne Ippolito. Katherine Lee, Nicholas Carlini, Eric Wallace, Shuang Song, Abhradeep Thakurta, Nicolas Papernot, et al. Measuring forgetting of memorized training examples. arXiv preprint arXiv:2207.00092 2022.
Joel Jang, Seonghyeon Ye, Changho Lee, Sohee Yang. Joongbo Shin, Janghoon Han, Gyconghun Kim, and Minjoon Seo. Temporalwiki: A lifelong benchmark for training and evaluating ever- evolving language models. arXiv preprint arXiv:2204. 1,2/1, 2022a.
Joel Jang, Seonghyeon Ye, Sohee Yang. Joongbo Shin, Janghoo Han, Gyeonghun KIM, Stan-
ley Jungkyu Choi, and Minjoon Seo. Towards continual knowledge learning of language models. In International Conference on Learning Representations, 2022b. URL https:
Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. Pubmedqa: A dataset for biomedical research question answering. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 2567-2577, 2019.
Nikhil Kandpal, Eric Wallace, and Colin Raffel. Deduplicating training data mitigates privacy risks in language models. arXiv preprint ar arXiv:2202.065.2 266.9969..., 2022.
Mojtaba Komeili, Kurt Shuster, and Jason Weston. Internet-augmented dialogue generation. In Proceedings of the 60th Anmual Meeting the Association for Computational Linguistics (Volume
1: Long Papers), pp. 8460-840 8478, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: w11113//1/2222...oo 1/20022.aclllongg5599
URL https: hetps://acllee http://aaaaatnooooo/oooooo oogg
2022. allaallllnoo..o... 579
11

---

Under review as a conference paper at ICLR 2023
Angeliki Lazaridou, Adhi Kuncoro, Elena Gribovskaya, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Gimenez, Cyprien de Masson d' Autume, Tomas Kocisky. Sebastian Ruder, et al. Mind the gap: Assessing temporal generalization in neural language models. Advances in Neural In- formation Processing Systems, 14429348-29366 2021.
Katherine Lee, Daphne Ippolito. Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison- Burch, and Nicholas Carlini. Deduplicating training data makes language models better. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8424-8445, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.1 11111863/1112222aal-ooa
URL https://aclantee
org/
2022.ac1-lona 557.
Eric Lehman, Sarthak Jain, Karl Pichotta, Yoav Goldberg, and Byron C. Wallace. Does bert pre- trained on clinical notes reveal sensitive data? In NAACL-HLT, pp. 946-959, 2021. URL https: http:::/ddoo..r//////
naacl-main.
Xuechen Li, Florian Tramer, Perey Liang, and Tatsunori Hashimoto. Large language models can be strong differentially private learners. In International Conference Learning Representations, 2022. URL
Pierre Lison, Ildikó Pilán, David Sanchez, Montserrat Batet, and Lilja Øvrelid. Anonymisation models for text data: State of the art, challenges and future directions. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 4188-4180000 Online, August 2021. Association for Computational Linguistics. doi: 10. 10.18653/11/2221..cl hwtp://22202...1ooo3322222
URL https: http://///acaatthoooogso
w0w..aalllo.nne.. 323
Jimit Majmudar, Christophe Dupuy, Charith Peris, Sami Smaili, Rahul Gupta, and Richard Zemel. Differentially private decoding in large language models. arXiv preprint arXiv:2205. 13621,, 2022.
Alessandro Mantelero. The eu proposal for a general data protection regulation and the roots of the 'right to be forgotten'. Computer Law & Security Review, 29(3):229-235, 2013.
Ronak Mehta, Sourav Pal, Vikas Singh, and Sathya N Ravi. Deep unlearning via randomized condi- tionally independent hessians. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 10422-10433 2022.
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The LAMBADA dataset:
Word prediction requiring broad discourse context. In Proceedings of the 54th An- mual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp- 1525-1534, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10.18653/v1/P16- 1144, URL httttt:://eettt ant http://cl.aattollloo.ooorr///--.
Jasmine Park. South korea: The first case where the personal information protection act was applied to an ai system, May 2021. URL shorturl.
Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. Language models as knowledge bases? EMNLP, 2019.
Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. Towards empathetic open- domain conversation models: A new benchmark and dataset. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 2370-5381, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 1861866//vll 1/111-11199991-11 //ac aaccantoolll www.yyoorg/P9911111
URL https:
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference at- tacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP), pp. 3-18. IEEE, 2017.
12

---

Under review as a conference paper at ICLR 2023
Eric Michael Smith, Mary Williamson, Kurt Shuster, Jason Weston, and Y-Lan Boureau. Can you put it all together: Evaluating conversational agents' ability to blend skills. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 2021-2030, On- line, July 2020. Association for Computational Linguistics. doi: 10. 00.18653/1122222aclll 183. URL httpt::////...... anthol o attp://222....mmmaamm... 183
Derek Tam. Rakesh R. Menon, Mohit Bansal, Shashank Srivastava, and Colin Raffel. Improving and simplifying pattern exploiting training. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 4980-499 Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v122021. emnlp-main.407, URL httttt:://acaaannhlllo/o////////
org/2021 emnlp ma i 407
Kushal Tirumala, Aram Markosyan, Luke Zettlemoyer, and Armen Aghajanyan. Memorization without overfitting: Analyzing the training dynamics of large language models. arXiv preprint arXiv:2205.107700 2022.
Eduard Fosch Villaronga, Peter Kieseberg, and Tiffany Li. Humans forget, machines remember: Artificial intelligence and the right to be forgotten. Computer Law & Security Review, 34(2): 304-313, 2018.
Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. Neural text generation with unlikelihood training. In International Conference on Learning Rep- resentations, 2020. URL htl.httss: http:///peeeevveeeeee ew ww..et.///..ee........m
net. wwwe/ffruueeee forum? wewerrrmmmmneeeeeeeeeee G=
Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi, Huseyin A Inan, Gautam Kamath, Janardhan Kulkarni, Yin Tat Lee, Andre Manoel, Lukas Wutschitz. Sergey Yekhanin, and Huishuai Zhang. Differentially private fine-tuning of language models. In International Conference on Learning Representations, 2022. |RLittps:: : /ttp///ppeeeeeeeeeeeeeeeeee net/ tareet/ffrrmne www.eoiouummmn...... JECO
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence? arXiv preprint 1rXiv:1905.0788: 2019.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et Opt: Open pre-trained transformer language models. arXiv preprint 2rXiv:2205.00000 2022.
Hattie Zhou, Ankit Vani, Hugo Larochelle, and Aaron Courville. Fortuitous forgetting in con- nectionist networks. In International Conference on Learning Representations, 2022. URL https::/cpenzeview net/ forum?id=eiii 2YsE
A
FULL RESULTS
We provide all of the results for the 5 random samplings for our main experimental setting in Table 4 and the full results for the domain analysis setting Table We also provide the evaluation of the 4 dialogue tasks for s = 32 for all model sizes in Table 6
B MEASURING PILE AND WIKITEXT PERPLEXITY
Table shows the results of measuring perplexity on 500 samples from the validation set of Pile and Wikitext corpora on the LMs from the main experimental setting (Table Results show that LMs that underwent knowledge unlearning show higher perplexity while main experimental table (Table ( does not show degradation of performance on 9 different LM benchmarks. We believe the discrepancy to be due to the inherent attributes of performing unlearning: since we are doing gradient ascent, we are likely softening the probability to generate each token from the vocabulary, giving it a more uniform distribution that will inevitably result in a higher perplexity. However, since it does not show much degradations in the LM benchmarks, also means that the argmaz of the most likely token to be generated has not changed much. However, further exploration of what exactly knowledge unlearning does to the representations of the LM should be done in future work.
13

---

Under review as a conference paper at ICLR 2023
Avg. LM Performance
Avg. LM Performance
60
Avg. Elaj
Avg. Fla
40-
20
0-
0
2
Epochs
Epuchs
(a) IE-4
(b) 8E-5
Avg. Performance
Avg. LM Performance
60
Avg. El.3o
60
Avg. Elag
40
40
20
20-
2
Epochs
Epochs
(c) SE-5
(d) 3E-5
Avg. LM Performance
Avg. ELi
40
20
Epachs
(e) IE-5
Figure 4: Varying the learning rate for unlearning the GPT-NEO 1.3B with s = 32. We report the average of 3 random samplings and display the standard deviations as the shaded regions. Red dotted lines denote the memorization accuracy forgetting threshold of the 1.3B model reported in Table
C COMPUTATION COMPARISON BETWEEN DEDUPLICATION AND KNOWLEDGE UNLEARNING
We show the FLOPs of pretraining OPT denoted as DEDUPLICATION and the average FLOPPs of performing knowledge unlearning until 32 token sequences reach the Forgetting Threshold denoted as UNLEARNING in Table We calculate FLOPs by (6 x Total Training Tokens x Parameter Size) following Brown et al. 2020).
D VARYING THE LEARNING RATE
In Figure 4 we show the results of varying the learning rate for knowledge unlearning where we fix the total epoch to 10 and perform 3 random runs with 32 on the GPT-NEO 1.3B. Overall, we observe that higher learning rates lead to faster forgetting, but with substantial LM performance degradation. While lower learning rates retain the LM performance, they fail to meet the Forgetting Threshold within 10 epochs. Thus, we set the learning rate to 5e-5 for our experiments to get the best trade-off.
14

---

Under review as a conference paper at ICLR 2023
40
60
30
20
20-
10
3
6
9
12
15
18
3
12
Epochs
Epuchs
(a) 125M
(b) 1.3B
EL10
40
EL
Elap
Elap
20
Avg. LM Performance
Epochs
(c) 2.7B
Figure 5: Additional results sequential unlearning for GPT-NEO 125M, 1.3B, and 2.7B. Red dotted lines denote the memorization accuracy forgetting threshold reported of each model in Table
E TEXT EXAMPLE FROM EACH DOMAIN
We show an example token sequence from each of the 8 domains used for the analysis section in Table 9
F MORE EXAMPLES OF PERFORMING EXTRACTION ATTACKS
In addition to the extraction attack example shown in the analysis section, we provide 3 additional examples to provide readers with more empirical examples of how knowledge unlearning ensures protection against extraction attacks in Table 10
G ADDITIONAL RESULTS OF SEQUENTIAL KNOWLEDGE UNLEARNING
We show how the E0 of each individual chunks and the average LM performance change as we perform sequential unlearning in Figure 5 Results show that the chunks that are forgotten stay forgotten and that later chunks are forgotten much faster (one or two epochs) compared to the initial chunk. We hypothesize that this might be because of the similarity of the token sequences from the 15,0 5,,000 examples from the Training Extraction Challenge Benchmark. Also, this result hints at the generalization of unlearning, which we not further explore because of the scope of this work.
H THE EFFECT OF VARYING N FOR EXTRACTION LIKELIHOOD (EL) METRIC
First we show the Extraction Liklihood (EL) Forgetting Threshold values for (==5,10.20,400 by measuring the value the 10,000 validation instances unseen during training in Table [T] Next, we show the average LM performance (on the classification benchmarks) where we perform un- learning the LM on 32 samples until the target token sequences forgotten (the EL MA value
15

---

Under review as a conference paper at ICLR 2023
both lower than the threshold values) in Table 12 Performance shows the average of 5 random samplings.
I
LIMITATIONS
While we provide privacy guarantee through unlearning, our Forgetting Threshold is dependent on which data samples chosen D'. Furthermore, varying the prefix length can be seen as naive way of varying the strength of the extraction attacks. real-world scenario, extraction attacks may be more complicated and may require other prevention methods. Also, could not directly compare our approach with Differential Privacy (DP) (Anil et al. 2021) approach because there are no open-sourced LMs pretrained with DP algorithm. We could not replicate the pretrainig phase because of the heavy computational resources needed to pretrain an LM with DP which is estimated require thousands of GPU hours. We leave this comparison for future work. Finally, recent work (Carlini et al. 2022b) has suggested that machine unlearning (for the vision domain) can bring negative effects harming the privacy of other users. Future work should explore this phenomenon in the setting of performing unlearning large language models as well.
16

---

Under review as a conference paper at ICLR 2023
Table 4: All of the individual runs for the Main Results
Model (+)
Params
El.yo (96)
MA (16)
Hella, (MOC)
Lambs. (ACC)
Wins. (MOC)
COPA (ACC)
ARC-E (NCC)
ARG-C (MOC)
Plaa (NOC)
MathQ
(ACC)
(ACO)
Pulo
Arg. (ADC)
Egurh
Noo
125M
30.9
71.4
28.3
37.6
51.8
62.0
45.
63.3
+0.2
+8.0
+1.9
+3.0
+0.0
+2.2
+0.0
+3.3
10.0
+2.0
125M
3.1
28.1
08.1
41.0
52.5
62.0
43.2
21.0
63.0
22.8
57.6
43.5
14.0
125M
0.0
27.6
28.3
26.9
50.8
67.0
42.3
23.7
62.5
42.1
NEo UL (a
12554
0.0
27.1
28.3
42.1
52.5
44.1
62.6
12.5
43.7
125M
38.3
$2.0
125M
0.0
28.1
28.4
33.9
51.5
21.7
62.5
43.2
125M
0.9
29.8
27.8
46.1
51.9
$2.0
37.4
19.7
00.5
22.3
41.5
125M
0.0
28.6
27.4
2.5
49.4
59.0
38.6
60.3
21.2
36.2
NEO UL (a
125M
9/6
30.9
93.4
37.3
125M
2.6
28,9
27.6
29.9
52.4
5000
36.5
22.2
39.5
125M
0.0
28.4
27.6
6.7
49.7
61.0
42.5
61.0
35.1
125M
0.0
28.5
27.6
35.0
51.8
51.0
37.6
60.3
22.4
40.1
125M
30.9
Nno Ul.
125M
0.3
29.6
28.0
52.2
55.0
40.2
21.9
42.0
125M
5.0
25.3
27.4
1.3
49.6
65.0
37.6
59.3
21.2
35.5
125M
0.0
28.2
27.99
5.3
50.5
61.0
41.5
22.4
60:3
21.5
51.4
55.0
18.0
125M
78-4
30.3
42.3
53.7
$6.0
57.6
121M
27.1
27.0
17.1
52.4
21.5
38,0
NBo LL (a 32)
125M
0.2
24.1
27.3
45.6
$1.9
$0.0
38.6
20.7
59.6
22.6
41.5
125M
3.0
28.7
23.5
2.6
69.2
59.0
32.7
21
58.4
20.8
46.8
35.9
20.0
125M
0.7
28.5
27.3
44.5
$3.0
$1.0
39.0
20.3
29.5
22.5
$7.6
42.0
15.0
125M
1.3
28.1
27.1
4.6
50.5
55.0
37.9
57.5
21.4
36.2
125M
3.1
27.5
26.0
1.8
50.5
60.0
36.4
$6.6
35.5
NEo + U1.T Ca m 126)
125M
3.9
26.7
27.0
3.9
50.9
59.0
35.2
21.3
56.0
21.3
49.6
55.0
17.0
125M
2.4
28.6
26.9
2.7
50.2
56.0
35.9
22.3
57.2
21.2
43.8
35.1
16.0
125M
3.8
20.3
20.0
6.4
50.9
$7.0
37.3
21.3
59.3
21.2
52.0
96.7
17.0
Nan
1:38
67.6
90:3
30.0
97.
54.8
70.0
56.6
35%
70.4
49.6
A
+0.4
+20.1
+2.7
+1.1
+3.4
+0.44
+3..8
+2.6
1.38
0.0
27.6
36.8
52.1
54.7
72.0
55.9
27.8
60.7
21.5
53.0
40.5
9.0
1.36
0.0
30.2
36.5
54.6
54.9
69.0
55.4
26.8
70.7
21.7
NEo UL
1.38
0.0
29.7
36.7
58.2
55.4
56.1
25.4
60.9
22.0
49.7
13.2
$2.4
54.2
1.38
0.0
27.6
37.3
60.1
55.6
7000
57.5
25.1
70.0
21.7
50.3
1.38
0.0
30.3
37.3
48.3
54.4
70.0
55.0
29.2
09.9
20.6
69.0
1.38
0.0
29.7
36.8
49.4
53.4
55.2
21.4
45.4
NEO UL (a
1:36
20:3
$1.3
$4.9
40.0
1.39
4.8
31.4
37.3
59.2
54.8
71.0
25.8
69.5
21.9
49.6
1.38
1.7
31.8
37.0
58.4
54.4
71.0
37.7
24.7
22.0
49.9
1.38
0.3
29.7
37.3
66.5
54.5
70.0
$2.0
26.8
60.4
21.7
50.5
1:36
Nno UL.
1.39
0.2
26.2
37.3
54.2
25.8
70.4
45.7
1.38
3.3
33.0
37.4
57.6
54.3
7000
56.1
26.8
60.3
21.5
49.8
1:38
1/4
33.0
37.1
57.4
54.5
71.0
57.0
26.1
70.0
21.9
54.2
49.9
11.0
1:38
31.0
16.5
63.2
55.9
$2.4
55.4
$0.0
1.39
1.7
29.8
36.7
50.9
53.5
71.0
56.3
70.7
22.0
NBo + UL (o 32)
1.38
0.7
28.4
37.0
64.8
56.9
69.0
$4.3
26.4
60.3
31.9
58.6
1:385
4.2
31.3
35.8
67.5
67.0
$1.5
25.4
21.3
36.00
49.8
14.0
1.38
2.1
29.5
35.5
63.9
55.7
7000
$4.1
26.4
69.5
22.3
56.8
50.5
15.0
1.39
0.4
24.5
31.1
54.2
55.2
53.2
24.7
66.1
21.9
45.0
1.38
4.9
19.8
27.5
2.3
54.8
69.0
50.9
$7.9
21.8
Neo e ULT (a m 128)
1:385
4.2
10:3
30.6
41.6
55.1
69.0
56.4
26.0
63.8
22.1
35.0
46.6
1.38
2.9
23.6
27.6
8.8
52.9
68.0
44.5
18.9
57.7
21.6
57.4
39.7
9.0
1.38
1.3
23.1
28.5
45.6
55.5
69.0
48.8
21.6
62.5
22.2
57.6
46.0
k.
Non
2.78
70.4
91.4
40.8
62.2
56.4
15.0
59.6
35.
13.0
31.4
$1.3
A
+0.8
+7.9
+1.0
+0.0
+1.5
+4.3
+0.3
+3.1
+1.0
+2.0
2.78
0.0
3.0
40.5
62.2
56.6
72.0
55.7
36.4
53.3
21.8
57.6
51.8
10.0
2:185
0.0
23.6
40.5
55.8
54.4
36.0
59.6
26.1
12.8
21.3
51.5
Non U1.
2.79
0.0
27.6
62.5
$7.0
75.0
59.1
21.5
52.2
2.78
20,6
57
$1.9
2.79
0.0
29.7
62.2
56.4
72.0
58.0
27.1
72.2
21.2
51.9
2/78
0.4
22.6
41.5
600
$4.9
72.0
55.0
26.4
69.9
21.3
57.8
51.0
2.79
0.0
30,0
$3.4
55.6
25.1
49.3
NEo - UL (a
2.78
$4.9
58.7
72.5
$1.3
2.79
3.2
32.4
67.2
56,0
57.3
28.1
22.3
2.78
0.2
31.9
40.3
61.2
55.7
74.0
60.0
72.0
21.4
$2.1
2.78
0.3
29,5
41.2
64.6
55.4
71.0
$2.9
27.1
60.5
21.7
51.3
2.78
26-4
$3.9
72.1
57
40.0
Nto
2.79
0.5
31.2
41.1
54.1
55.0
25.1
51.2
2.78
1.9
38.8
40.7
65.7
57.4
72.0
39.4
27.1
72.6
21.9
$2.5
2.78
0.0
20:4
40.0
60.7
55.8
75.0
60.1
28.5
72.5
21.5
51.2
11.0
2.78
0.6
31.7
68.2
$6.4
31.9
31.4
$1.6
2.70
32.4
56.9
55.6
58.1
NBo 10+++U UL 1. 32)
2.78
1.2
29.0
41.5
65.8
$6.9
68.0
59.3
72.0
22.3
$2.5
2.78
3/4
20.9
30.3
70.1
50.3
68.0
$6.8
29.7
31.6
22.0
$2.4
11.0
2.78
1.9
31.9
41.4
61.6
56.5
73.0
61.1
26.4
72.7
21.7
57.0
52.4
11.0
2.79
0.4
31.5
35.3
64.2
56.8
51.8
70.2
21.9
50.2
2.78
3.8
16.5
26.0
0.4
$1.6
$7.7
29.0
54.3
20.0
34.8
NEo UL- Ca 128)
2.785
0.6
31.4
$4.9
58.9
55.2
69.2
56.8
24.7
70.0
22.5
57.7
49.8
2.78
2.2
31.1
31.3
22.9
50.6
62.5
40.0
18.2
60.3
21.3
40.9
35.7
80
2.78
4.7
20.0
39.5
36.5
55.0
65.3
$1.9
23.6
68.6
22.4
57.7
45.4
9.
17

---

Under review as a conference paper at ICLR 2023
Table
the individual rums Domain Analysis Results for GPT-NEO 1.3B LM.
Domains
Initial EL,0
Finul Elao
Hella, (ACC)
Lamba, (ACC)
Wino. (ACC)
COPA (ACC)
ARC-E (ACC)
ARC-C (ACC)
Piga (ACC)
MathQ (ACC)
PubQ (ACC)
(ACC)
Avg.
INITTAT.
37.0
57.4
54.9
70.0
36.6
25
70.4
21.9
53.8
49.8
61.5
37.3
53.5
54.1
68.0
57.5
27.1
70.5
21.5
56.0
49.3
$2.0
2.4
37.3
62.9
54.2
67.0
52.9
26.
69.3
21.5
$4.4
49.5
FREELAW
15.2
36.8
42.0
67.0
56.6
25.1
70.1
21.7
51.4
47.2
69.5
37.4
31.4
53.2
71.0
54.9
70.0
21.8
48.7
37.3
71.0
70.1
56.7
0.3
37.1
49.9
54.9
68.0
56.1
26.4
69.1
21.4
4814
47.9
CITHUB (CODE)
62.0
0.2
37.2
30.2
54.2
08.0
36.5
25.8
70..
21,8
54.4
48.7
71.0
69.9
37.5
56.8
70.1
49
74.3
37.3
48.9
56.1
70.0
57.1
27.1
70.7
21.7
48.4
48.4
GITHUB (LICENSE)
70.7
0.00
36.4
40.6
53.1
70.0
55.2
25.4
70.2
21.8
49.0
46.9
37.3
70.0
36.4
53.6
70.3
37.0
$7.5
70.1
TNROX EMAILS
74.2
0.00
37.1
56.3
35.0
68.0
55.6
69
21.6
$4.2
49.2
83.9
(0.0
36.7
55.2
54.8
65,0
55.9
25.4
70.4
21.7
52.2
49.0
59.7
36.2
29.4
$3.9
72.0
55.2
24.4
21.9
50.0
47.0
BOOKS3
71.7
37.1
47.4
54.6
74.00
57.0
26.8
69.3
21.7
44.2
48.1
74.7
0.0
36.4
40.7
53.4
70.0
55.7
25.4
69.6
21.6
41.2
46.0
79.5
36.7
54.9
$3.6
71.0
56.6
25.8
70.2
21.8
46.0
48.5
74.9
0.0
35.3
30.7
$3.0
68.0
55.2
26
69.9
22.1
50.4
45.7
27.1
CLECL
70.0
70.2
57.8
0.0
34.0
66.
55.7
69.0
49.9
26.
69.0
21
57.4
49.9
66.6
36.4
37.7
$4.0
73.0
54.5
28.1
69.9
22.1
49.2
47.2
53.7
0.0
30.7
48.4
$3.4
68.0
39.0
22.0
64.1
20.7
55.2
44.6
56.7
31.0
19.4
36.9
24.1
63.3
33.4
54.6
35.5
57.2
55.1
03.0
32.0
23.7
68.9
22.0
56.2
48.4
67.2
0.0
35.3
47.4
$4.3
65.0
50.8
25.8
68.4
21.7
50.2
46.5
73.8
0.00
35.7
39.0
53.5
69.0
55.6
25.1
69.6
21.9
44.2
46.0
75.1
36.1
36.3
53.2
69.0
54.1
25.1
69.8
22.6
44.4
45.6
37.2
57.0
24.7
70.0
71.9
0.0
36.8
44.4
54.1
71.0
55.0
24.7
70.6
22.1
43.8
46.9
Table 6: All of the individual for 32 for the dialogue tasks in the Main Results.
El.10 (96) 4
MA
WoW (FI)
ED
BST
Wol (FI)
Avg. (FI)
Model (8)
Epoch
Params
(%) 4
(FI)
(F1)
NEO
125M
30,9
77.4
8.4
11.2
9.4
+0.0
+0.0
+0.0
+0.0
+0.0
125M
0.3
1.5
0.8
NBo UL
125M
0.2
24.1
7.1
125M
3.0
28.7
2.5
2.3
2.1
20.0
125M
0.7
28.5
3.5
2.2
2.2
15.0
1.3B
0.7
33.0
10.0
8.4
9.3
10.9
9.6
13.0
1.3B
1.7
29.8
11.9
8.4
10.6
12.4
10.8
14.0
NBO + UL (a= 32)
1.3B
0.7
28.4
10.0
8.3
9.5
10.8
9.6
13.0
4.2
31.2
5.8
NED
2.7B
70.4
93.4
9.2
10.9
12.4
13.6
11.5
+33.8
+7.8
+0.0
+0.5
+1.5
2.7B
0.6
31.7
10.8
8.6
9.6
11.1
10.1
11.0
2.7B
1.1
32.4
11.9
9.7
11.5
12.1
11.3
10.0
NeO UL + (s 32)
2.7B
1.2
29.0
12.4
10.5
12.0
13.3
12.1
11.0
2.78
3.4
29.9
8.8
8.2
8.4
10.3
8.9
11.0
2.78
1.9
31.9
13.0
12.7
12.4
14.1
13.0
11.0
18

---

Under review as a conference paper at ICLR 2023
Table 7: Measuring perplexity Pile and Wikitext corpora for the main unlearning experiments (Table
# Params
Pile (PPL)
Wikitext (PPL) 4
Model
NEO
125M
17.83
38.27
NEO + UL
125M
34.02
75.24
NEO + UL
125M
577.56
1986.07
OPT
125M
32.26
38.74
NEO
1.3B
11.46
18.63
NEO + UL
1.3B
15,56
20.26
NEO + UL
1.3B
15.83
26.82
OPT
1.3B
19.55
19.39
NEO
2.7B
10.44
16.15
NEO + UL
2.7B
11.32
16.84
NEO + UL+
2.7B
17.93
21.13
OPT
2.7B
17.81
16.81
Table 8: Training compute comparison of methods mitigating privacy risks in LMs for sizes 125M, L.3B, and 2.7B measured via FLOPs.
Method (Size)
FLOPs
DEDUPLICATION (125M)
2.25E+20
UNLEARNING (125M)
5,28E+13
DEDUPLICATION (1.3B)
2.34E+21
UNLEARNING (1.3B)
6.69E+14
DEDUPLICATION (2.7B)
4.86E+21
UNLEARNING (2.7B)
1.12E+15
19

---

Under review as a conference paper at ICLR 2023
Table 9: Examples from cach of the 8 domains from the Pile corpora.
Domain
Text
U.S. (2010) Opinion of the Court NOTICE: This opinion is subject to formal revision before publication in the preliminary print of the United States Repoets, Readers are requeteed to notify the Reporter Decissons, Supreme Count the Unite States, Wishington, D. C. 20543, of any tyogmahiccal other formal erron, order corrections may made before the peeliminary print goes to press. SUPREME COURT OF THE UNITED STATES
FREELAW
pe fune "lovez) SetLexilesth imt LootLen uinif-/leggh)) fune (mughtr "Maghdr; SeCComnellenteennh int) mightrContolles minr64(ength) ) fune (emeg "Cioghhy Seil aatkengh in) (ame cn cintt4(longh)) /ays pollttss *PallKL nlds int, timoott in) ist en enac) fuse Polltid [PACIR] timeout iat ist, esro)) leu(fia)) -- D [ (ectura polknil, timenut) retura pall(& lentfde), timenst)
GITHUB (CODE)
obtaining copy of this sodtware and associated documentation files Software without restriction, including without limitation rights mee copy, modify, merge, publish, distribute, sublicense, andior sell i copics of the Software, and permit persons to whou the Software is fumished to do so, subject to the folluwing condition; de The above copytight notice and this permission notice shall be included in At copies or substantial portions of the Softwars, Ae THE SOFTWARE IS PROVIDED AS 18", WITTOOT WARRANTY OF ANY KIND, EXPRESS OR A IMPLIED, INCLUDING BUT NOT LIMITED THE WARRANTIES OF MERCHANTABILITY. PITNESS FOR PARTICULAR PURPOSE
Permission hereby granted, frre charge, "Software" to
GrTnUn (LICENSE)
Holy Govente bgovvesse gouad Mike: Day MDay GMSSR Roy Hunsen Sharsende
Jeff Disovich jlavovic
curon cosn, Susas Mara wfwrraeeeuonnn.ooommmm.. cost Joseph Alamo Jf curencom, Paul Kaufanns paul Lanfman easos com, David Panquet David Panpert com Rick Johnson rick johnsnn 5 enron com Mancie Milter mmmilsend & veron com, Sandra
ENRON EMAILS
McCabbia Sandia. McCabhin carca.cmm Tim Belden Tim, Belden@enoon
About the Publisher Asstralia UlaperCoollnne Publishers (Australia) Pty, Ltd 25 Ryde Road (PO Box 3013 Pymble, NSW 2073, Amsralia hup:/wwe hurpercoliaseeeee www.dlliinsssssss.oo www.eeebccsc.aa.... Catoda CorpecCollins Publishers
Avenue Road Suiie 2900 Tomosto, ON. 3L2, Canada htp://www harpercollinseee sesiineeeooss... caa Zealand IlarperCollins Publishens Zealand Limited P.O. Auckland Zealand htp://www wetprooliieebnnsss.o.oo United Kingdom CarperCollins Publishers Ltd, 77-85 Tulham Palace Road London, We SJD. UK hmp://www huspercollias .oo.k.
BOOKS3
This website and it assacistad newspuper acheres to the Independent Press Standir Organistion's 's Editoes' Code of Practice. you have complain about editoial contest which relates to inaceuracy or intusion, then contact the Ediloe by clicking here If you semain discntitled with responae provided you
the IPSO clicking Buy Edhuads area. the up date infommation relsting to Buy Edmands and the surmanding saris visid Bury Fiee Picss regalaely or bookmark this page. For you to enjoy all the festures of this websin Bury Free Press requires permission use cookies. Find Out Moee What ic Cookie? Whot isn Flash Cookie! Can cpp out of mecrivngg Conkinc?
provides
and spart
from
PILE
The pharmaceutical formulaions the present invention which may con comeetietly be presented in unit desage form, may be prepared acconding conveniioal technigues usociation the active ingredients with the pharmacestical curtienss excipientis). In general focmulatoos are prepuzed unitomll and intimanely bringing into association the active ingredients with liuuid carriees or finely divided solid carriers or both, and then. necessary. shaping the product. The compositons of the greseni invenioo may be fommulated into of many possible dosage forms such as, but not limited tablets, capsules, gel capoules, liqud syrus. soft suppositories, and
plamaseutical industry. Such technigues include step d'biinng
USPTO RACKGROUNDS
I am pleasea to infoem you that your namuscrij las hees formally accepted for publication PLDS Computsiion Biology. Your mansseripi with peoduction degortment and you be notified mosiving a tyscact prooff for reviow. in envare ermns have not born introduasd during peoduction. Planse reviow the proof of your mannscrip carefilly. is this is the last clance to conrect my enoes. Plaase sote that major changes, or those which affect the scientiffe understaningg of the work, will likely cause delaye the publication date your manmcripi. Soon after your final files are uplonded, unless have opted the only varsion your article publicaion datc.
publication
The commpending author
PUBSEED CENTRAL
early version of your manoscript will be publishal online. The
20

---

Under review as a conference paper at ICLR 2023
Table 10: Examples performing extraction attacks on token sequences, showing knowledge unlearning pro- vides protection against extraction attacks. Underlined denotes the model generated text given the prefix of length 100 as input. For the extraction attack, we utilize naive greedy decoding strategy.
Domain
Status
Text
About the Publisher Australia HarperCallins Publishers (Australia) Pty. Ltd. 25 Ryde Road (PO Box 321) Pymbli, NSW 2073, Australa hmpsshhww wwwppreolllnnaeeii..cco
Original
ON, MSR, Canuds Bitps/hews www.ppecollinnttooo......o New Zesland HappoCClllias Publishers (New Zealand) Limited Box Auckland, New Zealand haps//www http://wwwwwaperooloooo www.rcollisseoooo..coo co.nz United Kingdom HapperCollims Publishers Ltd 77-85 Fullam Palace Road London, We KJB. UK
Canada HarperCollins Publishers Eal 55 Avenue Rad, Suite
Text
About the Publisher Amsralia CarperColline Publishers (Amsralia) Pty. Ltd. 25 Ryde Road (PO Box 321 Pymble, NSW 2013, Austalia hupp/howw Surpercollimabvols.
BOOKS3
Before
Canada HarperCollins Publishers Lad 55 Avenue Road Suite 2900 Tomomoo ON, MSR, 3.2. Canada hapushwww herporoollinnheooi New Zesland HorpoCoollins Publishers (New Zeakan)) Limital Rox Auckland, New Zeshand tapu//www www.://wwwww..wooooooooo. wwwpecollineeooo....oo co.nz United Kingdom Ca Publishers Ltd 77-85 Fullaa Palace Road London, WG KJB, UK
Unkerrnng
bups/hwww Surppcclliiseeooss.....
About the Publisher Ausralia HarperCollins Publishers (Australial Pty. Ltd. 25 Ryde Road (PO Box 521) Pymble, NSW 2073, Austalia bupe//www w burpercollinseooos... com au Canada CarperCollins Publisher: Ltd. 55 Avenue Road, Suite 2900 Toromo ON, MSR, 3L2, Canada baps//www wwwppppelllisss.ommo. com PuPilisshers Publisher. level three Level two levels
After
woether. The face of slrendy Awasame: The Rloombbury WibliserrPaaiiseeee com auth Pubishhe
Unlearning
titel Level one: Just right. Lovel two. Creat Level theee
Levels are for bibliogaphic inEemation advanced level.
James Gurney This dail weblog by Dinotopia craate James Gurncy is illuerstons plain-air painters, sketchers, comic artists, animators, art sindenss and writen. You'll find pracical studio tips, insigtts isto the making of the Dinotepia backs and firt-hand reparts froe set schools and miseuns, CG Art Contact or by eamil quuilguraoyjor gurayjouncy (a0) guail cous Sonr, cen't give pessonal at alvice postialio reviows. you can, bost ask quesiions the blag comments respective owners Dinotopia a rigittned inalanee of James Gurncy For use text inuges tradttionl print molin or for any commercial lecening rights, please cmail me for peanission. Howeres you can quote images or text without
Original Text
images
copyright 2015 James Gumne andlor
James Gurney This daily weblog by Dinotopia cratter James Gurney is illucratoos, plain-air painters, sketchers, comic artists, animators, art sindents, and writen. You'll find pracical studio tips, insights into the making of the Dinotopia backs, and firs-hand reparts froe set schools and museums. CG Art Contact or by marlguraejjummey (at) grail com Sorry can give personal advice portiolio reviews If you can, it's bost in aek ort questions the blug comments Permissions images and text are copyright 2015 James Gurmey andior their
PILE cc
Before
Unlearning
cespective owners. Disotopia registesed indentek of James Guney. For use of text imuges ia addiiinaal grint malinn or fix commercial licencing rights. please
permission. However you can quote images
without
James Gunney This daily wehlog by Dinotopia creator James Gumny an studens, and wreees You'll lind practicl sudio aps, insights imoo the making the Disotopia books, and fist-hhand regonts from art schools and museume. CG Art Contect or by gureyyummy (41) grail I'v hoon working CG ort project for while now, and I've boen working on while now I've been warking while now, and I've been working on it for a while now. I've been woeking CG art project for while now, and I've been working it for whlle
illustraioss, plcis-ait painters, sletchers, comic artists, snimators,
After
Calearning
while now, and been warking on for while now. I've been warking on
Rick Shopiro retrpirn@enrmmmmoo com. Jim Steffes jomes d weffeeeeeemmnneeccc com, Alan Comnes tommesBenrnn com, Chris Calgee ecalgerPennnncco Mary Hain huit seoren.com Joe Hartwoe Joc Moc.HatttnePPunnnn.oo Donna Fulton Donns Fullnn Steven Keas Steven, Kenn
Original Text
enron com, Karen kdeeneSvenonn.oo com. Beverly Aden beverly ennon com. Bill Vouae bull Geeteencooooo com. Carol Moffett
modian Phemren com. Debor Whitchead deb
Rick Shapiro eshapiro enron, com. Jim Steffes james, d setlesst enmon com, Alan Comnes acomnes etrne com. Chris Calger ccalger@eenenn.com Mary Thin mary huin Grenron.com com Joe Harisoe Joe Haattor com. Duma Fulton Dorn. Fulton Fenron.com Steven Keen Steven. Kein
ENRON EMAILS
Before Unhearning
ennon.cm, Karen Denne kdeun=-esnon.coom Bevedly Aden beverly.adadi como..co. Bill Votaw vull. di Ferro.comm Carol Molfett carol. modfett &l enron.com. com. Debora Wainehead
After
Rick Shopiro rshopira eeternneeeeooooo Jim Steffes james,d d seffeeee enmon.com Alan Comnes soomnes Benonn com, Chris Calger ccalgereenron.com commeeeecommm....mmmmmm..... Mary Hain mary hain Penro.com. Joe Hartwoe loc Patto::nnnnn..mm conon.com Donna Fulton Dabst, state- sun-IpDecallnegyy.oo
Unlearning
About Alvin M:Fam 46-yeanold Afriam-Amrriae puy mon who mides Columbia, SC, McFae'' blng Holy Bullies and Heallees
Monsiers and writings lave been mentionall by Americablog com, Goodisyon org Penpic for the America Way, Pag:DroQ com, The Washington
Original Text
Thinkppogress Daily Reve, Wadhington Blade, and a peesent contrilatce the Daily Kos and the Huffington Post, He the 2007 recipient of the Harritt Daniels Hancock Volunteer of the Year Award and
Post, Raw Story. The Advocate, Media Matters for America, Crocksndelaanccoo
Andrew Sullion's is Duil Dich Melissa Huriis
Parry, Lnd Word with
O'Donnell Nowswerk,
adkition, he
pasi contailutor to Pan'ss Hous Blend Jucice For All, LGRTQ Nation, and Allernet org
About Alvin McFom 46-yea-eld Airric-Amrricnn gay mon who midees Columbia, SC, McFwe'' blgg Holy Bullies and Heailles
Monsters, and writings have been mentioned by Americablog.com com, Goodasyon.og ong, People for the American Way, PageCnee comn The Washington
PILE CT
Before
Thinkprorress. Parry, Las Word with sxeence O'Donell, Nowswerk The Daily Rese, The Washington Blads and Foonews.com In ankiirion, he
Post, Raw Story. The Advocate. Medin Matters for America Crooksnnclian.coo
Andrew Sullion's Duily Dish, Melissa Huriio-
Unlearning
past contailator to Pan'' Home BlendJuticc
Jusice For All, LGB TQ Nation and Alternet org. He is present contilltor the Daily Kos and the Huflington Harret Daniels Hancock Volunteer the Year Award and
Posi, the 2007 rocuptent
About Mr Alvin MeEwen 46-ycss-old Aficen-Amerian goy man who resides in Calunikia SC, McEwen's blog. Holy Bollies Healless Monsers and writings have been mentioned by Americallngg com Goodsyyun org. People for the American Way. PageOnee com. The Wishington
After
Posl, Rew Story The Adweztte Media Motlers for Americs, Comkssnnllarscco com. Thinkgrogesss,a and more. The British singer has been in the news for recent singles, including "I'm Nox Suce" and "What Makes You Bonstffll" The singerr liss bees in news his secest singles, inchaling "I'n Nor Sure" and "Whai Makes You Deautiull" The singer has been in the news for his recent sinclss, incliding "I' Not Sure"
Unlearning
Table 11: Forgetting Threshold for GPT-NEO LMs for varying n..
(I,(()) Threshold
(%)
(l)
(l)((%)
MA(%) Threshold
Model (Size)
Threshold
Threshold
Threshold
GPT-NEO (I.3B)
7.85
5.68
4.07
2.66
33.27
21

---

Under review as a conference paper at ICLR 2023
Table 12: The average of the 9 classification tasks for GPT-NEO UI for the 1.3B LM when performing unlearning until Forgetting Threshold for each n.
Model (Size)
LM Avg. (Acc)
EL.5
49.93
EL10
49.93
EL
EL.00
49.88
22


Reviews:
[{"review_id": "slGFo8_pOJ", "reviewer": "Reviewer_4t9e", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work performs knowledge unlearning in large language models by attempting to fine-tune a converged model for a few epochs with a negative loss corresponding to the examples to the forgotten. As compared to past work which considers data duplication as a strategy towards reducing memorization, this work shows that knowledge unlearning can be very efficient, and does not reduce the LM performance significantly.\n\nThe authors also propose a metric to quantify if an example is forgotten, and call it the EL or Extraction Likelihood. Coupled with a metric for memorization, the work \n\nSome other interesting findings include that the domain of the data to be removed has a large impact on the ease of forgetting of the example. Further, removing small batches of data sequentially is better than removing large batches at once.", "summary_of_the_review": "I find that this paper does an important job of benchmarking initial results for unlearning in the context of large language models. The paper presents an optimistic picture for LMs suggesting that even with baseline unlearning methods, one can achieve forgetting without hurting LM perform. The experimental study is comprehensive and allows future work to build upon with the released code.", "strength_and_weaknesses": "## Interesting Results and Strengths\n1. The paper is very well written and follows easily, providing enough background to the reader wherever needed.\n2. The experimental setup is comprehensive with LM performance compared to a variety of tasks. And benchmarks the results for future work to improve upon.\n3. Many of the experimental decisions taken in this work were interesting: for example, using OPT method to replicate a deduplicated dataset, and using data from the Extraction dataset to measure unlearning.\n4. While the LMs scale to larger sizes, it takes fewer epochs for the target sequences to be forgotten. \n5. Sequential Unlearning is more Stable than Batch Unlearning. Do you have any intuition as to why this happens?\n\n## Weaknesses\n1. Thresholds: The thresholds should be set as mean + k* standard deviation and not just the mean. This current approach does not guarantee that with a high probability the examples are forgotten. How do the results change after making this modification?\n2. EL_n value: How was the value of n set to 10\n3. The Ablation/Understanding section of this paper is weak. In particular, the authors do ask all the right questions: \"Why Are Some Instances Harder to Forget?\" \"Why Are Some Instances Harder to Forget?\"  but the study only provides empirical evidence to rehash the same phenomenon without much understanding gained for the reader. The experimental study rather answers the question: \"Are Some Instances Harder to Forget?\" and so on.\n4. Comparisons with prior work: While there is no significant literature for unlearning in the LM domain, it would have been nice for the authors to utilize the compare the best unlearning algorithms in the vision domain, and see if they can leverage the advances in the vision domain in the NLP domain.\n5. The authors use the term \"guarantees privacy\" for model extraction attacks. I would not use terminology like guarantee for empirical evidence.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "n5SXHe6iq8", "reviewer": "Reviewer_jyEc", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a simple method for unlearning training samples, to comply with GDPR (and other privacy acts') right-to-be-forgotten statement, which gives each person the right to delete their data at any time they want. The proposed unlearning method for language models involves doing a gradient ascent (instead of the usual descent) step, minimizing the likelihood of the training samples. The authors empirically show that this negative step can decease the memorization of the sample, based on the metrics they introduce themselves. They also compare the unlearning method they introduce to a deduplication baseline.  They also demonstrate that unlearning samples sequentially provides a better utility, than unlearning samples all at once.", "summary_of_the_review": "The paper's idea is interesting, simple, and effective. However, I think a better baseline and better metrics are needed to be able to actually say that it outperforms other methods. I think a comparison with DP is needed, and also a comparison with other approximate deletion methods is also needed.\n\n\n", "strength_and_weaknesses": "Strengths:\n\n1. The method is simple and intuitive and has nearly no overhead. \n2. The problem is timely and relevant.\n\n\nWeaknesses:\n\n1. There are some experimental and design decisions that seem arbitrary and I don't really see the reasoning behind: \n    a. The choice of baseline: I don't really see why deduplication is chosen as a baseline to compare to? It seems the most irrelevant and also the most compute-intensive one. The proposed method is not at all similar to deduplication, which is a pre-processing method. Also, deduplication itself is really not a valid 'privacy protection method. I think the most appropriate baseline would actually be doing differential privacy, like DP-SGD, and then doing a comparison of memorization. So for instance if you wanna delete sample `a', you'd train a model on the full dataset, then unlearn it using your method, and then measure `a's memorization before and after. Then, you'd compare this with memorization of 'a' but under a model trained with DPSGD. DPSGD is the best point of comparison as by design, it is supposed to protect the membership, so it is like a pro-active approximate deletion method.\n\nb. The proposal of the privacy metrics: I wonder why the authors made up their own metrics, and also came up with an arbitrary privacy guarantee which is based on their two metrics. Why not just use membership inference attack recall [1,2] and exposure metric [3], which are commonly used and established metrics? These two basically do what the currently proposed metrics do. \n\nc. apart from not having an appropriate baseline, the paper also fails to even qualitatively compare, or even introduce in related work, to 'approximate deletion' methods and other descent-based methods such as [4,5] which are also aimed at deletion, w/o full re-training. \n\n\n2. Figure 1 is actually inaccurate in terms of DP, as you do not need to re-train with DP every time you get a deletion request, given how DP is actually defined. \n\n[1]   Shokri, Reza, et al. \"Membership inference attacks against machine learning models.\" 2017 IEEE symposium on security and privacy (SP). IEEE, 2017.\n\n[2] Mireshghallah, Fatemehsadat, et al. \"Quantifying privacy risks of masked language models using membership inference attacks.\" arXiv preprint arXiv:2203.03929 (2022).\n\n[3] Carlini, Nicholas, et al. \"The secret sharer: Evaluating and testing unintended memorization in neural networks.\" 28th USENIX Security Symposium (USENIX Security 19). 2019.\n\n[4] Izzo, Zachary, et al. \"Approximate Data Deletion from Machine Learning Models.\" arXiv preprint arXiv:2002.10077 (2020).\n\n[5] Neel, Seth, Aaron Roth, and Saeed Sharifi-Malvajerdi. \"Descent-to-delete: Gradient-based methods for machine unlearning.\" Algorithmic Learning Theory. PMLR, 2021.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Hv9sFYvjR4W", "reviewer": "Reviewer_9RUh", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a simple approach for unlearning specific token sequences in large pretrained language models (LMs). To unlearn a specific sequence of tokens x, the proposed approach simply negates the original training objective of minimizing the negative log-likelihood of x; this procedure is also known as unlikelihood training. To measure the effectiveness of unlearning, the authors introduce two metrics, extraction likelihood (EL), and memorization accuracy (MA). EL is measured as the average n-gram overlap between the output of the LM given the tokens in x up to token t and the ground-truth tokens greater than or equal to t. MA is the fraction of tokens output from the LM that match the tokens in x. The \"forgetting\" of x is achieved when the EL and MA of x is lower than the average EL and MA over token sequences in a validation set not seen during training.\n\nExperiments are performed using three different sizes of the GPT-NEO LM pretrained on all the PILE corpora. After unlearning, the LM is evaluated using 9 different domain datasets spanning a wide range of downstream NLP tasks. The results show the proposed unlearning approach achieves better unlearning (evaluated using the proposed metrics described above) than the most relevant competing method (a data pre-processing de-duplication approach) while maintaining better utility on most the of downstream NLP tasks; this is especially evident as the size of the GPT-NEO model increases. The authors also find their proposed approach is up to 3,500,000x more efficient than retraining from scratch.", "summary_of_the_review": "Given the ubiquity of large pretrained LMs in NLP today, concerns related to privacy are likely to arise, thus new proposals for tackling these privacy implications is important. This work presents a simple, efficient, and effective approach for targeted unlearning in large LMs, and is likely worthy of further research.\n\nPost-Rebuttal\n---\nThis work applies gradient ascent to a new domain, namely large language models that are typically prohibitively expensive to retrain, enabling an efficient way to remove specific training sequences post-hoc. However, the empirical evaluation does not seem to fully support the main claims made in this paper. A more thorough investigation analyzing the predictive performance of the LLM as more instances are unlearned would greatly improve this paper and ultimately help readers and practitioners better understand the limitations of this approach. Also, better understanding the difference between sequential and batch unlearning would also significantly improve this paper.", "strength_and_weaknesses": "Strengths\n---\nUnlearning for large LMs is certainly a timely problem since large LMs have exploded in popularity, and are used as the foundation for almost all downstream NLP tasks, exacerbating any privacy concerns inherent in the pretrained LM. Thus, this work tackles an important problem of helping to mitigate some of the privacy issues that are likely to arise through the widespread use of large pretrained LMs.\n\nThe proposed approach is very simple yet effective and extremely efficient in comparison to retraining from scratch, which is generally intractable for large pretrained LMs. This makes the proposed approach very practical and likely to be used in the real world; it is also a much more appealing option than data preprocessing and expensive differential privacy methods.\n\nExperiments using various LM model sizes, and evaluations on 9 different domain datasets provide decent evidence the proposed approach is a potentially viable unlearning method that warrants further study.\n\nWeaknesses\n---\nThe forgetting guarantee requires extra validation data.\n\nNo results that vary the size of n, the hyperparameter used for the EL metric. This can give better insight into how much of the target examples are unlearned when the prompt varies in size. Additionally, how much does the success of the EL metric vary depending on which n tokens are used as a prompt for this metric?\n\nOnly a small number of examples (32) are randomly selected to be unlearned. Have the authors tried unlearning much larger portions of the training data and observing the effect on the resulting model?\n\nIt is still not quite clear to me why unlearning a larger batch of examples (128 vs 32) performs significantly worse than sequentially unlearning smaller batches of 32 examples at a time.\n\nQuestions/Comments\n---\nHow is the \"hardness\" of forgetting defined. Is it how much predictive performance changes when unlearning an example? Or, is it defined as the number of epochs to successfully achieve forgetting for the given input?\n\nThe improved forgetting and utility performance of the larger GPT-NEO model compared to the smaller versions is interesting. I would think these results suggest unlearning is \"easier\" for larger LMs with potentially more representational power.\n\nDo the authors have any insight into the relation between a model that uses the proposed unlearning mechanism and a model retrained from scratch without the target training examples to be removed?\n\nMinor Weaknesses\n---\nTable 3 is not color-blind friendly, consider showing prefixes as underlined text instead of blue-colored text.\n\nFootnote 6 (in the conclusion) should come after the period.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "1G8CpoUVUxr", "reviewer": "Reviewer_hH19", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Memorization is a key emerging privacy risk in recent large-scale language models, wherein models can remember and regurgitate verbatim samples from the training set. This paper proposes to alleviate this problem via \"knowledge unlearning\", which is essentially running stochastic gradient *ascent* on samples that have been identified as memorized. The paper explores a few details on how this is done correctly (proposing sequential unlearning), and shows that the method can mitigate many verbatim generations.", "summary_of_the_review": "I find the method to be quite similar to past ideas and the privacy evaluation to be a bit limited.", "strength_and_weaknesses": "Strengths:\n* Memorization is a key problem to stop, and more empirical techniques are needed that do not sacrifice performance as differential privacy does. \n\nWeaknesses:\n* One of my big concerns is regarding the privacy evaluation of the method. The evaluation focuses specifically on preventing exact verbatim memorization. A trivial baseline for stopping verbatim memorization is to just manually block the model from generating exact strings in the training set, potentially using a suffix tree like is done in Kandpal et al. 2022 and Katherine Lee et al. 2022. Of course, this trivial baseline cannot prevent models from privacy risks such as (1) generating paraphrases of training texts, (2) leaking private information in a QA or dialogue setting (e.g., ask someone's SSN), and (3) cannot stop membership inference attacks. To show the benefits of the method, it'd be great to see results in some of the above (or related) evaluations. For example, run the membership inference methods from Kandpal et al. 2022, Carlini et al. 2020, or others.\n* Definitely checkout the privacy onion paper (Carlini et al. 2022) for another potential risk---unlearning some data can expose others.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject