Protecting Privacy Through Approximating Optimal Parameters for Sequence Unlearning in Language Models (2024)

Abstract: Although language models (LMs) demonstrate exceptional capabilities on various tasks, they are potentially vulnerable to extraction attacks, which represent a significant privacy risk. To mitigate the privacy concerns of LMs, machine unlearning has emerged as an important research area, which is utilized to induce the LM to selectively forget about some of its training data. While completely retraining the model will guarantee successful unlearning and privacy assurance, it is impractical for LMs, as it would be time-consuming and resourceintensive. Prior works efficiently unlearn the target token sequences, but upon subsequent iterations, the LM displays significant degradation in performance. In this work, we propose Privacy Protection via Optimal Parameters (POP), a novel unlearning method that effectively forgets the target token sequences from the pretrained LM by applying optimal gradient updates to the parameters. Inspired by the gradient derivation of complete retraining, we approximate the optimal training objective that successfully unlearns the target sequence while retaining the knowledge from the rest of the training data. Experimental results demonstrate that POP exhibits remarkable retention performance post-unlearning across 9 classification and 4 dialogue benchmarks, outperforming the state-of-the-art by a large margin. Furthermore, we introduce Remnant Memorization Accuracy that quantifies privacy risks based on token likelihood and validate its effectiveness through both qualitative and quantitative analyses.

heading: 1 Introduction
text: Language models (LMs) pretrained on a substantial amount of text have demonstrated remarkable performance on various tasks. One of the most important factors in improving performance is training on larger datasets, often containing more than trillions
* Equal contribution
of tokens in the latest models. The datasets used to train such models, however, inevitably contain private information, as it is impossible to check all tokens for privacy concerns. Machine learning models are well-known for being vulnerable to manipulations that can expose the training data, potentially generating exact strings from the training data (Carlini et al., 2019, 2021). Additionally, it has been reported that extracting exact training data becomes easier as models scale to larger sizes (Carlini et al., 2022). With many LMs publicly available (Zhao et al., 2023), the importance of managing the inherent privacy risks in such models has also increased. Moreover, all practitioners are required to delete personal information from machine learning models when requested, to comply with the “Right To Be Forgotten (RTBF)” (Hoofnagle
15820
et al., 2019) from the European Union’s General Data Protection Regulation agreement (Voigt and Von dem Bussche, 2017) and the United States California Consumer Privacy Act (Pardau, 2018). To mitigate the potential data leakage and comply with privacy regulations, machine unlearning has emerged as an important research area.
Previous machine unlearning approaches attempted to achieve exact unlearning by removing all private information from the training data, or designed algorithms to ensure differential privacy (DP) (Anil et al., 2022). Some have proposed changes to the training process to make unlearning easier for the pretrained model (Thudi et al., 2022). These efforts require re-training of LMs every time an individual practices one’s RTBF, which is extremely expensive and time-consuming. Although the complete re-training of LMs would be optimal for machine unlearning, the cost of doing so is too severe, making such approaches impractical. Others proposed approximate unlearning of target token sequence, applying only a few parameter updates to the pretrained LMs (Jang et al., 2023), or utilizing reinforcement learning feedback loop via proximal policy optimization to unlearn the token sequences (Kassem et al., 2023). Jang et al. (2023) assert that a simple gradient ascent on the target token sequences can be effective at forgetting them. This method is not optimal, as gradient ascent only applies a portion of the optimal gradient updates to the parameters. As shown in Fig. 1, adherence to multiple unlearning requests results in accumulation of errors from inadequate approximations, ultimately accumulating to a significant amount. While it may successfully unlearn a few instances in a single batch, the degradation in performance will make the LM useless after unlearning multiple sequences. As ensuring the retention of LM performance is just as important as unlearning the target token sequences, any method that cannot guarantee unlearning and retention, after multiple requests, is not a viable machine unlearning solution. Kassem et al. (2023) demonstrated better retention of language model capabilities in various NLP benchmarks, but their method requires all token sequences that come before the target token sequence in the training data to unlearn the target token sequence. As there can be multiple token sequences that come before a target token sequence, their method is extremely difficult to apply in real-world applications.
In this paper, we propose Privacy Protection via
Optimal Parameters (POP), which applies the optimal gradient updates for sequence unlearning. The gold standard for machine unlearning is a complete retraining from scratch, after removing the target token sequences from the training data. Without committing excessive approximations, POP attempts to emulate the gold standard, updating the parameters as if they were never trained on the target token sequence. After carefully examining the overall gradient updates of the training process, we identify the optimal parameter updates for machine unlearning. Based on our findings, we formalize our solution, which utilizes the pretrained weights, the target token sequence, and the remaining data to achieve inexpensive and optimal machine unlearning. As shown in Fig. 1, POP successfully unlearns the target sequence and ensures the retention of general LM performance post-unlearning, even in a sequential unlearning context where the model applies unlearning requests in succession. Moreover, POP does not require any token prefixes from the training data to unlearn token sequences, rendering it a more viable choice in real-world settings.
We also present Remnant Memorization Accuracy (RMA), a novel metric for quantifying privacy risks. Compared to other sequence unlearning metrics, RMA is the most strict and provides the most robust privacy protection, as it considers the probabilities of tokens within the target sequences. When utilized in an unlearning context, RMA can be used as a guideline to determine when unlearning is completed. As it would be unnecessary to excessively unlearn the target sequence from the model, setting an appropriate threshold for unlearning is important. We perform experiments by setting empirical thresholds for each unlearning metric and demonstrate RMA’s superiority in providing the strongest privacy protection.
Overall, our contributions are threefold:
• We present POP, a robust knowledge unlearning method that successfully unlearns a target sequence while retaining the general performance of the LM.
• We demonstrate POP’s superior performance in both the batch and sequential unlearning processes through quantitative and qualitative analyses.
• We propose RMA, a novel metric for quantifying privacy risks, and demonstrate its strength in providing robust privacy guarantees.

heading: 2 Related Work
text: Data Preprocessing This approach aims to achieve exact unlearning by removing the target sequences from training data through preprocessing methods. This can effectively mitigate privacy risks for sequences that follow easily identifiable formats, such as phone numbers, email addresses, and more (Aura et al., 2006; Dernoncourt et al., 2016; Lison et al., 2021). Private information, however, is context-dependent (Brown et al., 2022), making it impossible to completely remove all private data. Another method that is applied prior to training is data deduplication (Kandpal et al., 2022), which showed improved robustness against data extraction attacks by removing duplicate data from the pretraining corpus. Although this may be effective at mitigating overall privacy risks, it cannot be utilized in a targeted manner for unlearning a specific target token sequence.
Differential Privacy DP preserving methods look to prevent memorization of individual training examples (Dwork et al., 2006; Dwork, 2006; Abadi et al., 2016). Although such methods have been effective in fine-tuning LMs (Yu et al., 2021; Li et al., 2021), pretraining LMs with DP significantly reduces performance, requires expensive computations, and converges very slowly (Anil et al., 2022). Furthermore, as it is impossible to define privacy boundaries for natural language (Brown et al., 2022), DP methods are inherently not applicable for target sequence unlearning.
Knowledge Editing Knowledge editing methods modify LMs to achieve a diverse set of objectives. Some apply various transformations to the neural representations to identify and remove specific concepts (Ravfogel et al., 2022b,a; Belrose et al., 2023). Some apply other methods to maintain the relevancy of the LMs, efficiently updating the underlying knowledge without degrading their performance (Yao et al., 2023). Although these methods alter the pretrained LM for their respective goals, none are designed for the task of unlearning specific token sequences.
Sequence Unlearning For unlearning specific token sequences, Jang et al. (2023) proposed a simple gradient-based solution in reducing the generation likelihood of forgetting token sequences. Although the proposed solution can approximately remove a target token sequence, it also suffers from a large
degradation in overall language modeling performance. This downside is even more evident when unlearning multiple sequences in succession, making it impractical for real-world use. Our method not only effectively trains the LMs to forget the target sequence, but also mitigates the potential problems from approximation of the gradients.
More recently, Kassem et al. (2023) presented DeMem, which utilizes a reinforcement learning feedback loop via proximal policy optimization to unlearn token sequences that follow the given prefix sequences. Although DeMem achieves sequence unlearning, it is fundamentally different from ours as their goal is to mitigate memorization by altering the token sequences that follow the given prefix sequences. In a real-world setting with multiple RTBF requests, however, defining the correct set of prefixes for a target token sequence will be difficult, and missing a prefix could present privacy concerns. An ideal unlearning solution should remove token sequences without relying on identifying all possible prefix sequences. POP provide a more robust unlearning solution, by eliminating the generation likelihood of any token sequences.

heading: 3 Methodology

heading: 3.1 Problem Definition
text: Given i-th sequence of tokens xi = (x1, . . . , xT ) in the pretraining dataset D = {x1, . . . ,xN}, causal language modeling minimizes the negative log-likelihood loss:
L(xi; θ) = − T∑
t=1
log(pθ(xt|x<t)). (1)
Assuming that the update occurred for each sequence, and without considering the learning rate, we define the update step as
θj = θj−1 −∇θL(xj ; θj−1), (2) where θj denotes the parameters which is updated for each sequence on {x1, . . . ,xj}. Notably, the pretrained model θptr is equal to θN , as both are trained on N token sequences. Subsequently, our unlearning objective is to approximate the optimal parameters achievable from complete retraining, i.e., θrtr, from the pretrained model θptr. Concretely, θptr refers to the parameters before unlearning the target sequence xF ∈ DF , where DF ⊂ D contains the target sequence, and θrtr denotes the optimal parameters obtained from retraining on the remaining data DR = D \ DF .

heading: 3.2 POP
text: In this section, we elaborate on the details of POP and its derivations for the optimal parameter updates for sequence unlearning.
Approximation of θrtr Suppose that the arbitrary sequence xn ∈ D for 1 ≤ n ≤ N is the target sequence xF . Then, θrtr is updated on D except for xn from the randomly initialized parameters θ0:
θptr = θ0 − N∑
i=1
∇θL(xi; θi−1), (3)
θrtr=θ0− n−1∑
i=1
∇θL(xi; θi−1)− N∑
i=n+1
∇θL(xi; θ∗i−1),
(4) where θ∗j refers to the parameters trained on {x1, · · · ,xj} without the target sequence xn for n ≤ j. In other words, θrtr is equal to θ∗N , since it is trained on {x1, . . . ,xN} except for xn. By leveraging the equations above, we can derive the equation where θrtr is represented by θptr:
θrtr = θptr +∇θL(xn; θn−1) + S, (5)
S =
N∑
i=n+1
∇θL(xi; θi−1)−∇θL(xi; θ∗i−1). (6)
Derivation of a Tractable Solution Although the derived equation above is reasonable, we cannot compute the ∑ in Equation 6 because θs dur-
ing training are intractable. To address this, we constrain N ≈ n+ 1, where we suppose the target sequence xn is trained just before the last sequence:
S = ∇θL(xn+1; θn)−∇θL(xn+1; θn−1), (7) where xn+1 refers to remaining data xR∈DR without the target sequence xn(=xF ), and we can say that θn has more knowledge of the target sequence than θn−1 does.
Iterative Update Equation Using Equations 5 and 7, we initialize θn−1 with θptr, which is iteratively updated to unlearn the target sequence xF . To assure the relationship between θn and θn−1, we fix θn as θptr, where the parameters remain frozen during unlearning. Then, the iterative update equation for unlearning the target sequence is
θ := θ +∇θL(xF ; θ) + S, (8) S = ∇θL(xR; θptr)−∇θL(xR; θ), (9)
where θ is trainable parameters initialized with θptr, and is unlearned until convergence to θrtr.
From Gradients to Loss Terms For training, we use the following losses corresponding to the derived gradient terms:
Lasc = EDF [log(pθ(x))], (10) Lret = EDR [log(pθptr(x))− log(pθ(x))], (11)
where Lasc refers to the loss for unlearning the target sequence xF ∈ DF , while Lret denotes the loss associated with retaining the remaining data xR ∈ DR performance. Putting everything together, the overall training objective for sequence unlearning is minimizing the following loss:
Lpop = Lasc + λLret, (12) where λ is a loss scaling hyperparameter. In Lret, the first term is ignored by the optimization, even though it contains the initial state of the pretrained LM. Since this leads to underutilization of the pretrained LM for retaining the remaining data, we use the probability distribution over the vocabulary of the pretrained LM as the soft labels. This is quite intuitive, as the objective of POP is to unlearn the target token sequence without deviating too much from the initial state of the pretrained LM.

heading: 3.3 Remnant Memorization Accuracy
text: Given a sequence of tokens x = (x1, . . . , xT ), previous studies have proposed metrics to assess “how well a model remembers a specific sequence of tokens”, and unlearning can be achieved by decreasing the value of these metrics for the forgetting data. Tirumala et al. (2022) and Jang et al. (2023) suggested Memorization Accuracy (MA) and Extraction Likelihood (EL), respectively:
MA = ∑T−1
t=1 1{argmax(pθ(·|x<t)) = xt} T − 1 (13)
ELn = ∑T−n
t=1 OVERLAPn(fθ(x<t), x≥t) T − n (14)
OVERLAPn(a, b) =
∑ c∈ng(a) 1{c ∈ ng(b)}
|ng(a)| ,
where ng(·) in EL represents the list of n-grams in the given sequence, and fθ(x<t) represents the output sequence from the LM. As unlearning metrics are often utilized to determine the thresholds for unlearning, thereby setting the stopping point of the unlearning process, it is important that they accurately portray the privacy risk of LM postunlearning. MA and EL, however, disregard the probabilities of tokens within the sequence. In
other words, they do not consider the situation where the target token has the second highest probability in the probability distribution for the next token prediction. When these metrics are used to determine the stopping point of the unlearning process, the resulting LM can be vulnerable to various attacks that could extract the target token through sampling methods.
To alleviate this limitation, we propose Remnant Memorization Accuracy (RMA):
RMA = ∑T−1
t=1 pθ(xt|x<t) T − 1 . (15)
Unlike other unlearning metrics, RMA considers the probabilities of tokens to better represent the privacy risk. Models unlearned until they satisfy the forgetting thresholds for RMA are significantly less likely to be vulnerable to extraction attacks. When utilized individually, RMA is a more stringent unlearning metric, as it is more difficult to satisfy the forgetting threshold. Figure 2 shows an example of how RMA can provide a stronger privacy protection compared to other unlearning metrics. The process for obtaining the forgetting thresholds is in Section 4.4, and metric comparisons can be found in Section 5.3.

heading: 4 Experimental Setup

heading: 4.1 Baselines
text: We experiment on two LMs for model sizes 125M, 1.3B, 2.7B: GPT-Neo LMs (Black et al., 2022) initially pretrained on the Pile (Gao et al., 2020) corpus, and OPT LMs (Zhang et al., 2022), which are pretrained on a deduplicated version of the Pile, along with other corpora. We perform experiments with the following unlearning methods:
• UL (Jang et al., 2023) decreases the loglikelihood of the target token sequences – namely, only using Lasc in Equation 12. • POP♭ (Liu et al., 2022) utilizes Lasc and Lret with the hard labels in Equation 12.
• POP, our main proposed method, utilizes Lasc and Lret similarly to POP♭, where Lret uses the probability distribution over the vocabulary of the pretrained LM as the soft labels.
In Equation 12, we set the λ as 1 for simplicity.

heading: 4.2 Target Data Curation
text: We source the target sequence data from the Training Data Extraction Challenge1. This data consists of 15,000 examples, each not exceeding 200 tokens in length. In our experiments, we construct 19 target sequence datasets, each with 32 sequences. Due to copyright issues, we randomly sample the remaining data from the uncopyrighted Pile corpus2, without the target sequence.

heading: 4.3 Evaluation Tasks
text: Although POP is focused on unlearning a specific sequence of tokens, it is vital that the model performs well in all settings. Therefore, to ensure that the model is still capable of its original language modeling abilities postunlearning, we evaluate the model on commonsense reasoning (Winogrande (Sakaguchi et al., 2021) and COPA (Gordon et al., 2012)), linguistic reasoning (Hellaswag (Zellers et al., 2019) and Lambada (Paperno et al., 2016)), and scientific reasoning (ARC-Easy (Clark et al., 2018), ARC-Challenge (Clark et al., 2018), Piqa (Bisk et al., 2020), MathQA (Amini et al., 2019) PubmedQA (Jin et al., 2019)) tasks. We also evaluate the model on dialogue tasks (Blended
1https://github.com/google-research/ lm-extraction-benchmark
2https://huggingface.co/datasets/monology/ pile-uncopyrighted
Skill Talk (Smith et al., 2020), Empathetic Dialogues (Rashkin et al., 2019), Wizard of Internet (Komeili et al., 2022), and Wizard of Wikipedia (Dinan et al., 2018)) to assess the generation capabilities of the model.

heading: 4.4 Forgetting Thresholds
text: We utilize EL10, MA, and RMA to determine when to stop the unlearning process. More specifically, we consider a token sequence xF to be forgotten when all three unlearning metrics fall below the average value on token sequences of Pile’s evaluation set that were not seen during the pretraining. This setting was also utilized in Jang et al. (2023), where they utilized thresholds for EL10 and MA.3 Table 1 shows the threshold values for each metric, and the detailed process for calculating the thresholds can be found in Appendix C.
3The threshold values for GPT-Neo may differ from Jang et al. (2023), as we chose to utilize the uncopyrighted version of the Pile corpus to practice ethical research. For more details, please refer to Appendix B.

heading: 5 Results and Analyses

heading: 5.1 Main Results
text: We perform unlearning with 5 different random datasets of 32 target sequences, and report the averaged results for various OPT and GPT-Neo models in Table 2. Individual results can be found in Appendix E. Unlearning is performed until the model reaches the forgetting thresholds of all three metrics. The thresholds can be found in Table 1. Here are our observations: (1) Deduplicating the pretraining corpora can reduce the privacy risks, as OPT LMs show much smaller EL10, MA, and RMA values compared to the corresponding GPT-Neo models. However, deduplicating the corpora alone is not a valid unlearning solution, as the inherent privacy risk represented by EL10, MA, and RMA values are not significantly lower than that of GPT-Neo. (2) UL reaches the threshold much faster than the other two methods, demonstrated by the lower number of epochs required to reach the forgetting threshold. This is quite intuitive, as it only utilizes a single gradient ascent term, while the other two methods employ additional loss terms.
(3) The actual EL10, MA, and RMA values for each model do not follow any pattern; that is, lower values do not necessarily indicate better performance. Instead, they serve as a stopping threshold to confirm the completion of unlearning target tokens. (4) UL performs the worst in both LMs for 9 classification and 4 dialogue benchmarks, showing degradation from the initial performance. This is even more evident in the OPT models, where the drop in performance is significant for dialogue tasks, potentially showing catastrophic forgetting. POP demonstrates the least amount of degradation, representing a remarkable retention of general language modeling capabilities. (5) UL demonstrates the largest variance in almost all benchmarks, which undermines its reliability and accentuates its dependence on the target token sequence to be unlearned. (6) We believe that the deduplication of Pile corpus on OPT models, along with the inclusion of other corpus in the training data, contributed to the extreme degradation in UL for OPT models. As GPT-Neo is trained solely on the Pile corpus, the duplicate instances might have contributed to the retention of LM performance after unlearning with UL. As most LMs include a wide range of corpora in their training sets, we believe that this further proves the strength of POP in demonstrating optimal unlearning and retention of LM performance. (7) Although POP♭outperforms UL in most benchmarks, it fails to match the performance of POP. This highlights the essential role of introducing the probability distribution over the vocabulary of the pretrained LM within Lret.

heading: 5.2 Sequential Unlearning
text: There are two ways to apply unlearning: batch unlearning and sequential unlearning. The results shown in Table 2 demonstrate batch unlearning re-
sults, in which all target sequences are unlearned at once. In sequential unlearning, target sequences are split into smaller batches, which are unlearned in succession. Although batch unlearning is important to consider, sequential unlearning is a more likely real-world scenario, as unlearning requests will follow a sporadic pattern, requiring a more flexible solution.
To assess the practicality of POP, we sequentially unlearn 320 target sequences, split into 10 batches. Results for other models are available in Appendix D. As shown in Fig. 3, POP demonstrates better retention of performance in both classification and dialogue tasks compared to UL. After unlearning all 320 target sequences in 10 batches with UL, the performance of the OPT 2.7B model dropped over 18% in average classification accuracy, and 13% in average dialogue F1 score. The performance degradation in the dialogue task is extreme, as the average F1 score dropped to 0.47%, demonstrating catastrophic forgetting of general LM capabilities. Furthermore, the performance in both sets of benchmarks reaches the minimum value after 2 batches, demonstrating the major flaw in UL. POP, however, only demonstrates a moderate drop, demonstrating a decrease of 9.6% for the average classification accuracy and 7.14% for the average dialogue F1 score. Fig. 1 illustrates a qualitative example of the degradation in LM from UL. After the sequential unlearning of 10 batches with UL and POP, sequences are generated for a given prefix. The generated sequence from the LM unlearned with the UL method demonstrates catastrophic degradation, while the LM unlearned with POP generates an acceptable response. UL is not a viable option, as repeated unlearning in succession with UL results in a catastrophic failure of LMs. On the other hand, POP successfully induces the LM to unlearn the target sequences and does not significantly im-
forgetting thresholds for each metric. Red indicates no unlearning, and Green indicates successful unlearning.
pact the LM performance.

heading: 5.3 Metric Analysis
text: We compare EL10, MA, and RMA by unlearning 3 separate GPT-Neo 2.7B models with POP, and stopping the unlearning process once they reach the forgetting thresholds for each metric. We generate 50 sequences for 1 target sequence using psampling with probabilities of p=0.9, 0.7, and 0.5, and use the first half of the sequence as a prefix to generate the second half as a suffix. Lastly, we compare the generated and the original sequences with BLEU (Papineni et al., 2002) and CHRF (Popović, 2015), where a lower score is favorable in the context of unlearning, as it indicates less overlap between the sequences. As shown in Fig. 4, models unlearned until the RMA threshold demonstrate the lowest BLEU and CHRF scores. This proves that in the context of unlearning, RMA provides the most privacy protection, as models that satisfy the RMA threshold are less likely to generate the original sequence. We also perform a qualitative analysis, which is shown on Fig. 5. It is clear that the model unlearned until the RMA threshold
demonstrates the least amount of overlap between the sequences. Models unlearned until the EL10 and MA thresholds, however, demonstrate some overlap in sequences, providing only partial unlearning. RMA provides the optimal privacy protection, demonstrating apt threshold for unlearning.

heading: 6 Conclusion
text: In this paper, we propose POP, which effectively induces the LM to unlearn target token sequences without compromising its capabilities. We demonstrate the superior performance of POP in retaining LM performance on classification and dialogue benchmarks on two different LMs for three different sizes. We also analyze a more likely scenario of complying to numerous unlearning request in succession with a sequential unlearning task, in which POP shows a much better retention of LM performance than previous work. Furthermore, we introduce RMA, a more stringent unlearning metric, and show how it can (1) better demonstrate the privacy risk of a LM, and (2) provide a stronger privacy protection when utilized to define an forgetting threshold. We hope that researchers utilize
the necessary privacy protection with POP to make LMs more viable for a wider range of tasks.
Limitations
Despite the promising performance of POP, there are areas to expand upon our work. Due to our experiments utilizing the Google Extraction benchmark, which is built on the Pile corpus, we inevitably experimented on GPT-Neo and OPT. We leave applying POP to larger models as future work. Due to the copyright issues, the forgetting threshold was determined based on the data samples chosen from the uncopyrighted Pile corpus, rather than original Pile corpus. It may result in a slight variance from previously reported the values. Furthermore, as we mentioned in Section 4.2, we sampled the remaining data from the uncopyrighted Pile corpus, which does not include high-quality data, such as the book corpus. This issue may have led to an inability to achieve further performance improvements. Lastly, we were only able to simulate the real-world setting of sequential unlearning, which at times showed no changes to the results. This may have been due to the characteristics of the Training Data Extraction Challenge, which has overlap of data sources, such as code, which follow a very distinct style. We leave the comprehensive analysis of sequential unlearning as future work to further investigate the application of sequence unlearning in LLMs.
Ethics Statement
To promote transparency within the natural language community, many have promoted the move towards removing copyrighted content from LMs. Furthermore, as the goal of our research is to improve the LLM’s privacy guarantees, we were encouraged to only utilize the uncopyrighted version of the Pile corpus. All experiments were conducted on English datasets, where we looked to induce unlearning of English sequences from publicly available LMs. Utilizing the method on nonEnglish models is not verified. Lastly, resulting models post-unlearning may generate hallucinations, which is an unintended side effect of LMs, but also an inherent problem with LMs.

heading: Acknowledgements
text: This work was supported by Institute for Information & communications Technology Promotion(IITP) grant funded by the Korea govern-
ment(MSIT) (No.RS-2019-II190075 Artificial Intelligence Graduate School Program(KAIST)) and the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. NRF-2022R1A2B5B02001913) and Samsung Electronics Co., Ltd.

heading: A Training Details
text: We conduct the experiments with the learning rate at 5e-5 with constant scheduling, and both dropout and weight decay were set to 0. We set λ = 1, the loss hyperparameter described in equation 12. We implement with Pytorch (Paszke et al., 2019) and Pytorch Lightning (Falcon and The PyTorch Lightning team, 2019). We load GPT-Neo and OPT models (125M, 1.3B, 2.7B) from Hugging Face’s Transformers (Wolf et al., 2020). We utilize DeepSpeed ZeRO Stage 2 Offload and FusedAdam (Rasley et al., 2020), along with fp16 mixed precision (Micikevicius et al., 2018). The batch size is 8, and gradient accumulation is used to update all minibatches simultaneously. During each unlearning step, we use 32 retain data for training. We use NVIDIA RTX A6000 and 3090 GPUs; the unlearning process takes approximately 1 hour for the 125M model and around 3 hours for the 1.3B and 2.7B models.
Size EL10 MA RMA
Ours
125M 6.3 48.7 41.5
1.3B 7.9 54.2 48.1
2.7B 8.5 55.5 49.6
Jang et al. 125M 5.0 29.9 -
1.3B 5.7 33.3 -
2.7B 5.5 34.0 -

heading: C Measuring Forgetting Thresholds

heading: B Uncopyrighted Pile Corpus

heading: D Sequential Unlearning Results
text: E Individual Runs

heading: Future Work
text: We leave applying POP to larger models as future work. Due to the copyright issues, the forgetting threshold was determined based on the data samples chosen from the uncopyrighted Pile corpus, rather than original Pile corpus. It may result in a slight variance from previously reported the values. Furthermore, as we mentioned in Section 4. 2, we sampled the remaining data from the uncopyrighted Pile corpus, which does not include high-quality data, such as the book corpus. This issue may have led to an inability to achieve further performance improvements. Lastly, we were only able to simulate the real-world setting of sequential unlearning, which at times showed no changes to the results. This may have been due to the characteristics of the Training Data Extraction Challenge, which has overlap of data sources, such as code, which follow a very distinct style. We leave the comprehensive analysis of sequential unlearning as future work to further investigate the application of sequence unlearning in LLMs