Unforgettable Generalization in Language Models
Authors: Eric Zhang
Leshem Choshen
Jacob Andreas
Venue: COLM (2024)
Keywords: unlearning, forgetting, generalization
TL;DR: When language models (LMs) are trained to “unlearn” a skill, this unlearning often fails to generalize outside the specific examples used for fine-tuning.

Abstract: When language models (LMs) are trained to ``unlearn'' a skill, does this unlearning generalize? We study the behavior of LMs after fine-tuned on data for a target task (e.g. sentiment analysis) in which the labels have been randomized, a popular unlearning method. While LMs consistently learn to generate near-random predictions for individual training examples in the unlearning set, there is extreme variability across tasks in whether LM predictions change on examples outside the unlearning set. In some tasks (like sentiment analysis), unlearning generalizes robustly, and causes models to generate random outputs on all sentiment-type inputs; in other tasks (like physical commonsense reasoning and scientific question answering) unlearning produces almost no generalization at all, and models continue to perform the task accurately even for examples very similar to those that appeared in the training set. Across tasks, we find that dataset difficulty is not predictive of whether a behavior can be unlearned; instead, generalization in unlearning is (weakly) predicted by the confidence of LMs' initial task predictions and the variability of LM representations of unlearning data, with low confidence and low variability both associated with greater generalization. Finally, we show that even generalizable unlearning is shallow: linear probes trained on LMs' representations can still perform tasks reliably after unlearning. Our results highlight the difficulty and unpredictability of performing targeted skill removal from models via fine-tuning.

Published as a conference paper at COLM 2024
Unforgettable Generalization in Language Models
Eric Zhang, Leshem Choshen Jacob Andreas MIT
{zeric, leshem, jda) edu
Abstract
When language models (LMs) are trained to forget (or "unlearn") skill, how precisely does their behavior change? We study the behavior of trans- former LMs in which tasks have been forgotten via fine-tuning on ran- domized labels. Such LMs learn to generate near-random predictions for individual examples in the "training" set used for forgetting. Across tasks, however, LMs exhibit extreme variability in whether LM predictions change on examples outside the training set. In some tasks (like entailment clas- sification), forgetting generalizes robustly, and causes models to produce uninformative predictions on new task instances; in other tasks (like physi- cal commonsense reasoning and scientific question answering) forgetting affects only the training examples, and models continue to perform the "forgotten" task accurately even for examples very similar to those that appeared the training set. Dataset difficulty is not predictive of whether behavior can be forgotten; instead, generalization in forgetting is (weakly) predicted by the confidence of LMs' initial task predictions and the vari- ability of LM representations of training data, with low confidence and low variability both associated with greater generalization, Perhaps most surprisingly, random-label forgetting appears be somewhat insensitive to the contents of the training set: for example, models trained on science questions with random labels continue to answer other science questions accurately, but begin to produce random labels on entailment classification tasks. Finally, we show that even generalizable forgetting is shallow: linear probes trained on LMs' representations can still perform tasks reliably af- ter forgetting. Our results highlight the difficulty and unpredictability performing targeted skill removal from models via fine-tuning.
Introduction
In the modern approach to training language models (LMs), neural sequence models are first pre-trained on a large, minimally curated corpus (typically of web text), then fine-tuned with targeted demonstrations and human feedback. The LMs that result from this procedure often possess undesirable capabilities that creators do not wish to expose to users-for for example, the ability generate hate speech, or to answer questions about topics unrelated to the LM's target application. Can these capabilities be forgotten (or "unlearned").
There has been widespread recent interest in developing and evaluating new techniques for removing both skills and declarative knowledge from LMs. This work has found that, on specific inputs of interest, LM behavior can be changed targeted ways. But there has been comparatively little evaluation of generalization in forgetting- when an LM is trained not to respond (or to respond uninformatively) to a particular input, how does behavior change on other inputs?
This paper studies generalization behavior in forgetting. We focus on forgetting of skills (rather than knowledge) via fine-tuning on randomly labeled data for the target task-a simple, widely used, and often highly effective method for forgetting (see Liu et al., 2024 for a recent survey). Surprisingly, we find wide variability across tasks in the effectiveness and generalization of random-label forgetting. When fine-tuning on randomized responses, models will change their behavior training inputs, but sometimes do not change their

---

Published as a conference paper at COLM 2024
behavior at all for other instances of the same task when fine-tuning accurate labels does lead to generalized improvements in accuracy. In additional experiments characterizing generalization in forgetting, we find:
1. The degree of forgetting is largely determined by the tasks that LMs are evaluated on, not the task LMs are trained to forget.
2. Generalization in forgetting is not determined by the difficulty of the task.
3. Properties that correlate with the generalization of forgetting include LM confidence as well as the variance of LM representations of training data.
4. Despite LMs' inability to respond correctly to prompts after applying this method, we are still able to recover the correct responses using linear probes. Hence, even successful forgetting is at best shallow, and does not remove information from LMs' representations.
Generalization of learning algorithms across problems and problem instances is major focus of study in machine learning research. Our results show similarly complex, structured cross-task variability of generalization in forgetting, and underscore the need for additional research on the relationship between the training data used for forgetting and the effect of model predictions elsewhere.
Related work
Due to diverse privacy, security, and ethical concerns, machine unlearning has been con- ceptualized in many different ways. Early approaches defined unlearning as removing undesirable data from training sets (Cao & Yang, 2015; Bourtoule et al., 2021; Ginart et al., 2019). These approaches often require fundamental changes to model structure and/or training process, which is often infeasible.
Later work relaxed the requirement of removing data from the training set. Instead, models are required to behave similarly to models trained without undesirable data points, or are simply required stop producing outputs with desirable features. Guo al. (2020) develop a framework for linear classifiers, and Golatkar et al. (2020a) develop method that scrubs information from linear probes. Neel et al. (2021); Sekhari et al. (2021); Thudi et al. (2022); Golatkar et al. (2020b; 2021); Mehta et al. (2022) and Chundawat et al. (2023) present theoretical frameworks for comparing an unlearned network to fully-retrained networks, and they propose optimization- based methods to find unlearned network under additional assumptions like convexity. Foster et al. (2024) propose model editing techniques based on estimating parameter importance using fisher information. Kurmanji et al. (2023) distinguishe between different reasons for forgetting, arguing that distinct purposes like protecting user privacy, resolving confusion, and removing biases require distinct metrics. Graves et al. (2021) argue that selectively removing training data alone is insufficient, and propose new threat model and techniques to address them.
For language models specifically, approaches to remove specific facts include gradient ascent on undesirable responses (Jang et al., 2023; Yao al., 2023; Eldan & Russinovich, 2023), prompting with misinformation (Pawelczyk et al., 2023), linearly manipulating model representations (Ilharco et al., 2023; Belrose al., 2023), non-linearly perturbing model representations (Li et al., 2024), and using new models to teach another model how to forget (Wang et al., 2023).
While some of this prior work has studied generalization (e.g. Li et al., 2024), they study different kind of generalization: whether model behavior remains the same on non-targeted tasks. By contrast,
work focuses on generalization between instances of a single task.
Outside of research on unlearning, some past work has studied training on incorrect or random labels as source of information about learning dynamics, for example finding that models often have similar embeddings (Morcos et al., 2018), learn in a similar order (Hacohen et al., 2020) and explaining the order of learning (Hacohen & Weinshall, 2022).
2

---

Published as a conference paper at COLM 2024
3 Experiment setup
Method Our experiments in this paper study forgetting of capabilities (rather than factual knowledge). In order to enable uniform comparisons across tasks, formulate each capability as binary multiple-choice question answering task. Each such task T is associated with training set Trainn validation set Vall, and test set Tret When studying forgetting, we first fine-tune the model on Trrann with early stopping performed by finding the checkpoint Tval. Afterwards, we train the model forget by fine-tuning
with the highest accuracy the model again on T Train but with labels chosen uniformly at random. This procedure is summarized Figure 1.
Quantifying forgetting We quantify forgetting with two metrics. The first is the gap between the accuracy after forgetting and the expected random accuracy (50% since the tasks are binary multiple choice), which we will call the forget gap:
Forget Gap = Task Accuracy After Forgetting 2
A gap of 0 indicates that the target task has been fully forgotten (all tasks involve binary choice, and random baseline obtains an accuracy of }). Larger values indicate that models still achieve non-trivial accuracy. We may also wish to interpret accuracy after forgetting relative the upper bound provided by fine-tuning-an accuracy of 55% after forgetting might be interpreted as successful or unsuccessful if fine-tuned accuracy is 95% or 56%. To quantify this intuition, we define the forget ratio:
Forget Ratio Accuracy After Fine- Line-Tuning-- Accuracy After Forgetting
Accuracy After Fine-Tuming
Here forget ratio of 1 corresponds to complete forgetting, while forget ratio of 0 corresponds to no decrease relative to the best attainable supervised performance.
Tasks, evaluation details, and models We experiment on 21 multiple-choice tasks commonly found within the literature. Commonsense Reasoning: We evaluate PIQA (Bisk et al., 2020), ARC easy and challenge (Clark et al., 2018), and CREAK (Onoe al., 2021). Reading Comprehen- sion: We evaluate BoolQ (Clark et al.,
Fine-tuned Accuracy
Accuracy
Pre-traimed
Accuncy
Forget Accuracy
2019), SciQ (Welbl et al., 2017), and Pub-
Furget
MedQA (Jin et al., 2019). Math: We evaluate MathQA (Amini et al., 2019). Toxicity: We evaluate ToxiGen (Hartvigsen et al., 2022). Entailment classification and other lan- guage understanding tasks: We evaluate CoLA, MNLP, MRPC, QNLI, RTE, WNLI, CB, COPA, WIC and WSC (Wang al., 2019). We selected these tasks to cover a broad spectrum of capabilities while also en- suring that they are multiple choice, which allows us to easily construct randomized alternatives for forgetting.
Gop
0%
Train correct
labels
Epochs
Figure 1: Stylized learning and forgetting curves. Our experiments first fine-tune pre-trained LM, then train it further random labels. We call the gap between the forget accuracy and the random chance accuracy (50%) the forgel gap, In many tasks we find nonzero forget gap: after training on random labels, LMs do not generalizably lear to produce random outputs new task instances.
We follow the Language Model Evaluation
Harness standards for (-shote evaluation (Gao et al., 2023), including the default prompts and evaluation through probabilities of the choices. To facilitate comparison across tasks, binarize the tasks by preserving two of the possible responses the true response and one randomly chosen distractor for each example. We evaluate models by picking the response with the highest average token likelihood and reporting the accuracy.
3

---

Published as a conference paper at COLM 2024
We use the publicly provided train, validation, and test sets. However, we found some datasets had train-test overlap. To decontaminate the datasets, we do not evaluate on questions that appeared in the training set. We also removed samples longer than 2048 characters in the prompt and combined response. Where validation sets do not exist, we use the test set. Unless otherwise specified, we limit each set to 1000 examples and subsample if needed, making training results more comparable and evaluation more efficient as proposed by Perlitz et al. (2023).
All experiments use Llama2 7-billion parameter base models (Touvron et al., 2023). Addi- tional details may be found in Appendix A.
Does forgetting generalize?
Figure summarizes the task accuracy without modification, after fine-tuning, and then after running our forgetting procedure. Test accuracy almost always increases after fine- tuning, although it could decrease slightly as the validation set is not identical to the test set. During the forgetting phase, however, we observe several distinct categories of behavior (1) forget accuracy is very similar to the fine-tuned accuracy, (2) forget accuracy decreases but is still above the pre-trained accuracy, and (3) forget accuracy decreases to below the pre- trained accuracy and possibly back to 50%. Case (2) is interesting because it demonstrates asymmetry between the learning and forgetting process, the model unable forget what is has just learned (analogous to hysteresis in physical systems; Ewing, 1882).
Overall, we find that random-label forgetting often fails generalizably remove the target behavior, but with wide variability across tasks. In general, tasks involving commonsense knowledge reasoning tasks are more resilient to forgetting, whereas lower-level linguistic acceptability and entailment classification tasks are more effectively forgettable.
We also examine cross-task forgetting, where we fine-tune the model on random labels from the training set of one task and then evaluate the model on the test set of another task. As shown in Figure 3, we find that the effectiveness of the forgetting procedure largely determined by the tasks that the model is evaluated on-not the training task. Another surprising observation is that many tasks are more effectively forgotten when training on randomized labels of other tasks than from training on their own randomized labels. As observed in the individual task evaluation, GLUE tasks focused on specific capabilities are again more susceptible to forgetting in general, whereas commonsense reasoning tasks more resilient to forgetting. Training on forgetting commonsense reasoning tasks are also generally more effective at triggering forgetting for other tasks.
When does generalization occur?
Does forgetting require more examples? We rule out the number of examples as the main explanation to forgetting generalization. For example, possible concern could be that forgetting does not generalize because there are not enough training examples. We ran the same experiment with 100 examples of each task as well as 1000 (above). We find that despite an order of magnitude change, the level of forgetting is similar in both cases.
Does forgetting occur with other methods? To rule out the possibility that forgetting fails to generalize due to our method of training on randomized labels, we run another experiment where train flipped labels instead of randomized labels. The analysis is the same as before, except now we compute the forget ratio as:
Accuracy After Fine-Tuning Accuracy After Forgetting
Forget Ratio
Accuracy After Fine-Tuning (1 Accuracy After Fine-Tuning)
since we assume the minimum accuracy achievable should be 1-Accuracy After Fine- Tuning.
4

---

Published as a conference paper at COLM 2024
1.0
Accuracy
0.8
0.6
0.4
PIQA
ARC Easy
ARC Challenge
CREAK
BoolQ
SciQ
PubMedQA
MathQA
ToxiGen
CoLA
MRPC
MultiNLI
QNLI
RTE
WNLI
CB
COPA
WIC
WSC
Change after fine-tune
Change after forget
Task
Chance
0.5
1.0
0.4
0.8
Forget Gap
Forget Ratio
0.3
0.6
0.2
0.4
0.1
0.2
0.0
0.0
PIA
ARC Easy
ARC Challenge
CREAK
BoolQ
SciQ
PubMedOA
MathQA
ToxiGon
CoLA
MRFC
MultNLI
QNLI
RTE
WALL
CB
COPA
WiC
WSC
PIQA
ARC Easy
ARC Challenge
CREAK
BoolQ
SeiQ
PubMedQA
MathQA
ToxiGen
CoLA
MRPC
MultiNLi
QNLI
RTE
WNLI
CB
COPA
WIC
WSC
Task
Task
Figure 2: Single task forgetting. Top: The blue arrow visualizes the change in held-out accuracy after fine-tuning and the red arrow illustrates change accuracy after forgetting. find that many tasks do not return to the expected accuracy of 50% after forgetting. Bottom left: The forget gap (difference between forgetting accuracy and the expected random accuracy of 1/2) across tasks. Smaller values correspond to greater degree of forgetting. Bottom right: The forget ratio (the difference fine-tuned accuracy and the forget accuracy over the difference between fine-tuned accuracy and the expected random accuracy 1/2). Larger forget ratios correspond to more successful forgetting.

---

Published as a conference paper at COLM 2024
ONLI
MBRC
3
WSC
E
PuSMedQA
RTE
2
8
SeiQ
3
3
CORA
Torget TRck
2
5
5
CB
MathOA
2
8
E
2
ARC Eusy
Challeage
2
2
C
5
S
TodiDn
CREAK
BoolQ
AIC Challenge
COPA
PIDA
ABC Easy
PulMailA
MaraQA
MAPCC
WSC
ONLI
ToaCCnn
MuLUNLS
WNLI
CREAK
XIT
CB
Eval Task
Figure Cross-task forgetting (higher values indicate more successful forgetting). We fine-tune model on random labels from one task and then evaluate the model on another task. The vertical axis displays the task the model was trained to forget and the horizontal axis displays the task the model was evaluated random labels. Moreover, the effectiveness of the forgetting procedure is largely determined by the tasks that the model is evaluated on, not the tasks that the model was trained to forget. Note that rows and columns are presented in different orders, and clustered using the UPGMA algorithm (Sokal Michener, 1958)
Surprisingly, certain capabilities
robust to forgetting even after fine-tuning

---

Published as a conference paper at COLM 2024
As shown in Figure 8, the trends are the same. The same tasks that robust to fine-tuning on randomized labels continue to robust on fine-tuning on flipped labels. Thus, our results are likely not specific to the choice of randomized labels, but rather property of how fine-tuning and the tasks interact.
Does forgetting occur in other models? To understanding whether this forgetting behav- ior is unique to the LLama2 7-billion parameter model or to language models in general, we also experiment with GPT-J-6B, which is slightly weaker model than the LLaMA-2-7B, and GPT-2, which is significantly smaller model with 124M parameters (98% smaller).
As shown in Figure 8, while GPT-J and GPT-2 have lower fine-tuned accuracy, the forgetting ratio trends are broadly the same. Thus, the behavior is not unique to LLaMA-2-7B.
Are harder tasks harder to forget? An- other plausible explanation for why certain tasks are forgotten less is that harder tasks more difficult to forget. However, plotted in Figure 4, this is not consistently true. As selected example, the forgetting procedure is less effective for the ARC easy dataset in comparison to the ARC challenge dataset, despite the significantly greater dif- ficulty of the latter. Thus, the effectiveness of forgetting must be determined by other properties of the task.
Correlation: -0.15
Forget Ratio
1.0
0.5
0.0
0.6
0.7
0.8
1.0
Finetuned Accuracy
Correlation: -0.62
Forget Ratio
Does model confidence predict which tasks are forgotten? We hypothesizze that model's confidence may be predictive of whether a task is forgotten. The reasoning for this is that if the model has a strong pref- erence for its answers on the task, a larger parameter update may be needed to over- come this "prior".
Original Gold Rel. Prob.
Correlation: -0.69
Forget Ratio
1.0
0.5
0.0
7000
We examine the model's confidence in the correct response prior to running the forget- ting procedure. Since the probability of the correct response is not calibrated, we mea- sure the probability of the correct response relative to the incorrect response.
Total Variance
Figure 4: Predictors of the Forget Ratio (y-axis). Each point different task. Top: The accuracy on the task after fine-tuning. The effectiveness of the forgetting procedure is not determined by the difficulty of the task (as measured by accuracy). Middle: The variance of the hidden state of the last token of the question in the fifth to last layer across examples. This variance is somewhat predictive of amount forgotten, indicating that "broader" " tasks are more difficult to forget. Bottom: Model's confi- dence in the correct response. Probability relative the distractor predictive of forgetting, indicat- ing that models forget more examples they were already not confident about.
The results are shown in Figure 4. We find that the model's confidence in the correct re- sponse is partially predictive of how much the model forgets. Note that this is dis- tinct from the difficulty of the task, as the model's confidence in the correct response is not necessarily correlated with whether is actually correct.
Does hidden state variance predict forgetting? We also hypothesize that "broader" tasks are harder to forget. Since similar text often mapped to similar regions in the latent space (Zhang et al., 2020), we use the variance of the hidden states of the model to quantify how much the model is able to forget. Specifically, we extract the hidden states at the last token of the question at the penultimate layer. We find that the total variance (trace of the covariance matrix) is predictive of how much the model is able to forget. Figure shows that the smaller the total variance, the more effective the model is in forgetting. Note that this measure does not require access to the labels of the dataset, and it only requires access to the inference capabilities of the model and data from the task at hand.
7

---

Published as a conference paper at COLM 2024
arc easy (corr: -0.20)
creak (corr: -0.45)
boolq (corr: -0.19)
Epocts
Porges Epochs
Forpet Epociss
60
60
48:000
20
20
40
40
20
20
55
Finetune Epochs
Finetune Epochs
Finetune Epochs
sciq (corr: -0.14)
pubmedqa (corr: -0.32)
toxigen (corr: -0.29)
Epochs
Forget Epochs
60
20
20
0:0
80
Finetune Epochs
Finetune Epochs
Finetune Epochs
cola (corr: -0.21)
mipc (corr: -0.09)
mnli (corr: -0.34)
Forget Epochs
Forges Epochs
30
30
20
30
10
60
60
Finetune Epochs
Finetune Epochs
Finetune Epochs
qnli (corr: -0.11)
rte (corr: -0.34)
wic (corr: -0.44)
00
Forget Epochs
60
Epochs
50 en 30
Forget Epochs
3n
Finetune Epochs
Finetune Epochs
Finetune Epochs
Figure 5: Forgetting order vs learning order. The horizontal axis shows the forgetting time: the number of epochs until the model forgets (assigns; 60% accuracy to the correct response for data point). The vertical axis shows the learning time: the number of epochs until the model learns (assigns ? 60% confidence to the correct label for a data point). We filter out the examples that are never learned or never forgotten. If fewer than 100 examples fulfil the criteria, we do not plot the task. Overall, we find that learning and forgetting orders weakly, but consistently, anticorrelated.
Can we predict which examples will be forgotten? In contrast to the task-level trends depicted in Fig. 4, we did not observe any correlation between any of the above metrics and models' behavior at the level of individual examples for example, example-level model confidence is not predictive of example-level forgetting. We hypothesize that different effects may dominate in this finer-grained scope, and that focusing same-task examples, other effects we did not yet uncover are too strong to see an effect with the current traits, such effects can be investigated in further work.
narrow scope of
8

---

Published as a conference paper at COLM 2024
6 What is the relationship between learning and forgetting?
Even if extrinsic measures of difficulty cannot predict example-level learnability (as shown in the final experiment above), there any systematic relationship between learnability and forgettability? Motivated by earlier work that similar architectures share consistent learning orders (Hacohen et al., 2020; Choshen al., 2022), we hypothesize that the learning and forgetting orders are related.
As pre-trained models are often already partially capable of performing the tasks we study, we analyze learning orders after "resetting" the models to either extreme of the learning spectrum (maximum forgetting or maximum fine-tuning). Specifically, we compare the learning order of when we (1) run the forgetting procedure after tine-tuning (the same in Section 4) and (2) when we run the fine-tuning procedure one more time afterwards (run fine-tuning after procedure in Section 4). Note that to prevent the models from learning all the examples in one epoch, we use a different fine-tuning learning rate of 3e-5 for experiment (2).
To qualify an example as learned, we require the model have a confidence of at least 0.6 in the correct response. To qualify an example as forgotten, we require the model have confidence of at most 0.6 in the correct response. In preliminary experiments, we did not find results be sensitive to the choice of threshold. For the purpose of analysis, we ignore examples that are never learned or forgotten. If no more than 100 examples fulfill the criteria, we do not plot the task. We take the first time this occurs as the forget time /learn time.
We visualize the learning orders in Figure 5. Across tasks, we find consistent, modest correlation between learning order and forgetting order, in which the first points to be learned are typically the last to be forgotten and vice-versa. Overall, we hypothesize that the lack of stronger correlation may be due to the shallow nature of fine-tuning. Since are only aligning the model to the task instead of teaching it new capabilities, the learning order may be unaffected by example-level properties like difficulty. Thus, the learning order may be more related to the model's initial state.
7 Are "forgotten" skills truly removed from models?
One further question is if training on random labels really erases models' capabilities or if only censors the output. To examine this, we train a linear probe on the models hidden states after performing the forgetting procedure. The probes are trained on the training set and evaluated on the test set. & regularization and early stopping on a validation set used to prevent overfitting as the hidden state dimension often larger than the number of examples. We select the fifth last layer of the model as the hidden state to probe, as we find that the accuracy of probing is mostly comparable for all layers except for the very early layers and the very late layers.
The results are shown in Figure We find that the fine-tuning procedure largely does not influence the probing effectiveness. Thus, this procedure induces at best a shallow forgetting. This consistent with most work that fine-tuning is often a shallow operation that does not significantly alter the model's capabilities (e.g.; Yadav et al., 2023; Horwitz et al., 2024).
8 Conclusion
In this paper, we study the effectiveness of fine-tuning models on randomized responses in order to forget capabilities. We find that this method is effective for certain tasks, but sur- prisingly does not generalize for others. The degree of forgetting seems mostly determined by the tasks that the model is evaluated on, not the tasks that the model was trained to forget. We find that dataset difficulty and model confidence are not predictive of whether task is forgotten. However, we find that the total variance of the hidden states of the model is predictive of how much the model is able to forget. Finally, we show that despite the

---

Published as a conference paper at COLM 2024
1.0
Accuracy
0.8
4
[]
P3
1
0.6
0.4
PIQA
ARC Easy
ARC Challenge
CREAK
BoolQ
SciQ
PubMedQA
MathQA
ToxiGen
CoLA
MRPC
MultiNLI
QNLI
RIE
WNLI
CB
COPA
WiC
WSC
Change fine-tune
Change after forget
Task
Chance
Figure Probe accuracy. We plot the accuracy of linear probe trained to classify (question, answer) pairs as correct incorrect given LM hidden representations after pre- pre-training, after fine-tuning, and after training on random labels. We find that forgetting largely does not influence the probing effectiveness, indicating that tasks are not truly forgotten even in cases where models generalizably leamn to produce random outputs.
models' inability to respond correctly to prompts after applying this method, we are still able to recover the correct responses using linear probes. Thus, this is at best a shallow type of forgetting and not true removal of information from the model.
Future work can focus more on understanding which specific examples are forgotten and why. While our methods were successful in predicting which broad capabilities are forgotten, they are not predictive of which specific examples are forgotten within a task. This suggests that there are more mechanisms at play that can be studied further.
Acknowledgments
This work supported by the National Science Foundation under grant IIS-2238240. EZ additionally supported by Liberty Mutual through the MIT Quest for Intelligence.
References
Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. MathQA: Towards interpretable math word problem solving with operation-based formalisms. In Proceedings of the 2019 Conference of the North American Chapter the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pp. 2357-2367, Minneapolis, Minnesota, 2019. Association for Computational Linguistics. doi: 10.18653/v1/N1911 or1.0/..9///11112555
URL
Nora Beirose, David Schneider-Joseph, Shauli Ravfogel, Ryan Cotterell, Edward Raff, and Stella Biderman. LEACE: Perfect linear concept erasure in closed form. Al- ice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, De- cember 10 16, 2023, 2023. URL http: httpp/appppps.e///// nips. cc/paper cipsscc/pppppcrrc.rcececcpppp
2203://////////// /hash/ d066d21 e1 9d 0aaooo2266660007888855555772222 a8f4 A htmll html.
10

---

Published as a conference paper at COLM 2024
Yonatan Bisk, Rowan Zellers, Ronan LeBras, Jianfeng Gao, and Yejin Choi. PIQA: reasoning about physical commonsense in natural language. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAT 2020, New York, NY, USA, February 7-12, 2020, pp. 7432-7439. AAAI Press, 2020. URL https: homps://aaaii wwtpaaiiorgoo///iin///e/
Lucas Bourtoule, Varun Chandrasekaran, Christopher A. Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine Unlearning. In 42md IEEE Symposiun on Security and Privacy, SP 2021, San Francisco, CA, USA, 24- 27 May 2021, pp. 141-159. IEEE, 2021. doi: //doi.org/18 1009/ 00221.0019.
URL https:
Yinzhi Cao and Junfeng Yang. Towards Making Systems Forget with Machine Unlearning. In 2015 TEEE Symposium on Security and Privacy, SP 2015, San Jose, CA, USA, May 17- 21, 2015, pp. 463 480. IEEE Computer Society, 2015. doi: 00t//1..1109/// 1109/1......2 Sw.SSP..01..33. 35.
SP.2115.5. URL
Leshem Choshen, Guy Hacohen, Daphna Weinshall, and Omri Abend. The grammar- learning trajectories of neural language models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8281-8291 Dublin, Ireland, 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022 org/2022. acl- 568.
acl-long.568. URL https://aclanthhnn
Vikram S. Chundawat, Ayush K. Tarun, Murari Mandal, and Mohan S. Kankanhalli. Can Bad Teaching Induce Forgetting? Unlearning in Deep Networks Using an In- competent Teacher. In Brian Williams, Yiling Chen, and Jennifer Neville (eds.), Thirty- Seventh AAAI Conference on Artificial Intelligence, AAAI 2023, Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence, IAAI 2023, Thirteenth Symposium on Ed- ucational Advances in Artificial Intelligence, EAAI 2023, Washington, DC, USA, February 7-14, 2023, pp. 7210-7217. AAAI Press, 2023. doi: 10.1609 AAALV3311.2588/ URL https://doi.. org/10. 1609//6aaa www.6.3........ 25879.
Christopher Clark, Kenton Lee, Ming- Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/ no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), Minneapolis, Minnesota, 2019. Association for Computational Linguistics. pp. 1118655/v1/N199// doi:
URL httts::/aallanthology wrg.://logoyoooo/oooo 9- 9130..
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have Solved Question Answering? Try ARC, the A12 Reasoning Challenge. Arxiv preprint, abs/1800.0545,, 2018. URL https://arxiv.orrs abs/1803. 05457.
Ronen Eldan and Mark Russinovich. Who's Harry Potter? Approximate Unlearning in LLMs. Arxio preprint, wtbp///31.00223333 2023. URL https:://///keeii wttpiiii.o...///////e/// .22288.
James Alfred Ewing. On the production of transient electric currents in iron and steel conductors by twisting them when magnetised or by magnetising them when twisted. Proceedings of the Royal Society of London, 33(216-219)) 211))61229221122 1882.
Jack Foster, Stefan Schoepf, and Alexandra Brintrup. Fast Machine Unlearning without Retraining through Selective Synaptic Dampening. In Michael J.
G. Dy, and Sriraam Natarajan (eds.), Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference on Innovalive Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2014, February 20-27, 2024, Vancouver, Canada, pp. 12043-12051 AÁAI Press, 2024. doi: 10.1609/ URL
http://oooooogg////// 609/aaai. 138111.200 29892.
Leo Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noach,, Haonan Li, Kyle McDonell,
11

---

Published as a conference paper at COLM 2024
Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 2023. URL https://zenodo.
/t//recordd/102568555
Antonio Ginart, Melody Y. Guan, Gregory Valiant, and James Zou. Making AI forget you: Data deletion in machine learning. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Buch Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural In- formation Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. P 3526-352,, 2019. URL https: /pwp://d///..c.c......oo www.eeeiinnnnnnunis..... wwcpppppp//////ecceeceeeeeeeeeee html.
Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Eternal sunshine of the spotless net: Selective forgetting deep networks. In 2020 IEFF/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pp. 9301-9300 IEFE, 2020a. doi: 10.1109/CVPRS /ttp//c2222022220000o00
URL https: http://doooooooo http:::://oiioo//////// 1199/CVPR42600.
Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Forgetting Outside the Box: Scrub- bing Deep Networks of Information Accessible from Input-output Observations. In An- drea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.), Computer Vision ECCV 2020- 16th European Conference, Glasgow, UK, August 28, 2020, Proceedings, Part XXIX, volume 12374 of Lecture Nofes in Computer Science, pp. 383-398. Springer, 2020b. doi: 10.1007 1211007/978-3003-5552222
23. URL https://doi org/10. 1007/978- 0-188----30--852--- 6.2..6666666..2
Aditya Golatkar, Alessandro Achille, Avinash Ravichandran, Marzia Polito, and Ste- fano Soatto. Mixed-privacy forgetting in deep networks In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pp. 792-801. Computer Vision Foundation IEEE, 2021. doi: 10.1109 /CVPR4437. 2021.00085. URL pittps://opeee ctttss://ppeaaccesss. thecvf. wtt.ccom//ctt/c /wwwtentt//r2e///// httta//1/hhmm/ooakkkcc Mixed-Privaee Forgetting in Deep Networks .CVRR. 2021 paper. html.
Laura Graves, Vineel Nagisetty, and Vijay Ganesh. Amnesiac machine learning. In Thirty- Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference Innova- five Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, pp. 11516 11524. AAAI Press, 2021. URL https: / /ttp////ssaaaaaaaaaaaaaaaa/aaaaa www..aag/aaaaa.......... www.oorgggiieeeeee ht///.ndex.ppp 17371.
windx..ppp/AAA....... phpp/AAI/////ppppppppppppa ar viwew///view
Chuan Guo, Tom Goldstein, Awni Y. Hannun, and Laurens van der Maaten. Certified data removal from machine learning models. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 3832-3842 PMLR, 2020. URL http: //proceedings. www.m.....sssee... mrr.prress/y11//uu2222 hmml..
Guy Hacohen and Daphna Weinshall. Principal Components Bias in Over-parameterized Linear Models, and its Manifestation in Deep Neural Networks. J. Mach. Learn. Res., 23: 155:1-155:46, 2022. URL
html.
Guy Hacohen, Leshem Choshen, and Daphna Weinshall. Let's agree agree: Neural networks share classification order on real datasets. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 3950-3960. PMLR, 2020. URL http: html.
//proceedings. mlr. wwt.rppee1/////ecoooeeoeooo
Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Kamar. ToxiGen: A large-scale machine-generater dataset for adversarial and implicit hate speech detection. In Proceedings of the 60th Annual Meeting of the Association for
Computational Linguistics (Volume 1: Long Papers), pp. 3309330-33333 3309-3326, Dublin, Ireland, 2022. Association for Computational Linguistics. doi: 10.18653 /v11 v// 11/2022.aa 231. URL https://aclanthhoogg..orr wrgpo//....r..2022... acl -log-loo..222 234.
12

---

Published as a conference paper at COLM 2024
Eliahu Horwitz, Jonathan Kahana, and Yedid Hoshen. Recovering the Pre-Fine-tuning Weights of Generative Models. Are preprint, abs/ fobs/2400.10208, 2024. org/abs/2402. 10208.
Gabriel Ilharco, Marco Túlio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. The Eleventh Inter- national Conference Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https://openrevierr net wwt./formmmmimm (ttt//orrmmmt-ttttff-------
Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Seo. Knowledge Unlearning for Mitigating Privacy Risks in Language Models. In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pp. 14389-14408. Association for Computational Linguistics, 2023. doi: 10. 10.18653//// 101111855/////0 /Tt/////222..A-LLOOGGG. URL https: 805.
//doi.org/10. 186533/112023.
Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. PubMedQA: A dataset for biomedical research question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-IJCNLP), pp. 2567-2577, Hong Kong, China, 2019. Association for Computational Linguistics. doi: 10.18633/11/11-1122 wrw.///99/.///191225.
URL
Meghdad Kurmanji, Peter Triantafilluu, Jamie Hayes, and Eleni Triantafillou. To- wards Unbounded Machine Unlearning. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Informa- tion Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL http://papers nips 66221116777332222255446666666 cortree--enttttaaa---oooeeeeeeee Conferenceeeeeeeeeeeeeeeeoeeeeee html.
Nathaniel Li, Alexander Pan, Anjali Summer Yue, Daniel Berrios, Alice Gatti, Justin D. Li, Ann-Kathrin Dombrowski, Shashwat Goel, Long Phan, Gabriel Mukobi, Nathan Helm-Burger, Rassin Lababidi, Lennart Justen, Andrew Liu, Michael Chen, Isabelle Barrass, Oliver Zhang, Xiaoyuan Zhu, Rishub Tamirisa, Bhrugu Bharathi, Adam Khoja, Zhenqi Zhao, Ariel Herbert-Voss, Cort B. Breuer, Samuel Marks, Oam Patel, Andy Zou, Mantas Mazeika, Zifan Wang, Palash Oswal, Weiran Lin, Adam A. Hunt, Justin Tienken- Harder, Kevin Y. Shih, Kemper Talley, John Guan, Russell Kaplan, Ian Steneker, David Campbell, Brad Jokubaitis, Alex Levinson, Jean Wang, William Qian, Kallol Krishna Karmakar, Steven Basart, Stephen Fitz, Mindy Levine, Ponnurangam Kumaraguru, Uday Tupakula, Vijay Varadharajan, Ruoyu Wang, Yan Shoshitaishvili, Jimmy Ba, Kevin M. Esvelt, Alexandr Wang, and Dan Hendrycks. The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning. ArXiv preprint, abs/2403.00 24403.33288 2024. URL https://arxivv
2a0ss/2243..00 03218.
Sijia Liu, Yuanshun Yao, Jinghan Jia, Stephen Casper, Nathalie Baracaldo, Peter Hase, Yuguang Yao, Chris Yuhao Liu, Xiaojun Xu, Hang Li, Kush R. Varshney, Mohit Bansal, Sanmi Koyejo, and Yang Liu. Rethinking Machine Unlearning for Large Language Models. 2024. URL https: //axxiv. org/abs/2403 08787.
ArXio preprint,
Ronak Mehta, Sourav Pal, Vikas Singh, and Sathya N. Ravi. Deep Unlearning via Randomized Conditionally Independent Hessians. In JEEE/CVF Conference on Com- puter Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. 10412-10421. IEEE, 2022. doi: R11109//VR2228..... 01017.
URL https:
//doi.org/1e 1tt.1111////226686
Ari S. Morcos, Maithra Raghu, and Samy Bengio. Insights on representational similar- ity in neural networks with canonical correlation. In Samy Bengio, Hanna M. Wal- lach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 31: Annual Conference on Neu- ral Information Processing Systems 2018, NeuriPS 2018, December 3-8, 2018, Montréal,
13

---

Published as a conference paper at COLM 2024
Canada, pp. 5732-5741, 2018. URL UtLLLtttss//pocccdinn.eeero....s
4733d770c6d17773119999 4f htwp://edddenneeeeusssss./ooe/ee
html.
Seth Neel, Aaron Roth, and Saeed Sharifi- Descent-to-Delete: Gradient-based Methods for Machine Unlearning. Vitaly Feldman, Katrina Ligett, and Sivan Sabato (eds.), Algorithmic Learning Theory, 16-19 March 2021, Virtual Conference, Worldwide, volume 132 of Proceedings of Machine Learning Research, pp. 931-962, PMLR, 2021. URL http: //proceedings. mlr. wal...pees//v11 httss//v3///eeeeee /eeee///eeeeeeeee html.
Yasumasa Onoe, Michael J. Q Zhang, Eunsol Choi, and Greg Durrett. CREAK: Dataset for Commonsense Reasoning over Entity Knowledge. In Joaquin Vanschoren and Sai-Kit Yeung (eds.), Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurlPS Datasets and Benchmarks 2021, December 2021, virtual, 2021. URL httppp:///aattsssts-eennmaaaaaa proceedddrn..oooooeo..i....i... nnernnngs..neeueeeeeeeeeeee meurips.. ctp.iissccccpp/cc cosscccerpprrr 2021 /hash/ html.
Martin Pawelczyk, Seth Neel, and Himabindu Lakkaraju. In-context Unlearning: Language
Models as Few Shot Unlearners. ArXiv preprint, abs/2310.07579, 2023. URL https:
//arxiv.org/ hbtrg//bb//310... 07579.
Yotam Perlitz, Elron Bandel, Ariel Gera, Ofir Arviv, Liat Ein-Dor, Eyal Shnarch, Noam Slonim, Michal Shmueli-Scheuer, and Leshem Choshen. Efficient Benchmarking (of Language Models). Xiv preprint, 2308. 11696.
2023. URL https://arxiv org abs/
Ayush Sekhari, Jayadev Acharya, Gautam Kamath, and Ananda Theertha Suresh. Re- member what you want to forget: Algorithms for machine unlearning. In Mare' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: Annual Conference Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, vir- tual, pp. 18075-18086, 2021. URL https: http:///poooeecn...... reuidiinnnnnerriignn ips. cc/ cc/pappp///aee 2021 2221//ashh 9627c45df543c8 (addeeladddddddddddaaa 2222c44465444400363ffa4222266663
Abstract. html.
Robert R Sokal and Charles D Michener. statistical method for evaluating systematic relationships. 1958.
Anvith Thudi, Gabriel Deza, Varun Chandrasekaran, and Nicolas Papernot. Unrolling SGD: Understanding Factors Influencing Machine Unlearning. In 7th IEFF. European Symposium on Security and Privacy, EuroS&P 2022, Genoa, Italy, June 6-10, 2022, pp. 303-319. IÉEE, 2022. doi: 10.1109/ 10.1009/ERROSP53BB.. /EER/OSSSS544222200000000 URL https: /tw. org/ /0/. E009/E00S50.44
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brían Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, lliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open Foundation and Fine-tuned Chat Models. ArXio preprint, abs/2307.00 009888, 2023. URL https://ar/n wwteriivv......eeee. www.oogggg///....oooo.oo abs//307 09288.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Superglue: A stickier benchmark for general- purpose language understanding systems. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett (eds.),
14

---

Published as a conference paper at COLM 2024
Advances in Neural Information Processing Systems 32: Annual Conference Neural In- formation Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 3261-3275, 2019. URL httph:tpppppoooooo http:::://rooccddngs.eeeee neuripp...n http:ppppccpppp..pppppppppppppep 201 10/1//hhh/ Abbtreet.ttbtttttt.htttt.
Lingzhi Wang, Tong Chen, Wei Yuan, Xingshan Zeng, Kam-Fai Wong, and Hongzhi Yin. KGA: A General Machine Unlearning Framework Based on Knowledge Gap Alignment. In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), ACL. 2023, Toronto, Canada, July 9-14, 2023, pp. 13264-13276. Association for Computational Linguistics, 2023. doi: 10.18653 /1122023 18653/v1
URL https://dof org/10.
loog........ 740.
Johannes Welbl, Nelson F. Liu, and Matt Gardner. Crowdsourcing multiple choice science questions. In Proceedings of the 3rd Workshop on Noisy User-generated Text, pp. 94-106, Copenhagen, Denmark, 2017. Association for Computational Linguistics. doi: 10.18653/ v1/W17-4413. URL https:://claatthll..g/ hrtp/////ooo////113333
Prateek Yadav, Leshem Choshen, Colin Raffel, and Mohit Bansal. ComPEFT: Compression for Communicating Parameter Efficient Updates via Sparsification and Quantization. ArXiv preprint, abs abs/2311.13171, 2023. URL http::///aaxii..gg/bb/////// 13171.
Yuanshun Yao, Xiaojun Xu, and Yang Liu. Large Language Model Unlearning. ArXiv preprint, abs abs/2310.10683,, 2023. URL
wttpi//oogg/bb///3///e 10683.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with BERT. In 8th International Conference Learning Represen- tations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL
https://openreview. hwtp://wwwweooormiiisieeeeeerrre
15

---

Published as a conference paper at COLM 2024
A
Additional fine-tuning details
Unless otherwise stated, we perform full fine-tuning in half-precision with stochastic gradi- ent descent and learning rate of 3e with constant learning rate scheduling and gradient clipping of 1. Initial results with Adam were similar but required more memory. We fine- tune for 100 epochs with early stopping based on validation accuracy. We use batch size of 3 which was the largest batch size that would fit in V100's memory. We fine-tune only on the response and never on the prompt. We fine-tune for 100 epochs or until the training set reaches 99% accuracy.
For our forgetting procedure, we randomly select either the correct response or the distractor before fine-tuning the model on that response in each epoch. Since allowing arbitrarily large learning rates can always lead to forgetting, we selected a learning rate where forgetting oc- gradually
multiple epochs, le 4. To prevent undertraining, run the forgetting procedure for 100 epochs or until the model's 's test accuracy drops below 50%, whichever comes first.
B Reduced dataset size
Task
Small Forget Accuracy
Large Forget Accuracy
PIQA
0.71
0.69
ARC Easy
0.84
0.86
ARC Challenge
0.66
0.50
CREAK
0.71
0.77
BoolQ
0.77
0.50
SciQ
0.84
0.76
PubMedQA
0.73
0.63
MathQA
0.52
0.56
ToxiGen
0.80
0.77
CoLA
0.59
0.50
MRPC
0.77
0.80
MultiLII
0.61
0.79
QNLI
0.71
0.50
RTE
0.57
0.50
WNLI
0.97
0.97
CB
0.61
0.50
COPA
0.51
0.50
WiC
0.62
0.50
WSC
0.63
0.66
Figure 7: Small dataset forgetting. To explore whether we have enough sample points for forgetting, we also run an experiment where only 100 examples used for forgetting instead of 1000 in the large setting. We find that certain datasets exhibit less forgetting with the smaller dataset. However, the general trends remain the same, showing that the problem due explained fully by dataset size.
16

---

Published as a conference paper at COLM 2024
C Flipped-label task
Chailinge
6.3
0.5
6.3
0.0
6.4
CREAK
0.0
0.0
6.1
1.0
0.3
6.0
0.3
6.0
6.6
0.2
6.5
0.5
6.4
0.7
0.4
0.0
0.5
0.7
BoaK)
0.0
0.3
033
0.2
6.7
0.1
0.2
6:6
6.6
9-2
0.2
6.5
0.5
6.5
6.6
0.4
0.0
0.5
0.7
SoQ
0.1
0.1
0.3
0.5
0.6
0.2
0.3
0.3
0.6
0.2
0.3
0.5
0.5
6-6
6.6
0.5
6.0
0.4
0.6
PLRMENIQ
0.0
0.0
6.1
0.5
0.2
0.0
0U6
0.3
0.6
0.7
0.7
0.5
0-4
6.5
0.3
6.0
0.5
0.4
MudiQ4
0.4
MRRC
MuMNNII
6.1
0.3
6
ONLI
0.0
0.0
6.1
0.2
0.3
0.0
0.0
0.3
0-6
9-2
6.5
6
1.0
6
6.6
6.3
6.1
KTE
0.0
0.3
6.2
0.5
0.0
0.0
0.3
0.0
0.G
0.3
6.8
6.5
0.5
1.0
6.6
9.5
6.1
WNLL
0.1
0.0
0.1
0.2
6.2
0.0
00.1
0.0
0.6
0-4
0.5
0.5
6-6
6.9
0.3
6.0
0.5
0.7
CB
0.0
0.0
6.1
0.3
0.1
0.0
0..1
0.0
0.6
0.3
0.3
0.5
0.5
0.7
0.5
0.0
0.5
0.6
COTA
6.2
0.4
0.8
PHON
ABC E309
Caalleng
Bool0 NEAA
Sal
PabbedGA
MotiOA
MtLMINL
QNLI
RTB
WNLI
C9
CORN
WIC
W50
PIQA
Euy
Chullange
CREAK
6.6
1.0
Panlo
0.1
644
6-4
1.0
9.9
D.5
1.0
SdQ
0.1
0.2
0.5
0.7
1.0
0.0
0.4
6.2
1.0
0-4
1.0
6.1
1.0
3.0
Pa
0.1
0.0
6:3
6.9
0.4
0.0
0.1
0.2
1.0
3.0
6.1
1.0
0.9
MathQA
0.1
0.3
0.2
0.9
6.3
0.0
0.1
6.2
1.0
3.0
1.0
3.0
TuaiGen
0.1
1.0
0.3
1.0
1.0
1.0
MuhINL
ONLI
0.33
3.0
1.0
RTT
0.1
6.3
1.0
6.0
0.2
6.0
1.0
1.01
0.66
WNLI
0.1
0.0
0.2
0.3
0.2
0.0
0.0
0.0
1.0
0-4
6.4
6.9
1.0
L.0
0.6
0.1
1.0
3.0
CB
0.0
0.3
6.2
0.5
0.2
0.0
0.1
0.0
1.0
9.4
6.5
6.7
6.6
L.0
0.5
6.1
1.0
3.0
CDRN
0.1
0.3
0.4
0.4
0.3
0.0
0.3
0.5
1.0
0.4
0.4
0.B
3.0
6.8
L.0
0.7
6.3
1.0
3.0
6.4
0.7
1.0
0.8
1.0
1.0
PIDA
ARC E333
:cballegge
GREAK
3000
S40
PubMedOA
Natho
CoLA
XRR2C
MontNLI
GNLI
RTE
WXL
C9
CO8K
WSC
Figure 8: Flipped label forgetting. Top: Forget ratios for forgetting on the flipped task (higher values indicate more successful forgetting) vs Boffom: Forget ratios on the randomized task. We fine-tune the model random/ flipped labels from The vertical axis displays the task the model was trained forget and the horizontal axis displays the task the model was evaluated on. We see similar trends forgetting generalization lization both task constructions.
task and then evaluate the model another task.
17

---

Published as a conference paper at COLM 2024
D Other language models
0.7
Chailinge
0.3
6.1
0.9
6.5
0.0
1.0
CREAK
0.1
0.1
64
0.6
6.5
0.0
0.1
0.4
1.0
9.8
1.0
1.0
BooK)
0.0
0.3
6.4
1.0
6.6
0.0
0.3
0.5
1.0
3.0
0.8
0.1
1.0
1.0
Sa1
0.1
0.1
644
6.7
6.9
6.0
0.2
0.5
2.0
0.1
1.0
0.6
PLRMENIQ
0.1
0.1
0.3
1.0
0.6
6.1
0.3
4.4
6
622
1.0
3.0
MudiQ4
0.6
1.0
1.0
1.0
MRRC
MUINNII
6.4
0.9
2.0
1.0
3.0
ONL
0.1
0.1
6.2
0.9
0.0
0.2
1.0
9.8
1.0
1.0
XTE
0.3
0.3
6:3
0.9
6.4
0.0
0.3
6.3
L.0
0-4
6
G
1.0
L.0
0.4
6.1
1.0
3.0
WNLI
0.1
0.1
6.3
0.8
6.5
0.0
0.1
6-4
1.0
1.0
1.0
0.6
0.1
1.0
1.0
CB
0.1
0.3
6.4
1.0
0.3
0.0
0.1
0.7
1.0
0.4
6.4
Q.B
0.9
6
6.0
0.2
6.1
1.0
3.0
COTA
0.9
0.2
1.0
1.0
PIDA
ABC E333
coalleng
Boo0Q REAA
Sol0
Pab
MatoQA
MRPC
MtLMNNL
CNLI
RTS
WNLJ
C9
0009
WWC
PIQA
Easy
0.2
1.0
Challange
0.7
Panl0
0.0 1.0
6.8
0.0
0.0
0.0
1.0
0.8
6.0
1.00
1.0
SdQ 0.0
0.0
1.0
0.0
6.8
0.0
0.0
0.0
1.0
3.0
1.0
6.5
0.9
1.0
6.0
0.5
0.0
1.0
0.3
PaMMedOA
0.0 0.0
1.0
0.8
0.2
0.0
0.0
0.0
1.0
0.0
6
1.0
6.
0.6
0.0
1.0
1.0
MathQA
0.0 0.0
1.0
1.0
6.7
0.0
0.0
0.0
1.0
3.0
6.4
0.9
6.9
3.0
6.9
3.0
6.0
1.0
3.0
Tontoon
0.0
0.0
1.0
1.0
0.1
0.0
0.0
0.0
0.0
0.0
3.0
1.0
6.9
1.0
3.0
0.0
1.0
3.0
1.0
MBFO
MahNNL
ONLI
2.0
0.5
0.0
0.81
3.0
RTT
1.0
m.3
0.0
0.0
0.0
0.0
1.0
0.
6.0
0.N
6.0
1.00
1.0
WNLI
0.0
0.0
1.0
0.0
0.1
0.0
0.0
0.0
1.0
0.3
6.9
6.5
1.0
1.0
1.0
0.5
0.0
1.0
1.0
CB
0.0
0.0
1.0
1.0
0.0
0.0
0.0
0.0
1.0
0.3
0.2
0.1
0.9
6.0
0.0
0.0
1.0
1.0
CORN
0.0
0.0
1.0
0.1
6.1
0.0
0.0
0.0
1.0
0.0
0.3
0.4
3.0
6.9
1.0
0.7
0.0
1.0
3.0
WIC
0.0
0.0
1.0
0.7
0.0
0.0
0.0
0.0
1.0
0.0
0.8
0.6
3.0
6.6
1.0
0.7
0.0
0.8
3.0
1.0
MON
ARC E399
:cballegge
ORAAK
Bool
8e0
PubMedQK
Nathon
COLA
XRR2C
MOBNNII
GNLI
RTB
WNLI
CB
CORK
WBCC
Figure 9: Forgetting performance of other models. Top: Forget ratios for cross task on the randomized task for GPT-J-6B (higher values indicate more successful forgetting) vs Bottom: Forget ratios for cross-task forgetting the randomized task for GPT-2. The vertical axis displays the task the model was trained forget and the horizontal axis displays the task the model was evaluated on. We see similar trends in forgetting generalization in both models and also the Llama2-7B model, which is shown in the bottom of Figure 8.
18


Reviews:
[{"review_id": "8SZt3bdupl", "reviewer": "Reviewer_7Lpc", "summary": "The paper investigates the unlearning process in language models by fine-tuning on randomized labels. The paper is presented clearly with a systematic analysis of the factors affecting the generalization of unlearning. The results show that the effectiveness of unlearning is task-dependent and if often shallow. However, the analysis is limited to random-label unlearning, and the reasons behind the generalization of unlearning are not well-explained.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments and suggestions. We will revise our paper as follows.\n\n**Method of unlearning is limited to fine-tuning with randomized labels.**\n\nFollowing your question, we ran another experiment where we used flipped labels instead of randomized labels. The analysis is the same, except now we compute the forget ratio as:\n$$\\frac{\\text{Fine-tuned Accuracy} - \\text{Forgotten Accuracy}}{\\text{Fine-tuned Accuracy} - (1- \\text{Fine-tuned Accuracy})}$$\nsince we assume that the minimum accuracy achievable should be $1- \\text{Fine-tuned Accuracy}$.\n\nAs shown in the [figure](https://ibb.co/P6Vp038), the trends are the same. The same tasks that are robust to fine-tuning on randomized labels continue to be robust on fine-tuning on flipped labels. Thus, our results are not specific to the choice of randomized labels, but are rather a property of how fine-tuning and the tasks interact.\n\n**Extending the experiments to another language model besides LLaMA might prove more insights into the generalizability of unlearning.**\n\nWe conducted a new experiment with GPT-J-6B, which is a slightly weaker model than the LLaMA-2-7B model we used. As shown in the [figure](https://ibb.co/pRfwByd), while GPT-J has a lower fine-tuned accuracy, the forgetting ratio trends are broadly the same. Thus, the behavior is not unique to LLaMA-2-7B.\n\nMoreover, to explicitly test what happens to models of different sizes, we also run a new experiment on GPT-2, which has 124M parameters (98\\% smaller). As shown in the [figure](https://ibb.co/TK9x7j4), GPT-2 tends to have significantly lower fine-tuned accuracy, but the forgetting ratio trends are broadly the same even in the smaller model size regime.\n\nWe will add these experiments and their discussion.\n\nAs the generality of our findings was judged to be the main weakness, we hope this cleared the doubts about the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper presents a novel investigation into the unlearning process in language models. The results show that the effect of unlearning can be task-dependent and can be uneffective for many tasks, which calls for a deeper understanding of the unlearning process and more effective methods for unlearning with language models.\n\n* The paper is well-written, with a clear structure and presentation of the experimental results. The authors present a comprehensive analysis of multiple factors that could affect the generalization of unlearning, such as dataset difficulty, model confidence, representation variability, and learning orders.", "reasons_to_reject": "* Method of unlearning is limited to fine-tuning with randomized labels, and it is unclear from the paper whether the failure of generalization of unlearning is due to the unlearning method being ineffective or due to the nature of the language model and the tasks.\n\n* Absent of clear explanation on why unlearning generalizes for some tasks and not for others. The paper claims that the effectiveness of unlearning generalization is task-dependent, but the reasons behind this are not well-explained. This limits the practical implications of the findings.", "questions_to_authors": "* Would other unlearning methods yield different results in terms of generalization? For example, using label-flipping, label transformation, or label of a different task on the same inputs?\n\n* Would the size of the model affect the generalization of unlearning? Also, extending the experiments to another language model besides LLaMA might provide more insights into the generalizability of unlearning.", "extra_scores": {}}, {"review_id": "tlgqlC754D", "reviewer": "Reviewer_Rju9", "summary": "This paper explores unlearning of skills for language models. Specifically they explore what is the extent to which the unlearning procedure of finetuning w/ random label generalizes. They find that generalization is very task dependent but surprisingly does not depend on what dataset was used for the unlearning procedure. The paper then analyzed some factors which could be helpful to predict what can or cannot be unlearned — dataset difficulty, model confidence, and how ‘broad’ the task is. Lastly the paper shows that unlearning even if it generalizes is still shallow – we can train linear probes for the skill very easily.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": true, "final_justification": "", "rebuttal": "Thank you for your insightful comments and suggestions. We will revise our paper as follows.\n\n**\"The discussion of prior work and relevant literature is somewhat lacking.\"**\n\nWe took the suggestion to expand prior work to heart and will do so. For arxiv.org/abs/2403.03218, they use the same term \"generalization\" but study a different kind of generalization--- not breaking unrelated tasks. Our paper focuses on generalization within a single task. We will add this discussion.\n\n**\"The paper only looks at one method (training w/ random labels).\"**\n\nInstead of studying generalization in all possible unlearning methods, we focus on fine-tuning which is simple and a common baseline.\n\nAdditionally, fine-tuning is especially interesting because it may reveal connections between learning and unlearning (e.g. there seems to be wide vs narrow tasks, which we imagine would have consequences in learning).\n\nTo further address this, we replicated our results with a new experiment using flipped labels instead of randomized labels. The analysis is the same, except now we compute the forget ratio as:\n$$\\frac{\\text{Fine-tuned Accuracy} - \\text{Forgotten Accuracy}}{\\text{Fine-tuned Accuracy} - (1- \\text{Fine-tuned Accuracy})}$$\nsince we assume that the minimum accuracy achievable should be $1- \\text{Fine-tuned Accuracy}$.\n\nAs shown in the figure ibb.co/P6Vp038, forgetting trends are the same. Our results are not specific to the choice of randomized labels.\n\n**\"Only one model.\"**\n\nWe conducted a new experiment with GPT-J-6B. As shown in ibb.co/pRfwByd, the forgetting ratio trends are broadly the same, so the behavior is not unique to LLaMA-2-7B.\n\nMoreover, to test what happens to models of different sizes, we also run a new experiment on GPT-2 with 124M parameters. As shown in ibb.co/TK9x7j4, the forgetting ratio trends still remain the same.\n\n**\"Do you have a hypothesis/guess for why it doesn't matter what dataset or task you use?\"**\n\nWe suspect that certain capabilities are inherently more fragile, so fine-tuning on any incorrect data is harmful. This resonates with recent findings that highly quality datasets are important.\n\n**\"How was the threshold of 0.6 determined?\"**\n\nWe picked a threshold of 0.6 which is between 0.5 and the original confidence. The experiments are robust to thresholds in that neighborhood.\n\n**Typos, clarifications, and adding data**\n\nThank you. We will amend the typos, make the suggested clarifications, and add the requested results to the appendix.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The problem studied seems quite important and relevant! Often in areas like knowledge editing people look at the specificity of the editing procedure — this paper explores a similar idea but for unlearning. \n\n2. The datasets used seem quite exhaustive, and the evaluation setup seems pretty standard (LM evaluation harness) which is good.\n\n3. The paper has some interesting findings which I believe could be both impactful and practically useful.\n\n    a. They show that there is sometimes asymmetry between learning and forgetting process (figure 2 top)\n\n    b. Figure 3 – It shows that it doesn't really matter what task/dataset we use for unlearning – the effects of unlearning on different dataset remain the same. This is very surprising and counterintuitive. \n    \n    c. Figure 4 – The overall finding that model confidence and how niche/broad the task (as measured by hidden state variance) are correlated with forget ratio is also pretty interesting.\n\n4. The paper is well written, and I think all result figures seem thorough and informative.", "reasons_to_reject": "1. While the datasets explored are quite exhaustive, the methods for unlearning and the models explored are very limited. The paper only looks at one method (training w/ random labels) but there are a lot of other methods proposed in the literature; but more importantly only one model Llama2-7B is explored — it’s unclear if the findings are more general or specific to this model. I do think the paper could be much stronger by adding some more models in the analysis.\n\n2. The discussion of prior work and relevant literature is somewhat lacking\n\n    a. It seems like other papers have studied generalization of unlearning –e.g. https://arxiv.org/abs/2403.03218 shows that their proposed unlearning method can maintain performance on other domains (e.g. biology, computer science). A more detailed discussion and comparison to prior work would be useful. \n\n    b. (minor) There are also works from slightly different domains which seem relevant and could be included here — generalization of the editing/unlearning procedure (also called as ‘ripple effects’) in some papers – https://arxiv.org/abs/2307.12976, https://arxiv.org/abs/2305.01651 etc. ; methods like iterative null space projection (https://arxiv.org/abs/2004.07667) and follow-ups.", "questions_to_authors": "1. The figure 3 findings are pretty interesting! Do you have a hypothesis/guess for why it doesn't matter what dataset or task you use for unlearning?\n\n2. Section 6 – how was the threshold of 0.6 determined? \n\n3. Conclusion second para(typo?) – “methods were not successful” → “methods were successful”? \n\n4. Section 5 – can you add the actual results for training with 100 examples? (maybe in the appendix)\n\n5. Section 5 / figure 4 – it would be useful to state explicitly that you use finetuned accuracy as a measure of the hardness of the task.", "extra_scores": {}}, {"review_id": "u7JgwoNIhW", "reviewer": "Reviewer_1y8m", "summary": "This work presents an analysis of targeted forgetting in LLMs via random-label fine-tuning. Random-label fine-tuning is a previously proposed approach to train an LLM to \"unlearn\" certain tasks: this work presents an analysis of when and why random-label fine-tuning actually results in \"unlearning\" of a task beyond the data that the model is trained to forget.\n\nThe authors fine-tune an a Llama2 7B LLM on 21 NLP tasks with their true labels and then fine-tune the resulting model on those tasks with random-labels. They show that not all tasks are effectively unlearned by random-label fine-tuning: while some tasks achieve random-chance on test data after unlearning, other tasks still generalize well to their test data. They also evaluate \"cross-task forgetting\", where they analyze the impact of unlearning a task on the generalization of other tasks; here they find, surprisingly, that many tasks are consistently forgotten during unlearning, regardless of the task being unlearned.\n\nThe authors then analyze why certain tasks are easily forgotten when others are not. They rule out several possibilities, showing low correlation between forgetting and task difficulty (measured as gold-label fine-tuned accuracy) and that more data does not improve forgetting. They instead find that a model's confidence (the probability of the correct response to the incorrect response) and the variance of the penultimate representations for each task are similarly correlated with forgetting (less confident tasks are easier to forget and tasks with similar representations across inputs are easier to forget).\n\nNext, the authors study the order in which samples are learned vs. unlearned. They show that there is a loose relationship here, finding that samples which are learned later are forgotten earlier during fine-tuning, although samples that are learned early can also be forgotten early.\n\nFinally, the authors present an analysis of how forgetting can affect the information stored within the intermediate representations of a model by training linear probes before and after training to forget. They find that, generally, training a model to forget a task does not significantly diminish the information of that task in the intermediate layers, as linear probes can often still generalize well to tasks after forgetting has occurred.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments and suggestions. We will revise our paper as follows.\n\n**\"I feel that the work would be improved if a depth-first approach to the analysis had been taken, i.e. if more resources had been devoted to understanding why certain tasks are always forgotten\"**\n\nWe agree that there is more work required to understand why certain tasks are always forgotten when other tasks are not. We did try other more bespoke approaches (including analyzing distances in representation space) but found no more success than in the simple factors already presented.\n\nAs you have stated, our main observation is that \n> fine-tuning on random labels does not always generalize well across tasks but instead that certain tasks are almost always forgotten during cross-task forgetting is an interesting and important observation.\n\nOur main contribution is observing this and considering immediate explanations that might come to mind. This intriguing finding definitely raises more questions for follow-up work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Targeted forgetting is an important topic in LLMs that is still poorly understood and this work takes steps in improving our understanding of it. The main observation that fine-tuning on random labels does not always generalize well across tasks but instead that certain tasks are almost always forgotten during cross-task forgetting is an interesting and important observation.\n\nThe work then presents a series of broad analyses that individually improve our understanding of when and why certain tasks are forgotten over others, and what the underlying mechanisms behind forgetting might be.", "reasons_to_reject": "While the initial findings on the generalization of unlearning are interesting and the breadth of analysis of the rest of the work is impressive, I find the final sections (5-7) to be somewhat shallow individually.\n\nMore specifically, I think the work makes several interesting observations (certain tasks are always forgotten during unlearning regardless of target task, certain factors are somewhat correlated with unlearning generalization, hard samples are forgotten first, task-specific information is still retained in the layers). However, there is not a cohesive take-away from this analysis, i.e. it is not clear how this information should be used to improve targeted forgetting. I feel that the work would be improved if a depth-first approach to the analysis had been taken, i.e. if more resources had been devoted to understanding why certain tasks are always forgotten, or using the analysis in section 5 to present interventions that could improve forgetting.\n\nAs it stands, I think the work is weakened by the fact that the series of findings in 5-7 do not have a cohesive and clear impact on our ability to improve forgetting.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vQVev4McdU", "reviewer": "Reviewer_Leia", "summary": "The paper studies how \"unlearning\" of LM capabilities affects LM generalization. The main procedure is simply to finetune a LM on a target task and then fine-tune again with the labels being randomized. . The authors demonstrate with a series of analyses that this random-label unlearning procedure has variance in its effectiveness across tasks. The authors also show that unlearning does not in fact completely remove all information learnt during pre-training. It is possible to still use some linear probes to get correct responses.", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments and suggestions. We will revise our paper as follows.\n\n**\"How different is this random label distribution from the original label distribution? What happens if the majority of labels match the original labels? Also, I would also like to know how the much variance is there in the unlearning process because of this randomness.\"**\n\nWhen preparing our datasets, we binarized our multiple-choice questions to always have two answers: the correct answer and a distractor. Thus, when we randomize our labels each epoch, there is a 50\\% chance of it being the correct label. Note that the effect of randomizing our labels every epoch when combined with the fact that we are training on about 1000 examples for each task means that it is highly unlikely that a significant majority of labels match the original labels. The variance of label selection really is not that significant given all this repeated randomization.\n\nTo demonstrate this, we also have a new experiment where we train on flipped labels instead of using randomized labels. Thus, in this setup, there is no variance due to randomized label selection.\n\nThe analysis is the same, except now we compute the forget ratio as:\n$$\\frac{\\text{Fine-tuned Accuracy} - \\text{Forgotten Accuracy}}{\\text{Fine-tuned Accuracy} - (1- \\text{Fine-tuned Accuracy})}$$\nsince we assume that the minimum accuracy achievable should be $1- \\text{Fine-tuned Accuracy}$.\n\nAs shown in the [figure](https://ibb.co/P6Vp038), the trends are the same. The same tasks that are robust to fine-tuning on randomized labels continue to be robust on fine-tuning on flipped labels. Thus, our results are not specific to the choice of randomized labels, but are rather a property of how fine-tuning and the tasks interact.\n\n**\"Some of the discussions about the experiments are vague/ambiguous. It is really hard to understand what the author means by a LM's \"forget capabilities\" or \"unlearning of skills\". It would be nice to explain these more in the paper or use different wording completely that is clearer.\"**\n\nThank you for pointing this out. We will revise our paper and explain those terms. Specifically in regards to \"forget capabilities\" and \"unlearning of skills,\" we use these interchangeably. In practice these refer to the out-of-sample performance on the tasks degrading--- which is what we call forgetting. We will be more careful with defining these terms and using them consistently.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The contribution is novel.\n2. The work explores both upsides and downsides of unlearning in significant detail. The authors do run an exhaustive series of experiments to demonstrate their claims.", "reasons_to_reject": "1. It is unclear how the choice of random labels during the second phase of finetuning impacts the results. From my understanding of the paper, the labels are randomized. How different is this random label distribution from the original label distribution? What happens if the majority of labels match the original labels? Also, I would also like to know how the much variance is there in the unlearning process because of this randomness.\n1. Some of the discussions about the experiments are vague/ambiguous. It is really hard to understand what the author means by a LM's \"forget capabilities\" or \"unlearning of skills\". It would be nice to explain these more in the paper or use different wording completely that is clearer.", "questions_to_authors": "1. Can you explain the impact of the choice of random labels on your experiments in more details?", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper studies unlearning in LMs, whereby an LM is "finetuned" on a task with randomized labels. The paper empirically studies the effects of this procedure, especially, how forgetting does or does not generalize across tasks. The findings are likely to have implications for model editing and targeted forgetting, an important topic in LLMs.

Reviewers were all positive. Summarizing my reading of the reviews and the papers, and taking the author responses into account (which have ameliorated some issues), I believe the following are relevant:

Strengths:
- novel (7Lpc) and important (Rju9, 1y8m, Leia)
- interesting empirical results (Rju9, 1y8m, Leia)

Weaknesses:
- no explanation of why the effect of unlearning depends on tasks (7Lpc) and, as a consequence, unclear take-aways (1y8m)

Taken together, I concur with the reviewers that this is a novel approach to studying the effects of model editing, an important topic for today's LMs. While the observed phenomena are not explained in the paper, I believe the paper still has sufficiently interesting and rigorous empirical findings.
Decision: Accept