Unforgettable Generalization in Language Models

Authors: eric  zhang
leshem  choshen
jacob  andreas
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-22 15:53:49

Abstract: When language models (LMs) are trained to ``unlearn'' a skill, does this unlearning generalize? We study the behavior of LMs after fine-tuned on data for a target task (e.g. sentiment analysis) in which the labels have been randomized, a popular unlearning method. While LMs consistently learn to generate near-random predictions for individual training examples in the unlearning set, there is extreme variability across tasks in whether LM predictions change on examples outside the unlearning set. In some tasks (like sentiment analysis), unlearning generalizes robustly, and causes models to generate random outputs on all sentiment-type inputs; in other tasks (like physical commonsense reasoning and scientific question answering) unlearning produces almost no generalization at all, and models continue to perform the task accurately even for examples very similar to those that appeared in the training set. Across tasks, we find that dataset difficulty is not predictive of whether a behavior can be unlearned; instead, generalization in unlearning is (weakly) predicted by the confidence of LMs' initial task predictions and the variability of LM representations of unlearning data, with low confidence and low variability both associated with greater generalization. Finally, we show that even generalizable unlearning is shallow: linear probes trained on LMs' representations can still perform tasks reliably after unlearning. Our results highlight the difficulty and unpredictability of performing targeted skill removal from models via fine-tuning.

Main paper: 
 introduction: 
 In the modern approach to training language models (LMs), neural sequence models are first pre-trained on a large, minimally curated corpus (typically of web text), then fine-tuned with targeted demonstrations and human feedback. The LMs that result from this procedure often possess undesirable capabilities that creators do not wish to expose to users-for example, the ability to generate hate speech, or to answer questions about topics unrelated to the LM's target application. Can these capabilities be forgotten (or "unlearned")?There has been widespread recent interest in developing and evaluating new techniques for removing both skills and declarative knowledge from LMs. This work has found that, on specific inputs of interest, LM behavior can be changed in targeted ways. But there has been comparatively little evaluation of generalization in forgetting-when an LM is trained not to respond (or to respond uninformatively) to a particular input, how does its behavior change on other inputs?This paper studies generalization behavior in forgetting. We focus on forgetting of skills (rather than knowledge) via fine-tuning on randomly labeled data for the target task-a simple, widely used, and often highly effective method for forgetting (see Liu et al., 2024 for a recent survey). Surprisingly, we find wide variability across tasks in the effectiveness and generalization of random-label forgetting. When fine-tuning on randomized responses, models will change their behavior on training inputs, but sometimes do not change their behavior at all for other instances of the same task-even when fine-tuning on accurate labels does lead to generalized improvements in accuracy. In additional experiments characterizing generalization in forgetting, we find:1. The degree of forgetting is largely determined by the tasks that LMs are evaluated on, not the task LMs are trained to forget.2. Generalization in forgetting is not determined by the difficulty of the task.3. Properties that correlate with the generalization of forgetting include LM confidence as well as the variance of LM representations of training data.4. Despite LMs' inability to respond correctly to prompts after applying this method, we are still able to recover the correct responses using linear probes. Hence, even successful forgetting is at best shallow, and does not remove information from LMs' representations.Generalization of learning algorithms across problems and problem instances is a major focus of study in machine learning research. Our results show similarly complex, structured cross-task variability of generalization in forgetting, and underscore the need for additional research on the relationship between the training data used for forgetting and the effect of model predictions elsewhere. 
related work: 
 Due to diverse privacy, security, and ethical concerns, machine unlearning has been conceptualized in many different ways. Early approaches defined unlearning as removing undesirable data from training sets (Cao & Yang, 2015; Bourtoule et al., 2021; Ginart et al., 2019) . These approaches often require fundamental changes to model structure and/or training process, which is often infeasible.Later work relaxed the requirement of removing data from the training set. Instead, models are required to behave similarly to models trained without undesirable data points, or are simply required to stop producing outputs with desirable features. Guo et al. (2020) develop a framework for linear classifiers, and Golatkar et al. (2020a) develop a method that scrubs information from linear probes. Neel et al. (2021) ; Sekhari et al. (2021) ; Thudi et al. (2022) ; Golatkar et al. (2020b; 2021) ; Mehta et al. (2022) and Chundawat et al. (2023) present theoretical frameworks for comparing an unlearned network to a fully-retrained networks, and they propose optimization-based methods to find unlearned network under additional assumptions like convexity. Foster et al. (2024) propose model editing techniques based on estimating parameter importance using fisher information. Kurmanji et al. (2023) distinguishe between different reasons for forgetting, arguing that distinct purposes like protecting user privacy, resolving confusion, and removing biases require distinct metrics. Graves et al. (2021) argue that selectively removing training data alone is insufficient, and propose a new threat model and techniques to address them.For language models specifically, approaches to remove specific facts include gradient ascent on undesirable responses (Jang et al., 2023; Yao et al., 2023; Eldan & Russinovich, 2023) , prompting with misinformation (Pawelczyk et al., 2023) , linearly manipulating model representations (Ilharco et al., 2023; Belrose et al., 2023) , non-linearly perturbing model representations (Li et al., 2024) , and using new models to teach another model how to forget (Wang et al., 2023) .While some of this prior work has studied generalization (e.g. Li et al., 2024) , they study a different kind of generalization: whether model behavior remains the same on non-targeted tasks. By contrast, our work focuses on generalization between instances of a single task.Outside of research on unlearning, some past work has studied training on incorrect or random labels as a source of information about learning dynamics, for example finding that models often have similar embeddings (Morcos et al., 2018) , learn in a similar order (Hacohen et al., 2020) and explaining the order of learning (Hacohen & Weinshall, 2022) . 
experiment setup: 
 Method Our experiments in this paper study forgetting of capabilities (rather than factual knowledge). In order to enable uniform comparisons across tasks, we formulate each capability as binary multiple-choice question answering task. Each such task T is associated with a training set T train , a validation set T val , and test set T test . When studying forgetting, we first fine-tune the model on T train with early stopping performed by finding the checkpoint with the highest accuracy on T val . Afterwards, we train the model to forget by fine-tuning the model again on T train but with labels chosen uniformly at random. This procedure is summarized in Figure 1 .We quantify forgetting with two metrics. The first is the gap between the accuracy after forgetting and the expected random accuracy (50% since the tasks are binary multiple choice), which we will call the forget gap:Forget Gap = Task Accuracy After Forgetting -1 2A gap of 0 indicates that the target task has been fully forgotten (all tasks involve a binary choice, and a random baseline obtains an accuracy of 1 2 ). Larger values indicate that models still achieve non-trivial accuracy. We may also wish to interpret accuracy after forgetting relative to the upper bound provided by fine-tuning-an accuracy of 55% after forgetting might be interpreted as successful or unsuccessful if fine-tuned accuracy is 95% or 56%. To quantify this intuition, we define the forget ratio:Forget Ratio =Accuracy After Fine-Tuning -Accuracy After Forgetting Accuracy After Fine-Tuning -1 2Here an forget ratio of 1 corresponds to complete forgetting, while a forget ratio of 0 corresponds to no decrease relative to the best attainable supervised performance. Our experiments first fine-tune a pre-trained LM, then train it further on random labels. We call the gap between the forget accuracy and the random chance accuracy (50%) the forget gap. In many tasks we find a nonzero forget gap: after training on random labels, LMs do not generalizably learn to produce random outputs on new task instances.We experiment on 21 multiple-choice tasks commonly found within the literature. Commonsense Reasoning: We evaluate PIQA (Bisk et al., 2020) , ARC easy and challenge (Clark et al., 2018) , and CREAK (Onoe et al., 2021) . Reading Comprehension: We evaluate BoolQ (Clark et al., 2019) , SciQ (Welbl et al., 2017) , and Pub-MedQA (Jin et al., 2019) . Math: We evaluate MathQA (Amini et al., 2019) . Toxicity: We evaluate ToxiGen (Hartvigsen et al., 2022) .We evaluate CoLA, MNLP, MRPC, QNLI, RTE, WNLI, CB, COPA, WIC and WSC (Wang et al., 2019) . We selected these tasks to cover a broad spectrum of capabilities while also ensuring that they are multiple choice, which allows us to easily construct randomized alternatives for forgetting.We follow the Language Model Evaluation Harness standards for 0-shote evaluation (Gao et al., 2023) , including the default prompts and evaluation through probabilities of the choices. To facilitate comparison across tasks, we binarize the tasks by preserving two of the possible responses-the true response and one randomly chosen distractor-for each example. We evaluate models by picking the response with the highest average token likelihood and reporting the accuracy.We use the publicly provided train, validation, and test sets. However, we found some datasets had train-test overlap. To decontaminate the datasets, we do not evaluate on questions that appeared in the training set. We also removed samples longer than 2048 characters in the prompt and combined response. Where validation sets do not exist, we use the test set. Unless otherwise specified, we limit each set to 1000 examples and subsample if needed, making training results more comparable and evaluation more efficient as proposed by Perlitz et al. (2023) .All experiments use Llama2 7-billion parameter base models (Touvron et al., 2023) . Additional details may be found in Appendix A. 
does forgetting generalize?: 
 Figure 2 summarizes the task accuracy without modification, after fine-tuning, and then after running our forgetting procedure. Test accuracy almost always increases after finetuning, although it could decrease slightly as the validation set is not identical to the test set. During the forgetting phase, however, we observe several distinct categories of behavior (1) forget accuracy is very similar to the fine-tuned accuracy, (2) forget accuracy decreases but is still above the pre-trained accuracy, and (3) forget accuracy decreases to below the pretrained accuracy and possibly back to 50%. Case ( 2) is interesting because it demonstrates asymmetry between the learning and forgetting process, as the model is unable to forget what is has just learned (analogous to hysteresis in physical systems; Ewing, 1882).Overall, we find that random-label forgetting often fails to generalizably remove the target behavior, but with wide variability across tasks. In general, tasks involving commonsense knowledge reasoning tasks are more resilient to forgetting, whereas lower-level linguistic acceptability and entailment classification tasks are more effectively forgettable.We also examine cross-task forgetting, where we fine-tune the model on random labels from the training set of one task and then evaluate the model on the test set of another task. As shown in Figure 3 , we find that the effectiveness of the forgetting procedure is largely determined by the tasks that the model is evaluated on-not the training task. Another surprising observation is that many tasks are more effectively forgotten when training on randomized labels of other tasks than from training on their own randomized labels. As observed in the individual task evaluation, GLUE tasks focused on specific capabilities are again more susceptible to forgetting in general, whereas commonsense reasoning tasks are more resilient to forgetting. Training on forgetting commonsense reasoning tasks are also generally more effective at triggering forgetting for other tasks. 
when does generalization occur?: 
 Does forgetting require more examples? We rule out the number of examples as the main explanation to forgetting generalization. For example, a possible concern could be that forgetting does not generalize because there are not enough training examples. We ran the same experiment with 100 examples of each task as well as 1000 (above). We find that despite an order of magnitude change, the level of forgetting is similar in both cases.To rule out the possibility that forgetting fails to generalize due to our method of training on randomized labels, we run another experiment where we train on flipped labels instead of randomized labels. The analysis is the same as before, except now we compute the forget ratio as: Forget Ratio = Accuracy After Fine-Tuning -Accuracy After Forgetting Accuracy After Fine-Tuning -(1 -Accuracy After Fine-Tuning) since we assume the minimum accuracy achievable should be 1-Accuracy After Fine-Tuning. Figure 2 : Single task forgetting. Top: The blue arrow visualizes the change in held-out accuracy after fine-tuning and the red arrow illustrates the change in accuracy after forgetting. We find that many tasks do not return to the expected accuracy of 50% after forgetting. Bottom left: The forget gap (difference between forgetting accuracy and the expected random accuracy of 1/2) across tasks. Smaller values correspond to a greater degree of forgetting. Bottom right: The forget ratio (the difference fine-tuned accuracy and the forget accuracy over the difference between fine-tuned accuracy and the expected random accuracy of 1/2). Larger forget ratios correspond to more successful forgetting. Figure 3: Cross-task forgetting (higher values indicate more successful forgetting). We fine-tune the model on random labels from one task and then evaluate the model on another task. The vertical axis displays the task the model was trained to forget and the horizontal axis displays the task the model was evaluated on. Surprisingly, certain capabilities are robust to forgetting even after fine-tuning on random labels. Moreover, the effectiveness of the forgetting procedure is largely determined by the tasks that the model is evaluated on, not the tasks that the model was trained to forget. Note that rows and columns are presented in different orders, and clustered using the UPGMA algorithm (Sokal & Michener, 1958) As shown in Figure 8 , the trends are the same. The same tasks that are robust to fine-tuning on randomized labels continue to be robust on fine-tuning on flipped labels. Thus, our results are likely not specific to the choice of randomized labels, but rather a property of how fine-tuning and the tasks interact.To understanding whether this forgetting behavior is unique to the LLama2 7-billion parameter model or to language models in general, we also experiment with GPT-J-6B, which is a slightly weaker model than the LLaMA-2-7B, and GPT-2, which is a significantly smaller model with 124M parameters (98% smaller).As shown in Figure 8 , while GPT-J and GPT-2 have lower fine-tuned accuracy, the forgetting ratio trends are broadly the same. Thus, the behavior is not unique to LLaMA-2-7B. Each point is a different task. Top: The accuracy on the task after fine-tuning. The effectiveness of the forgetting procedure is not determined by the difficulty of the task (as measured by accuracy).Middle: The variance of the hidden state of the last token of the question in the fifth to last layer across examples. This variance is somewhat predictive of amount forgotten, indicating that "broader" tasks are more difficult to forget. Bottom: Model's confidence in the correct response. Probability relative to the distractor is predictive of forgetting, indicating that models forget more examples they were already not confident about.Are harder tasks harder to forget? Another plausible explanation for why certain tasks are forgotten less is that harder tasks are more difficult to forget. However, as plotted in Figure 4 , this is not consistently true. As a selected example, the forgetting procedure is less effective for the ARC easy dataset in comparison to the ARC challenge dataset, despite the significantly greater difficulty of the latter. Thus, the effectiveness of forgetting must be determined by other properties of the task.We hypothesize that a model's confidence may be predictive of whether a task is forgotten. The reasoning for this is that if the model has a strong preference for its answers on the task, a larger parameter update may be needed to overcome this "prior".We examine the model's confidence in the correct response prior to running the forgetting procedure. Since the probability of the correct response is not calibrated, we measure the probability of the correct response relative to the incorrect response.The results are shown in Figure 4 . We find that the model's confidence in the correct response is partially predictive of how much the model forgets. Note that this is distinct from the difficulty of the task, as the model's confidence in the correct response is not necessarily correlated with whether it is actually correct.We also hypothesize that "broader" tasks are harder to forget. Since similar text is often mapped to similar regions in the latent space (Zhang et al., 2020) , we use the variance of the hidden states of the model to quantify how much the model is able to forget. Specifically, we extract the hidden states at the last token of the question at the penultimate layer. We find that the total variance (trace of the covariance matrix) is predictive of how much the model is able to forget. Figure 4 shows that the smaller the total variance, the more effective the model is in forgetting. Note that this measure does not require access to the labels of the dataset, and it only requires access to the inference capabilities of the model and data from the task at hand. Figure 5 : Forgetting order vs learning order. The horizontal axis shows the forgetting time: the number of epochs until the model forgets (assigns ¡ 60% accuracy to the correct response for a data point). The vertical axis shows the learning time: the number of epochs until the model learns (assigns ¿ 60% confidence to the correct label for a data point). We filter out the examples that are never learned or never forgotten. If fewer than 100 examples fulfil the criteria, we do not plot the task. Overall, we find that learning and forgetting orders are weakly, but consistently, anticorrelated.Can we predict which examples will be forgotten? In contrast to the task-level trends depicted in Fig. 4 , we did not observe any correlation between any of the above metrics and models' behavior at the level of individual examples-for example, example-level model confidence is not predictive of example-level forgetting. We hypothesize that different effects may dominate in this finer-grained scope, and that focusing on a narrow scope of same-task examples, other effects we did not yet uncover are too strong to see an effect with the current traits, such effects can be investigated in further work.6 What is the relationship between learning and forgetting?Even if extrinsic measures of difficulty cannot predict example-level learnability (as shown in the final experiment above), is there any systematic relationship between learnability and forgettability? Motivated by earlier work that similar architectures share consistent learning orders (Hacohen et al., 2020; Choshen et al., 2022) , we hypothesize that the learning and forgetting orders are related.As pre-trained models are often already partially capable of performing the tasks we study, we analyze learning orders after "resetting" the models to either extreme of the learning spectrum (maximum forgetting or maximum fine-tuning). Specifically, we compare the learning order of when we (1) run the forgetting procedure after fine-tuning (the same as in Section 4) and ( 2) when we run the fine-tuning procedure one more time afterwards (run fine-tuning after procedure in Section 4). Note that to prevent the models from learning all the examples in one epoch, we use a different fine-tuning learning rate of 3e-5 for experiment (2).To qualify an example as learned, we require the model have a confidence of at least 0.6 in the correct response. To qualify an example as forgotten, we require the model have a confidence of at most 0.6 in the correct response. In preliminary experiments, we did not find results to be sensitive to the choice of threshold. For the purpose of analysis, we ignore examples that are never learned or forgotten. If no more than 100 examples fulfill the criteria, we do not plot the task. We take the first time this occurs as the forget time/learn time.We visualize the learning orders in Figure 5 . Across tasks, we find a consistent, modest correlation between learning order and forgetting order, in which the first points to be learned are typically the last to be forgotten and vice-versa. Overall, we hypothesize that the lack of a stronger correlation may be due to the shallow nature of fine-tuning. Since we are only aligning the model to the task instead of teaching it new capabilities, the learning order may be unaffected by example-level properties like difficulty. Thus, the learning order may be more related to the model's initial state.7 Are "forgotten" skills truly removed from models?One further question is if training on random labels really erases models' capabilities or if it only censors the output. To examine this, we train a linear probe on the models hidden states after performing the forgetting procedure. The probes are trained on the training set and evaluated on the test set. ℓ 2 regularization and early stopping on a validation set used to prevent overfitting as the hidden state dimension is often larger than the number of examples. We select the fifth last layer of the model as the hidden state to probe, as we find that the accuracy of probing is mostly comparable for all layers except for the very early layers and the very late layers.The results are shown in Figure 6 . We find that the fine-tuning procedure largely does not influence the probing effectiveness. Thus, this procedure induces at best a shallow forgetting. This is consistent with most work that fine-tuning is often a shallow operation that does not significantly alter the model's capabilities (e.g.; Yadav et al., 2023; Horwitz et al., 2024) . 
conclusion: 
 In this paper, we study the effectiveness of fine-tuning models on randomized responses in order to forget capabilities. We find that this method is effective for certain tasks, but surprisingly does not generalize for others. The degree of forgetting seems mostly determined by the tasks that the model is evaluated on, not the tasks that the model was trained to forget. We find that dataset difficulty and model confidence are not predictive of whether a task is forgotten. However, we find that the total variance of the hidden states of the model is predictive of how much the model is able to forget. Finally, we show that despite the models' inability to respond correctly to prompts after applying this method, we are still able to recover the correct responses using linear probes. Thus, this is at best a shallow type of forgetting and not true removal of information from the model.Future work can focus more on understanding which specific examples are forgotten and why. While our methods were successful in predicting which broad capabilities are forgotten, they are not predictive of which specific examples are forgotten within a task. This suggests that there are more mechanisms at play that can be studied further. 
 
 Appendix: Unless otherwise stated, we perform full fine-tuning in half-precision with stochastic gradient descent and a learning rate of 3e -3 with constant learning rate scheduling and gradient clipping of 1. Initial results with Adam were similar but required more memory. We finetune for 100 epochs with early stopping based on validation accuracy. We use a batch size of 3 which was the largest batch size that would fit in V100's memory. We fine-tune only on the response and never on the prompt. We fine-tune for 100 epochs or until the training set reaches 99% accuracy.For our forgetting procedure, we randomly select either the correct response or the distractor before fine-tuning the model on that response in each epoch. Since allowing arbitrarily large learning rates can always lead to forgetting, we selected a learning rate where forgetting occur gradually over multiple epochs, 1e -4. To prevent undertraining, we run the forgetting procedure for 100 epochs or until the model's test accuracy drops below 50%, whichever comes first. 7 : Small dataset forgetting. To explore whether we have enough sample points for forgetting, we also run an experiment where only 100 examples are used for forgetting instead of 1000 in the large setting. We find that certain datasets exhibit less forgetting with the smaller dataset. However, the general trends remain the same, showing that the problem is not due explained fully by dataset size. 


Decision: This paper studies unlearning in LMs, whereby an LM is "finetuned" on a task with randomized labels. The paper empirically studies the effects of this procedure, especially, how forgetting does or does not generalize across tasks. The findings are likely to have implications for model editing and targeted forgetting, an important topic in LLMs.

Reviewers were all positive. Summarizing my reading of the reviews and the papers, and taking the author responses into account (which have ameliorated some issues), I believe the following are relevant:

Strengths:
- novel (7Lpc) and important (Rju9, 1y8m, Leia)
- interesting empirical results (Rju9, 1y8m, Leia)

Weaknesses:
- no explanation of why the effect of unlearning depends on tasks (7Lpc) and, as a consequence, unclear take-aways (1y8m)

Taken together, I concur with the reviewers that this is a novel approach to studying the effects of model editing, an important topic for today's LMs. While the observed phenomena are not explained in the paper, I believe the paper still has sufficiently interesting and rigorous empirical findings.

Reviews: review: paper_summary: paper_summary: The paper investigates the unlearning process in language models by fine-tuning on randomized labels. The paper is presented clearly with a systematic analysis of the factors affecting the generalization of unlearning. The results show that the effectiveness of unlearning is task-dependent and if often shallow. However, the analysis is limited to random-label unlearning, and the reasons behind the generalization of unlearning are not well-explained.
strength_weakness: strength_weakness: * Method of unlearning is limited to fine-tuning with randomized labels, and it is unclear from the paper whether the failure of generalization of unlearning is due to the unlearning method being ineffective or due to the nature of the language model and the tasks.

* Absent of clear explanation on why unlearning generalizes for some tasks and not for others. The paper claims that the effectiveness of unlearning generalization is task-dependent, but the reasons behind this are not well-explained. This limits the practical implications of the findings.
questions: questions: * Would other unlearning methods yield different results in terms of generalization? For example, using label-flipping, label transformation, or label of a different task on the same inputs?

* Would the size of the model affect the generalization of unlearning? Also, extending the experiments to another language model besides LLaMA might provide more insights into the generalizability of unlearning.
paper_summary: paper_summary: This paper explores unlearning of skills for language models. Specifically they explore what is the extent to which the unlearning procedure of finetuning w/ random label generalizes. They find that generalization is very task dependent but surprisingly does not depend on what dataset was used for the unlearning procedure. The paper then analyzed some factors which could be helpful to predict what can or cannot be unlearned — dataset difficulty, model confidence, and how ‘broad’ the task is. Lastly the paper shows that unlearning even if it generalizes is still shallow – we can train linear probes for the skill very easily.
strength_weakness: strength_weakness: 1. While the datasets explored are quite exhaustive, the methods for unlearning and the models explored are very limited. The paper only looks at one method (training w/ random labels) but there are a lot of other methods proposed in the literature; but more importantly only one model Llama2-7B is explored — it’s unclear if the findings are more general or specific to this model. I do think the paper could be much stronger by adding some more models in the analysis.

2. The discussion of prior work and relevant literature is somewhat lacking

    a. It seems like other papers have studied generalization of unlearning –e.g. https://arxiv.org/abs/2403.03218 shows that their proposed unlearning method can maintain performance on other domains (e.g. biology, computer science). A more detailed discussion and comparison to prior work would be useful. 

    b. (minor) There are also works from slightly different domains which seem relevant and could be included here — generalization of the editing/unlearning procedure (also called as ‘ripple effects’) in some papers – https://arxiv.org/abs/2307.12976, https://arxiv.org/abs/2305.01651 etc. ; methods like iterative null space projection (https://arxiv.org/abs/2004.07667) and follow-ups.
questions: questions: 1. The figure 3 findings are pretty interesting! Do you have a hypothesis/guess for why it doesn't matter what dataset or task you use for unlearning?

2. Section 6 – how was the threshold of 0.6 determined? 

3. Conclusion second para(typo?) – “methods were not successful” → “methods were successful”? 

4. Section 5 – can you add the actual results for training with 100 examples? (maybe in the appendix)

5. Section 5 / figure 4 – it would be useful to state explicitly that you use finetuned accuracy as a measure of the hardness of the task.
paper_summary: paper_summary: This work presents an analysis of targeted forgetting in LLMs via random-label fine-tuning. Random-label fine-tuning is a previously proposed approach to train an LLM to "unlearn" certain tasks: this work presents an analysis of when and why random-label fine-tuning actually results in "unlearning" of a task beyond the data that the model is trained to forget.

The authors fine-tune an a Llama2 7B LLM on 21 NLP tasks with their true labels and then fine-tune the resulting model on those tasks with random-labels. They show that not all tasks are effectively unlearned by random-label fine-tuning: while some tasks achieve random-chance on test data after unlearning, other tasks still generalize well to their test data. They also evaluate "cross-task forgetting", where they analyze the impact of unlearning a task on the generalization of other tasks; here they find, surprisingly, that many tasks are consistently forgotten during unlearning, regardless of the task being unlearned.

The authors then analyze why certain tasks are easily forgotten when others are not. They rule out several possibilities, showing low correlation between forgetting and task difficulty (measured as gold-label fine-tuned accuracy) and that more data does not improve forgetting. They instead find that a model's confidence (the probability of the correct response to the incorrect response) and the variance of the penultimate representations for each task are similarly correlated with forgetting (less confident tasks are easier to forget and tasks with similar representations across inputs are easier to forget).

Next, the authors study the order in which samples are learned vs. unlearned. They show that there is a loose relationship here, finding that samples which are learned later are forgotten earlier during fine-tuning, although samples that are learned early can also be forgotten early.

Finally, the authors present an analysis of how forgetting can affect the information stored within the intermediate representations of a model by training linear probes before and after training to forget. They find that, generally, training a model to forget a task does not significantly diminish the information of that task in the intermediate layers, as linear probes can often still generalize well to tasks after forgetting has occurred.
strength_weakness: strength_weakness: While the initial findings on the generalization of unlearning are interesting and the breadth of analysis of the rest of the work is impressive, I find the final sections (5-7) to be somewhat shallow individually.

More specifically, I think the work makes several interesting observations (certain tasks are always forgotten during unlearning regardless of target task, certain factors are somewhat correlated with unlearning generalization, hard samples are forgotten first, task-specific information is still retained in the layers). However, there is not a cohesive take-away from this analysis, i.e. it is not clear how this information should be used to improve targeted forgetting. I feel that the work would be improved if a depth-first approach to the analysis had been taken, i.e. if more resources had been devoted to understanding why certain tasks are always forgotten, or using the analysis in section 5 to present interventions that could improve forgetting.

As it stands, I think the work is weakened by the fact that the series of findings in 5-7 do not have a cohesive and clear impact on our ability to improve forgetting.
paper_summary: paper_summary: The paper studies how "unlearning" of LM capabilities affects LM generalization. The main procedure is simply to finetune a LM on a target task and then fine-tune again with the labels being randomized. . The authors demonstrate with a series of analyses that this random-label unlearning procedure has variance in its effectiveness across tasks. The authors also show that unlearning does not in fact completely remove all information learnt during pre-training. It is possible to still use some linear probes to get correct responses.
strength_weakness: strength_weakness: 1. It is unclear how the choice of random labels during the second phase of finetuning impacts the results. From my understanding of the paper, the labels are randomized. How different is this random label distribution from the original label distribution? What happens if the majority of labels match the original labels? Also, I would also like to know how the much variance is there in the unlearning process because of this randomness.
1. Some of the discussions about the experiments are vague/ambiguous. It is really hard to understand what the author means by a LM's "forget capabilities" or "unlearning of skills". It would be nice to explain these more in the paper or use different wording completely that is clearer.
questions: questions: 1. Can you explain the impact of the choice of random labels on your experiments in more details?

Comments: title: Response to Rebuttal
Rebuttal Response
comment: Thanks for responding to the review! I think the authors have satisfactorily addressed the concerns I had --- specifically they ran new experiments with one more model, as well as a slightly different unlearning method (flipped label). I hope the authors add more discussion about related work as they mentioned in the rebuttal.

Considering this, I'm happy to increase the score.
Thank you for your response. I definitely agree that there is a lot of work still to do to understand this area, and I appreciate that this work makes a meaningful contribution to our understanding with it's core observation.

I still feel as though the paper overall lacks a cohesive takeaway throughout the later sections and so I will keep my score the same (leaning positive because I do feel that the main finding outweighs the negatives).
Thanks for your response. The newly added results seems convincing and addresses my concerns, therefore, I have improved the score. I look forward to see a more comprehensive discussion with the expanded results in the revision of the paper.
Thanks for the detailed response. I have decided to go with my original score. I would encourage the authors to include these clarifications in the revised version of the paper.