Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Memorization is a key emerging privacy risk in recent large-scale language models, wherein models can remember and regurgitate verbatim samples from the training set. This paper proposes to alleviate this problem via "knowledge unlearning", which is essentially running stochastic gradient *ascent* on samples that have been identified as memorized. The paper explores a few details on how this is done correctly (proposing sequential unlearning), and shows that the method can mitigate many verbatim generations.
strength_and_weaknesses: Strengths:
* Memorization is a key problem to stop, and more empirical techniques are needed that do not sacrifice performance as differential privacy does. 

Weaknesses:
* One of my big concerns is regarding the privacy evaluation of the method. The evaluation focuses specifically on preventing exact verbatim memorization. A trivial baseline for stopping verbatim memorization is to just manually block the model from generating exact strings in the training set, potentially using a suffix tree like is done in Kandpal et al. 2022 and Katherine Lee et al. 2022. Of course, this trivial baseline cannot prevent models from privacy risks such as (1) generating paraphrases of training texts, (2) leaking private information in a QA or dialogue setting (e.g., ask someone's SSN), and (3) cannot stop membership inference attacks. To show the benefits of the method, it'd be great to see results in some of the above (or related) evaluations. For example, run the membership inference methods from Kandpal et al. 2022, Carlini et al. 2020, or others.
* Definitely checkout the privacy onion paper (Carlini et al. 2022) for another potential risk---unlearning some data can expose others.
clarity,_quality,_novelty_and_reproducibility: * I have some concerns regarding the technical novelty. The general idea of gradient ascent on some data to cause models to avoid learning is an idea that has been well-explored in the literature. For example, the canonical Ganin et al. 2015 paper that proposes "gradient reversal" layers. More recently, there is the large body of work on machine unlearning. Although the paper is correct that this is mainly focused on CV and basic regression tasks, there isn't a ton of new technical novelty required to apply the ideas to NLP.   
* I find it weird to discuss de-duplication as a sort of baseline. Deduplication should be almost always used as a first data processing stage to mitigate memorization (Kandpal et al. 2022; Carlini et al. 2022), and in that sense, it's quite orthogonal to the method proposed here. 
* The quality and clarity is otherwise great, easy to read and follow. Reasonable experiments.

summary_of_the_review: I find the method to be quite similar to past ideas and the privacy evaluation to be a bit limited.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work performs knowledge unlearning in large language models by attempting to fine-tune a converged model for a few epochs with a negative loss corresponding to the examples to the forgotten. As compared to past work which considers data duplication as a strategy towards reducing memorization, this work shows that knowledge unlearning can be very efficient, and does not reduce the LM performance significantly.

The authors also propose a metric to quantify if an example is forgotten, and call it the EL or Extraction Likelihood. Coupled with a metric for memorization, the work 

Some other interesting findings include that the domain of the data to be removed has a large impact on the ease of forgetting of the example. Further, removing small batches of data sequentially is better than removing large batches at once.
strength_and_weaknesses: ## Interesting Results and Strengths
1. The paper is very well written and follows easily, providing enough background to the reader wherever needed.
2. The experimental setup is comprehensive with LM performance compared to a variety of tasks. And benchmarks the results for future work to improve upon.
3. Many of the experimental decisions taken in this work were interesting: for example, using OPT method to replicate a deduplicated dataset, and using data from the Extraction dataset to measure unlearning.
4. While the LMs scale to larger sizes, it takes fewer epochs for the target sequences to be forgotten. 
5. Sequential Unlearning is more Stable than Batch Unlearning. Do you have any intuition as to why this happens?

## Weaknesses
1. Thresholds: The thresholds should be set as mean + k* standard deviation and not just the mean. This current approach does not guarantee that with a high probability the examples are forgotten. How do the results change after making this modification?
2. EL_n value: How was the value of n set to 10
3. The Ablation/Understanding section of this paper is weak. In particular, the authors do ask all the right questions: "Why Are Some Instances Harder to Forget?" "Why Are Some Instances Harder to Forget?"  but the study only provides empirical evidence to rehash the same phenomenon without much understanding gained for the reader. The experimental study rather answers the question: "Are Some Instances Harder to Forget?" and so on.
4. Comparisons with prior work: While there is no significant literature for unlearning in the LM domain, it would have been nice for the authors to utilize the compare the best unlearning algorithms in the vision domain, and see if they can leverage the advances in the vision domain in the NLP domain.
5. The authors use the term "guarantees privacy" for model extraction attacks. I would not use terminology like guarantee for empirical evidence.
clarity,_quality,_novelty_and_reproducibility: While there may not be a lot of novelty in the methods designed in this work, I do believe that it has a significant contribution to the field of machine unlearning in large LMs. The quality of writing is good, and the authors provide code for reproducibility.
summary_of_the_review: I find that this paper does an important job of benchmarking initial results for unlearning in the context of large language models. The paper presents an optimistic picture for LMs suggesting that even with baseline unlearning methods, one can achieve forgetting without hurting LM perform. The experimental study is comprehensive and allows future work to build upon with the released code.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a simple method for unlearning training samples, to comply with GDPR (and other privacy acts') right-to-be-forgotten statement, which gives each person the right to delete their data at any time they want. The proposed unlearning method for language models involves doing a gradient ascent (instead of the usual descent) step, minimizing the likelihood of the training samples. The authors empirically show that this negative step can decease the memorization of the sample, based on the metrics they introduce themselves. They also compare the unlearning method they introduce to a deduplication baseline.  They also demonstrate that unlearning samples sequentially provides a better utility, than unlearning samples all at once.
strength_and_weaknesses: Strengths:

1. The method is simple and intuitive and has nearly no overhead. 
2. The problem is timely and relevant.


Weaknesses:

1. There are some experimental and design decisions that seem arbitrary and I don't really see the reasoning behind: 
    a. The choice of baseline: I don't really see why deduplication is chosen as a baseline to compare to? It seems the most irrelevant and also the most compute-intensive one. The proposed method is not at all similar to deduplication, which is a pre-processing method. Also, deduplication itself is really not a valid 'privacy protection method. I think the most appropriate baseline would actually be doing differential privacy, like DP-SGD, and then doing a comparison of memorization. So for instance if you wanna delete sample `a', you'd train a model on the full dataset, then unlearn it using your method, and then measure `a's memorization before and after. Then, you'd compare this with memorization of 'a' but under a model trained with DPSGD. DPSGD is the best point of comparison as by design, it is supposed to protect the membership, so it is like a pro-active approximate deletion method.

b. The proposal of the privacy metrics: I wonder why the authors made up their own metrics, and also came up with an arbitrary privacy guarantee which is based on their two metrics. Why not just use membership inference attack recall [1,2] and exposure metric [3], which are commonly used and established metrics? These two basically do what the currently proposed metrics do. 

c. apart from not having an appropriate baseline, the paper also fails to even qualitatively compare, or even introduce in related work, to 'approximate deletion' methods and other descent-based methods such as [4,5] which are also aimed at deletion, w/o full re-training. 


2. Figure 1 is actually inaccurate in terms of DP, as you do not need to re-train with DP every time you get a deletion request, given how DP is actually defined. 

[1]   Shokri, Reza, et al. "Membership inference attacks against machine learning models." 2017 IEEE symposium on security and privacy (SP). IEEE, 2017.

[2] Mireshghallah, Fatemehsadat, et al. "Quantifying privacy risks of masked language models using membership inference attacks." arXiv preprint arXiv:2203.03929 (2022).

[3] Carlini, Nicholas, et al. "The secret sharer: Evaluating and testing unintended memorization in neural networks." 28th USENIX Security Symposium (USENIX Security 19). 2019.

[4] Izzo, Zachary, et al. "Approximate Data Deletion from Machine Learning Models." arXiv preprint arXiv:2002.10077 (2020).

[5] Neel, Seth, Aaron Roth, and Saeed Sharifi-Malvajerdi. "Descent-to-delete: Gradient-based methods for machine unlearning." Algorithmic Learning Theory. PMLR, 2021.
clarity,_quality,_novelty_and_reproducibility: The paper is well-written and well-structured. The proposed method seems novel to me and the paper seems reproducible. However it is lacking justification for the choice of baselines and metrics.
summary_of_the_review: The paper's idea is interesting, simple, and effective. However, I think a better baseline and better metrics are needed to be able to actually say that it outperforms other methods. I think a comparison with DP is needed, and also a comparison with other approximate deletion methods is also needed.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose a simple approach for unlearning specific token sequences in large pretrained language models (LMs). To unlearn a specific sequence of tokens x, the proposed approach simply negates the original training objective of minimizing the negative log-likelihood of x; this procedure is also known as unlikelihood training. To measure the effectiveness of unlearning, the authors introduce two metrics, extraction likelihood (EL), and memorization accuracy (MA). EL is measured as the average n-gram overlap between the output of the LM given the tokens in x up to token t and the ground-truth tokens greater than or equal to t. MA is the fraction of tokens output from the LM that match the tokens in x. The "forgetting" of x is achieved when the EL and MA of x is lower than the average EL and MA over token sequences in a validation set not seen during training.

Experiments are performed using three different sizes of the GPT-NEO LM pretrained on all the PILE corpora. After unlearning, the LM is evaluated using 9 different domain datasets spanning a wide range of downstream NLP tasks. The results show the proposed unlearning approach achieves better unlearning (evaluated using the proposed metrics described above) than the most relevant competing method (a data pre-processing de-duplication approach) while maintaining better utility on most the of downstream NLP tasks; this is especially evident as the size of the GPT-NEO model increases. The authors also find their proposed approach is up to 3,500,000x more efficient than retraining from scratch.
strength_and_weaknesses: Strengths
---
Unlearning for large LMs is certainly a timely problem since large LMs have exploded in popularity, and are used as the foundation for almost all downstream NLP tasks, exacerbating any privacy concerns inherent in the pretrained LM. Thus, this work tackles an important problem of helping to mitigate some of the privacy issues that are likely to arise through the widespread use of large pretrained LMs.

The proposed approach is very simple yet effective and extremely efficient in comparison to retraining from scratch, which is generally intractable for large pretrained LMs. This makes the proposed approach very practical and likely to be used in the real world; it is also a much more appealing option than data preprocessing and expensive differential privacy methods.

Experiments using various LM model sizes, and evaluations on 9 different domain datasets provide decent evidence the proposed approach is a potentially viable unlearning method that warrants further study.

Weaknesses
---
The forgetting guarantee requires extra validation data.

No results that vary the size of n, the hyperparameter used for the EL metric. This can give better insight into how much of the target examples are unlearned when the prompt varies in size. Additionally, how much does the success of the EL metric vary depending on which n tokens are used as a prompt for this metric?

Only a small number of examples (32) are randomly selected to be unlearned. Have the authors tried unlearning much larger portions of the training data and observing the effect on the resulting model?

It is still not quite clear to me why unlearning a larger batch of examples (128 vs 32) performs significantly worse than sequentially unlearning smaller batches of 32 examples at a time.

Questions/Comments
---
How is the "hardness" of forgetting defined. Is it how much predictive performance changes when unlearning an example? Or, is it defined as the number of epochs to successfully achieve forgetting for the given input?

The improved forgetting and utility performance of the larger GPT-NEO model compared to the smaller versions is interesting. I would think these results suggest unlearning is "easier" for larger LMs with potentially more representational power.

Do the authors have any insight into the relation between a model that uses the proposed unlearning mechanism and a model retrained from scratch without the target training examples to be removed?

Minor Weaknesses
---
Table 3 is not color-blind friendly, consider showing prefixes as underlined text instead of blue-colored text.

Footnote 6 (in the conclusion) should come after the period.
clarity,_quality,_novelty_and_reproducibility: The work appears novel, the writing is clear and is easy to read, and the approach is simple and straightforward enough that reproducing the results is likely feasible.
summary_of_the_review: Given the ubiquity of large pretrained LMs in NLP today, concerns related to privacy are likely to arise, thus new proposals for tackling these privacy implications is important. This work presents a simple, efficient, and effective approach for targeted unlearning in large LMs, and is likely worthy of further research.

Post-Rebuttal
---
This work applies gradient ascent to a new domain, namely large language models that are typically prohibitively expensive to retrain, enabling an efficient way to remove specific training sequences post-hoc. However, the empirical evaluation does not seem to fully support the main claims made in this paper. A more thorough investigation analyzing the predictive performance of the LLM as more instances are unlearned would greatly improve this paper and ultimately help readers and practitioners better understand the limitations of this approach. Also, better understanding the difference between sequential and batch unlearning would also significantly improve this paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The main contribution of this work lies in proposing the use of gradient ascent on equation 1 for unlearning to provide empirical privacy guarantees for large language models.

This paper has generated considerable response to the authors' rebuttal and an active discussion from the reviewers. 

While we have found the findings interesting, two main concerns remain after reviewing the authors' responses:

(A) It is unclear if the authors' claim of "the simple approach of gradient ascent ... results in little to no degradation of general performance for LMs" would indeed hold when larger datasets are unlearned. In particular, when would be the "breaking point"? Reviewers 9RUh and 4t9e have both raised questions regarding this, but have not received a satisfactory response. Answering this question would allow us to better understand concretely the extent and specificity of this claim.

(B) Related to A, it is not clear why unlearning a larger batch of examples (128 vs 32) performs significantly worse than sequentially unlearning smaller batches of 32 examples at a time. The authors were not able to give a convincing explanation. Addressing this concern will require the authors to perform a careful detailed investigation. 

The authors are strongly encouraged to revise their work based on the above and the reviewers' other feedback and suggestions.
justification_for_why_not_higher_score: See the aforementioned main concerns (A) and (B).
justification_for_why_not_lower_score: N/A.
Decision: Reject