UnSTAR: Unlearning with Self-Taught Anti-Sample Reasoning for LLMs

Authors: Yash Sinha, Murari Mandal, Mohan Kankanhalli
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: Machine Unlearning, Large Language Models

Abstract:
The key components of machine learning are data samples for training, models for learning patterns, and loss functions for optimizing accuracy. Analogously, unlearning can potentially be achieved through anti-data samples (or anti-samples), unlearning methods, and reversed loss functions. While prior research has explored unlearning methods and reversed loss functions, the potential of anti-samples remains largely untapped. In this paper, we introduce UnSTAR: $\underline{\text{Un}}$learning with $\underline{\text{S}}$elf-$\underline{\text{T}}$aught $\underline{\text{A}}$nti-Sample $\underline{\text{R}}$easoning  for large language models (LLMs). Our contributions are threefold: first, we propose a novel concept of anti-sample-induced unlearning; second, we generate anti-samples by leveraging misleading rationales, which help reverse learned associations and accelerate the unlearning process; and third, we enable fine-grained targeted unlearning, allowing for the selective removal of specific associations without impacting related knowledge—something not achievable by previous works. Results demonstrate that anti-samples offer an efficient, targeted unlearning strategy for LLMs, opening new avenues for privacy-preserving machine learning and model modification.

Decision: reject
Reviewer scores: 8, 5, 5, 5, 5, 5

Reviews and discussion:
<thread>
creation date:
1730401103355

thread_type:
official_review

<thread_object>
creator: reviewer_f1yx

<content>
summary:
The paper presents a novel approach to unlearning QA pairs in large language models by constructing and using anti-examples. The author evaluated the effectiveness of this framework on the WPU dataset across different types of QA knowledge.

strengths:
1. The paper introduces a new unlearning method.

2. The paper is easy to follow, with each concept and idea illustrated by examples.

weaknesses:
1. I am concerned that intuitively constructing anti-samples may exacerbate hallucination issues, which we aim to avoid in practice. Could the author comment on this concern? (I recognize that the experiments show a score related to hallucination avoidance, but in principle, why would this approach not lead to significant hallucination problems?)

2. The experimental section is not that convincing. First, the paper does not specify important experimental details, such as dataset size, training settings, or the hyperparameters of each unlearning baseline. Prior literature [1,2] has shown that unlearning methods can behave quite differently depending on parameters and tasks. Second, I question the relevance of some metrics used, such as the GPT Privacy Score, GPT Quality Score, and GPT Rejection Rate. Are these metrics widely accepted in the privacy field? It would be helpful if the author could provide more justification. Lastly, the paper focuses on only one dataset and setting (without any ablation studies), which may limit its impact.

[1] Pawelczyk et al., Machine unlearning fails to remove data poisoning attacks. 

[2] Shi et al., Muse: Machine unlearning six-way evaluation for language models.

questions:
See Weaknesses.

Suggestions:

1. In Line 107, there is a typo in the reference to the random label method.

2. In Eq. (1), $p_\mathcal{M}$ refers to the conditional probability of only “answers - $a$,” correct? How is $r$ sampled from the distribution, or does this notation refer to probability $p$? The notation could be made more rigorous.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730520535605

thread_type:
official_review

<thread_object>
creator: reviewer_84co

<content>
summary:
This paper presents UNSTAR, a method for unlearning in large language models (LLMs) through the use of anti-samples. While previous research has focused on unlearning methods and reversed loss functions, this work explores the potential of anti-samples, which are generated using misleading rationales to reverse learned associations and accelerate the unlearning process. UNSTAR enables fine-grained, targeted unlearning, allowing specific associations to be removed without affecting related knowledge. The results demonstrate that anti-samples offer an efficient and precise approach to unlearning.

strengths:
1. This work chooses an interesting data-centric angle to solve LLM unlearning. Reshaping the data for more effective unlearning is promising and well-motivating.

2. This work can achieve fine-grained LLM unlearning by destroying unwanted associations within the forget data, which is an underexplored problem in LLM unlearning.

3. The paper writing is well-organized and easy to follow.

weaknesses:
1. Achieving LLM unlearning from a data perspective is interesting, but the proposed anti-sample has marginal novelty. In my view, it is nearly the same as refusal response or random labels in terms of unlearning. The difference is only to use evidently incorrect words. The anti-sample generation process is quite naïve and lacks technical novelty with the prompt “Generate 20 words similar to this word”. It is more appealing if the generation process is iterative like prompt engineering.

2. In terms of specific methodology, quite a few details are unclear. For example, how UnStar adjusts the prompt to generate Semantically Divergent Questions and Near-Correct Incorrect Answers when the existing generation is semantically too close to the original. In addition, in line 7 of Algorithm 1, what does that mean ‘do similar steps for retain set Qr’?

3. The Reinforcement Learning Style Policy Gradient Approximation is weird, what do the authors want to express? In the claim of lines 270-273, any preference optimization-based unlearning can achieve this separate gradient computation. I don’t think UnStar offers any unique approximation to RL policy learning.

4. testing UnStar on only one dataset is not enough for the experiments. Besides, most implementation details of UnStar and baseline approaches are lacking (cannot be found even in the appendix). There are no statistical error bars or similar metrics. More important, there is no ablation study.

questions:
See the weaknesses above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730667431138

thread_type:
official_review

<thread_object>
creator: reviewer_9vhw

<content>
summary:
This paper introduces a UNSTAR -- a method to unlearning in Large Language Models (LLMs), which leverages "anti-samples" – data designed to induce unlearning by neutralizing or reversing learned associations. The paper evaluates UNSTAR and show that it succeeds in fine-grained targeted unlearning.

strengths:
+ Unlearning is an important topic

weaknesses:
+ Adhoc non-comparable evaluation methods
+ Missing references to prior work
+ I am confused with the base definition and the goal

questions:
+ I am slightly confused regarding the context and the setting in which this work is situated. Classical unlearning, as well as, a lot of works in the related section considers a very different setting -- unlearning for privacy. For example work by Bourtoule et al. (2021) is not comparable to anything in the llm unlearning space, I am also not sure what guarantees are being pursued (e.g. see discussion in [1]). Without defining a precise evaluation metric and what exactly is the reason for that metric I am not sure how one is to assess performance. This performance also matters a lot e.g. [2] show that posing the question is a slightly different way poses a significantly different result. Note that it should also make meaningful sense relative to the overall goal of unlearning. 
+ Random relabeling has been explored in prior literature e.g. [3], which should definitely be discussed. If the papers argument is that the random relabeling is not the random, but is semantic then clear separation from [4] is required. I do not understand how or what is semantically different between wroks. 
+ Ironically, there is a broken citation in the related work section next to the random relabeling. It would be great to see comparison in performance to it, alongside an investigation how it is different if at all. 
+ Related work subsection for unlearning is almost a direct copy of a section from [5]. In both phrasing, citations, and the order in which things appear. I am not claiming this is plagiarism, but it is very close in my eyes.  
+ Table 2 shows how semantically close the responses are. Does that make the task easier? How to verify that it did in fact unlearn the related knowledge.
+ If you change the decoding strategy, what happens with the results? Or if you noise the policy very slightly? I dont understand how to think about hallucination/adversarial eval that is ran in a greedy decoding strategy. 
+ Given that the results are very similar to that of WHP+ and RWHP, yet the impact on the model is different I am left wondering if we are just comparing apples to oranges. 
+ Eval claims to use (Anil et al., 2024; Schwinn et al., 2024) for adversarial evaluations. Both these works are not references in the manuscript, but are clearly copied verbatim from [5]. Yet, the paper does not go into any details, as is discussed in appendix of [5].
+ Adversarial eval details are missing and are not replicable in any way. 

[1] "Ununlearning: Unlearning is not sufficient for content regulation in advanced generative ai" by Shumailov et al
[2] "Inexact Unlearning Needs More Careful Evaluations to Avoid a False Sense of Privacy" by Hayes et al
[3] "Random Relabeling for Efficient Machine Unlearning" by Li and Ghosh
[4] "Who’s Harry Potter? Approximate Unlearning in LLMs" by Eldan and Russinovich
[5] "Revisiting Who’s Harry Potter: Towards Targeted Unlearning from a Causal Intervention Perspective" by Liu et al
</content>
</thread_object>

</thread>
<thread>
creation date:
1730684553639

thread_type:
official_review

<thread_object>
creator: reviewer_qhwt

<content>
summary:
the paper observes that recent progress with anti-sample reasoning could be efficiently used for unlearning purposes. It creates anti-samples by (1) paraphrasing and (2) falsifying arguments for unlearning.

strengths:
- I really liked how accurate and targeted the unlearning could be in terms of concepts, i.e. you can be very selective. 
- Paper figures and visualizations are easy to follow
- Writing is clear and expressive enough
- Really good and intuitive example with Harry Potter, I think it transfer the idea very clearly

weaknesses:
- I am not sure if the novelty of the method is sufficient. Authors have described the existing problem of unlearning and existing method of using STAR and combined these methods together. It does not seem like there are any challenges to this method, however I am happy to be convinced otherwise. 
- Evaluation is not as strong as it only uses one dataset for unlearning and Figure 2 does not split performance by subgroups. Figure 3 is also not clear if it contributes anything to the discussion
- There was no discussion on what constitutes  challenges to unlearning using this method, especially given that the task performance is 100% it would be interesting to study where this method fails --> are all unlearning tasks could be solved with UNSTAR?
- Most importantly the paper does not discuss any membership attacks which are much stronger to test unlearning performance.

questions:
Overall, while I really like the trick to unlearn using anti-samples, I struggle to believe that evaluation is sufficient to prove this claim under more realistic assumptions (MIA, harder cases). Additional evaluation might significantly strengthen the paper.

Also would be good to justify that the method in STAR is challenging to apply in the domain of unlearning and emphasize authors contributions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730692132743

thread_type:
official_review

<thread_object>
creator: reviewer_fwuc

<content>
summary:
This paper offers a new perspective on existing LLM unlearning algorithms by focusing on the data perspective. It proposes a data-driven algorithm that can help LLMs unlearn and provides comprehensive and diverse evaluation metrics to ensure completeness and diversity during assessment.

strengths:
1. This paper considers the problem of transferring learning to unlearning from a macro perspective on LLM learning. It divides the learning methods into three steps and successfully summarizes other methods within these steps, thus uncovering a new approach to tackle unlearning.
2. This paper evaluates a comprehensive range of LLM algorithms in its main experiments and designs various evaluation metrics (particularly metrics related to Response Quality and Hallucination Avoidance), offering more perspectives for understanding LLM unlearning.
3. This paper provides several reasonable constraints when designing prompts to ensure the feasibility of using LLMs for regeneration.

weaknesses:
It appears that the completion level of this paper is not very high. It only includes a comparison of algorithms under different metrics and an analysis of iterations. Although it presents a good method, it still requires some analysis regarding the algorithm’s time complexity. For more detailed weaknesses or questions, please refer to the “Questions” section.

questions:
1. Regarding the algorithm: I noticed that, both in Algorithm 1 and in the steps provided on page 4, the number of iterations mainly depends on the total number  $N$  of generated Paraphrased Questions and Incorrect Answers. The while loop continues until  $Q^*$  is empty. Is it possible that within the same  $Q^*$ , there are several instances where  $\hat{a} = a$ ? If so, does this mean that for the same question, you would fine-tune using all the found tuples $(q^*, \bar{a}, r)$ multiple times? If that is the case, what is the expected average number of repetitions? And what is the relationship between this expectation and  $N$?

2. Regarding the experiments, I am curious about the impact of  N  on the experimental results. Given that this algorithm involves extensive data regeneration and fine-tuning, could the authors compare the time required for different algorithms and examine whether there is a trade-off between time and performance, or if this algorithm is a “free lunch”?

3. Regarding the randomness, for the task of data regeneration, the main challenge lies in uncontrollability. How did the authors address the randomness introduced by the LLM algorithms?

4. Regarding the writing format issues: Line 106 mentions “random labels” (?), and Lines 356-357 reference a paper that cannot be found.
5. No code sources.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730706611395

thread_type:
official_review

<thread_object>
creator: reviewer_dtlx

<content>
summary:
The paper focuses on the idea to use anti-samples coupled with reasoning tuning to enable target machine unlearning. Given a question, the methods utilizes language models to generate incorrect answers and paraphrased questions. The model is then fine-tuned on this as well as induced explanations for incorrect answers. This ensures unlearning quality while only modify the model at "reasoning" step. So the model retain good performance on each individual concepts.

strengths:
The unlearning method is able to achieve targeted unlearning (e.g. dissociation between two concepts) without harming the representation/knowledge of both concepts. 

The encourage of reasoning seems to be an effective way to combat adversarial attacks.

weaknesses:
It seems that the method is significantly more involved than other unlearning methods. There is a lack of comparison of time cost for it.

It also lacks comparison to other representation-based unlearning algorithms such as RMU.

questions:
Any reason to use harmonic mean to combine evaluation metrics?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732252734791

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Dear Reviewers, Thank you very much for your dedication and insightful comments. We have thoroughly gone through your comments, taken your suggestions into deep consideration, and provided comprehensive responses.

We answer below a comment that we believe is of interest to all reviewers. It concerns the performance comparison and run time comparison with existing LLM unlearning methods on additional datasets. We present 3 tables displaying results from new experiments we have run to address comments by more than one reviewer. These tables serve as supporting evidence for several of the arguments we make in our rebuttal. 

We address other details in each reviewer's rebuttal section.

**More supporting results**

We show the time cost comparison with three existing state-of-the-art methods in Table 1.

Table 1: Unlearning time cost comparison of our UnStar with SNAP, WAGLE, and NPO across Harry Potter, Peter Parker, and TOFU datasets. (time in seconds)

| **Unlearning Type** | **Fine Grained** |              |              | **Targeted** |       |       |       |
|----------------------|------------------|--------------|--------------|--------------|-------|-------|-------|
| **# Facts**         | 1                | 1            | 1            | 100          | 100   | 200   | 400   |
| **Dataset**         | Harry Potter     | Peter Parker | TOFU         | Harry Potter | Peter Parker | TOFU | TOFU |
| **UnStar**          | **6**                | **11**           | **8**            | **698**          | **1229**  | **1637**  | **3242**  |
| **SNAP**            | 1907             | 2107         | 2427         | 1839         | 2030  | α | α |
| **WAGLE**           | φ              | φ          | φ          | $            | $     | $     | 4046  |
| **NPO**             | φ              | φ          | φ          | $            | $     | $     | 4015  |

α: SNAP struggles to generate a sufficient number of questions forming distinct clusters via agglomerative clustering, often resulting in prolonged runtimes without clear termination.

φ:  Struggle to work for fine grained unlearning

$:  Omitted: expected to align with 400-fact results.

We also compare the ROUGE-L scores for UnStar with SNAP across three datasets: Harry Potter, Peter Parker, and TOFU in Table 2.

Table 2: Unlearning results comparison with SNAP method.
| **Dataset/Method** | **UnStar** | **SNAP**   |
|---------------------|------------|------------|
| **Harry Potter**    | 0.02997    | 0.14752    |
| **Peter Parker**    | 0.20611    | 0.24044    |
| **TOFU**            | 0.04507    | 0.11362    |

SNAP: Choi M, Rim D, Lee D, Choo J. SNAP: Unlearning Selective Knowledge in Large Language Models with Negative Instructions. arXiv preprint arXiv:2406.12329. 2024.

WAGLE: Jia, J., Liu, J., Zhang, Y., Ram, P., Baracaldo, N., & Liu, S. WAGLE: Strategic Weight Attribution for Effective and Modular Unlearning in Large Language Models. NeurIPS 2024.

NPO: Zhang R, Lin L, Bai Y, Mei S. Negative preference optimization: From catastrophic collapse to effective unlearning. arXiv preprint arXiv:2404.05868. 2024.

TOFU: Maini P, Feng Z, Schwarzschild A, Lipton ZC, Kolter JZ. TOFU: A Task of Fictitious Unlearning for LLMs. ICLR 2024 Workshops

We show the impact of different values of N on the experimental results in Table 1. 

Table 3: Impact of N on the experimental results

| Iteration | N   | Epoch | Accuracy | Time   | Retained after Filtering | Filtered |
|-----------|-----|-------|----------|--------|--------------------------|----------|
| 1         | 0   | 0     | 0.00%    | 0.6509 | –                        | –        |
| 2         | 5   | 10    | 4.59%    | 0.6569 | 5                        | 15       |
| 3         | 20  | 20    | 18.35%   | 0.6809 | 15                       | 5        |
| 4         | 31  | 30    | 28.44%   | 0.6469 | 11                       | 9        |
| 5         | 39  | 40    | 35.78%   | 0.6579 | 8                        | 12       |
| 6         | 51  | 50    | 46.79%   | 0.6449 | 12                       | 8        |
| 7         | 62  | 60    | 56.88%   | 0.6169 | 11                       | 9        |
| 8         | 74  | 70    | 67.89%   | 0.6689 | 12                       | 8        |
| 9         | 92  | 80    | 84.40%   | 0.6369 | 18                       | 2        |
| 10        | 109 | 90    | 100.00%  | 0.5789 | 17                       | 3        |
</content>
</thread_object>

</thread>


Metareview:
metareview:
This submission falls slightly below the acceptance threshold, as reflected in the final review ratings: 8 (Reviewer DtLx, with lower confidence 2), 5 (Reviewer fwuc), 5 (Reviewer qhwt), 5 (Reviewer 9vHW), 5 (Reviewer 84co), and 5 (Reviewer F1yx). While the authors' rebuttal addressed some concerns raised by reviewers, most notably the issue of similarity with [5] in related work, highlighted by Reviewer fwuc, most of the reviewers revealed a consensus that the paper is not yet ready for acceptance in its current form. Reviewer DtLx, who provided the highest score (8), expressed lower confidence in the assessment. Reviewer DtLx highlighted the authors' rebuttal effort but also agreed on concerns that existed in this submission, e.g., the novelty of using anti-samples and the explanation on real-world applications. In light of the reviewers' evaluations, I recommend rejection.