UnSTAR: Unlearning with Self-Taught Anti-Sample Reasoning for LLMs

## Abstract

Abstract The key components of machine learning are data samples for training, model for learning patterns, and loss function for optimizing accuracy. Analogously, unlearning can potentially be achieved through anti-data-samples (or anti-samples), unlearning method, and reversed loss function. While prior research has explored unlearning methods and reversed loss functions, the potential of anti-samples remains largely untapped. In this paper, we introduce UnStar : Un learning with S elf- T aught A nti-Sample R easoning for large language models (LLMs). Our contributions are threefold: first, we propose a novel concept of anti-sample-induced unlearning; second, we generate anti-samples by leveraging misleading rationales, which help reverse learned associations and accelerate the unlearning process; and third, we enable fine-grained targeted unlearning, allowing for the selective removal of specific associations without impacting related knowledge—something not achievable by previous works. Results demonstrate that anti-samples offer an efficient, targeted unlearning strategy for LLMs, opening new avenues for privacy-preserving machine learning and model modification.

## 1 Introduction

In recent years, self-improvement approaches like STaR (Zelikman et al. () and RFT Yuan et al. ()) have shown that large language models (LLMs) can improve themselves through reasoning. Now, imagine using these reasoning processes not to enhance learning, but to guide the model in selectively forgetting specific information, ensuring privacy and control. This concept forms the core of UnStar: Unlearning with Self-Taught Anti-Sample Reasoning for LLMs.

Why unlearn? The ability of LLMs to absorb vast amounts of human-authored content—often viewed as their greatest strength—has also presented concerns over data privacy (Huang et al. ()), copyright violations (Carlini et al. (); Shi et al. ()), and the potential misuse of AI in harmful domains such as bio-weapons and cyber-attacks (Barrett et al. (); Sandbrink (); Li et al. ()). In this context, AI safety necessitates the ability to erase specific information without compromising overall model performance. Thus, how can LLMs effectively unlearn specific knowledge after being trained on extensive text corpora? (Nguyen et al. (); Voigt & Von dem Bussche (); Zhang et al. ()) Legal compliance (Gursoy et al. ()), particularly with privacy laws and copyright regulations, necessitates mechanisms for selective unlearning . Furthermore, ethical considerations drive the need to eliminate biased or harmful data from models, ensuring fair and responsible use. Finally, the removal of obsolete or irrelevant information is essential to maintain models’ accuracy and alignment with evolving requirements.

Ways to unlearn? Machine learning models improve accuracy through training by leveraging three key components: data samples, learning methods, and loss functions. Analogously, unlearning can also be potentially achieved by counteracting one or more of these core elements: anti-data-samples (or anti-samples), unlearning methods, and reversed loss functions. While much attention has been given to unlearning methods (Bourtoule et al. (); Chundawat et al. (); Sinha et al. ()) and the manipulation of loss functions to reverse learning (You et al. (); Sinha et al. ()), the potential of anti-samples remains largely untapped. This paper aims to fill that gap.

In this work, UnStar leverages anti-samples to facilitate unlearning LLMs. A sample is a data point used to train the model. When an unlearning request is made, this sample becomes part of the forget set that we aim to unlearn. An anti-sample is a data point designed to induce unlearning by neutralizing or reversing the association learned from the sample. The key questions are: what constitutes a suitable anti-sample for effectively the inducing unlearning of a sample in the forget set, and how can we generate such an anti-sample?

For an LLM, a sample is a question-answer pair, such as Where did Harry Potter study? Hogwarts School of Witchcraft and Wizardry. To unlearn, UnStar intentionally provides incorrect answers and their justifications as an anti-sample. For instance, it generates Where did Harry Potter study? Ilvermorny. Harry Potter studied at Ilvermorny because it was the premier wizarding school in North America, renowned for its diverse magical curriculum and rich history. This enables the LLM to forget specific information while minimizing disruption to its broader knowledge base. To achieve this, we leverage STaR Zelikman et al. (), a technique originally designed to enhance reasoning in LLMs by generating step-by-step rationales.

In addition to introducing the novel concept of anti-sample unlearning, we demonstrate that previous unlearning techniques can inadvertently disrupt the LLM’s broader knowledge. To address this challenge, we propose fine-grained targeted unlearning, which allows for the selective removal of specific associations. In the aforementioned example, other related facts—such as that Harry Potter is a wizard and Hogwarts is a boarding school of magic for young wizards—should not be forgotten. This capability sets our approach apart from previous methods (Eldan & Russinovich (); Liu et al. ()).

Our contributions are:
❶ Anti-sample induced unlearning: We introduce the novel concept of using anti-samples, rather than typical data samples, to drive the unlearning process.
❷ Misleading rationales as justifications: We employ misleading rationales as justifications to guide the model in forgetting, leveraging reasoning that flips answers rather than reinforcing them.
❸ Fine-grained targeted unlearning: Our approach enables the selective removal of specific associations, such as unlearning that Harry Potter studied at Hogwarts while retaining other relevant facts about both Harry Potter and Hogwarts. This capability distinguishes our method from previous approaches. Our results demonstrate that anti-samples present a promising and efficient strategy for targeted unlearning in LLMs.

## 2 Related Work

Machine Unlearning.
Recent advancements in machine unlearning Cao & Yang (); Bourtoule et al. () span domains like image classification Tarun et al. (); Chundawat et al. (; ); Sharma et al. (), regression Tarun et al. (), federated learning Wu et al. (); Chundawat et al. (), and graph learning Sinha et al. (). Exact unlearning Bourtoule et al. () focuses on modifying the training process to remove the influence of specific data points by retraining the model, ensuring it behaves as if those data were never seen. While this offers strong guarantees, exact unlearning is computationally intensive and typically suited to simpler models.

In contrast, approximate unlearning (Chundawat et al. ()), which focuses on reversed loss functions, reduces the influence of target data points through parameter-level updates, significantly lowering computational costs. Although approximate unlearning doesn’t completely eliminate the influence of the data, it is far more practical for large-scale models where full retraining would be too costly.

Despite their effectiveness, both exact and approximate unlearning methods have largely overlooked the potential of anti-samples. UnStar introduces anti-samples and reasoning to guide the unlearning process in a more granular and efficient manner, offering a promising alternative for precise, targeted model modifications

LLM Unlearning. LLM unlearning has garnered significant research interest as a means to improve privacy, enhance safety, and mitigate bias in large language models Lu et al. (); Kassem et al. (); Wang et al. (); Patil et al. (); Huang et al. (); Yu et al. (); Wu et al. (); Zhang et al. (); Liu et al. (); Jia et al. ()). The predominant approach utilizes gradient ascent to maximize prediction loss on the data to be forgotten (Jang et al. (); Yao et al. ()). Other techniques involve training the model to produce alternative responses, such as “I don’t know” (Ishibashi & Shimodaira ()), random labels (yao2024largelanguagemodelunlearning), or predictions based on perturbed inputs (Eldan & Russinovich ()). Additionally, recent studies have investigated task arithmetic (Ilharco et al. (); Barbulescu & Triantafillou ()) and training-free methods for unlearning by incorporating specific instructions or in-context examples (Thaker et al. (); Pawelczyk et al. ()).

However, these methods often lack the granularity required for fine-tuned control over what specific information is forgotten, which is where our approach—utilizing anti-samples—proposes a more refined solution.

Self-improvement reasoners.
Self-Taught Reasoner (STaR; Zelikman et al. ()) is an iterative method where a language model refines itself through correctness feedback. In each iteration, the model generates solutions for problems, evaluates them against ground truth, and retains only the correct ones. The model is then fine-tuned on this filtered dataset, iteratively improving its accuracy. Rejection Sampling Fine-tuning (RFT; Yuan et al. ()) follows a similar process but is not iterative. Instead, RFT samples multiple solutions for each problem and augments the original dataset with correct completions for fine-tuning. STaR iterations can also incorporate rejection sampling techniques, as in methods like ReSTEM (Singh et al. ()). V-STaR (Hosseini et al. ()) enhances STaR by training a verifier using both correct and incorrect solutions to judge correctness, resulting in more accurate reasoning and verification on benchmarks like math and code generation.

Our work builds upon these reasoning frameworks but repurposes the concept of self-taught reasoning for unlearning rather than improving model accuracy. Instead of refining correct answers, UnStar leverages misleading rationales to generate anti-samples, which in turn aid in the forgetting of specific information. This novel application of reasoning to the domain of unlearning has not been explored in prior works.

## 3 UnStar

Figure: Figure 1: An overview of UnStar. For a question-answer pair in the forget set, paraphrased questions and incorrect answers are generated using LLM. The justification is achieved through “rationalization” based on STaR. Following the unlearning of a question, more challenging paraphrased versions are generated to further enhance the unlearning process.
Refer to caption: x1.png

Problem Formulation.
Let the language model with parameters $\varphi$ be denoted by $\mathcal{M}(\cdot,\varphi)$. Let $\mathcal{Q}=\{(q,a)\}$ represent the dataset of question-answer pairs. Let $\hat{a}=\mathcal{M}(q,\varphi)$ is the answer produced by the model $\mathcal{M}$ for $q$. We define the forget set $\mathcal{Q}_{f}\subset\mathcal{Q}$ as the subset of question-answer pairs related to facts we wish to unlearn (e.g., Harry Potter studied at Hogwarts). The retain set $\mathcal{Q}_{r}=\mathcal{Q}\setminus\mathcal{Q}_{f}$ consists of the remaining question-answer pairs. It holds that:
$\mathcal{Q}_{r}\cup\mathcal{Q}_{f}=\mathcal{Q}\text{ and }\mathcal{Q}_{r}\cap
\mathcal{Q}_{f}=\emptyset.$
Let $\hat{a}^{\prime}=\mathcal{M}(q,\varphi^{\prime})$ represent the answers produced by the unlearned model $\mathcal{M}(\cdot,\varphi^{\prime})$ with updated parameters $\varphi^{\prime}$ for each question $q$. After unlearning, we want the following conditions to hold:
❶ For all $(q,a)\in\mathcal{Q}_{f}$, the answers should no longer match the original:
$\hat{a}^{\prime}\neq a.$
❷ For all $(q,a)\in\mathcal{Q}_{r}$, the model should retain the correct answers:
$\hat{a}^{\prime}=a.$
This ensures that after unlearning, the model provides incorrect answers for the forget set while maintaining the correct answers for the retain set.

Targeted unlearning. Given a language model $\mathcal{M}(\cdot,\varphi)$, update the model to forget all questions $q_{f}$ related to a target $t$: $\hat{a}_{f}^{\prime}\neq a_{f}$, where $(q_{f},a_{f})\in\mathcal{Q}_{f}$ while preserving correct answers for unrelated questions: $\hat{a}_{r}^{\prime}=a_{r}$, where $(q_{r},a_{r})\in\mathcal{Q}_{r}$.

UnStar performs these steps for the forget set $\mathcal{Q}_{f}$.

- 1.
Selection of Question-Answer Pair: Select a question-answer pair $(q,a)$ from the forget set $\mathcal{Q}_{f}$. This pair represents a specific fact that we wish to unlearn.
- 2.
Generation of Paraphrased Questions and Incorrect Answers: Generate $n$ paraphrased versions of the selected question $q$, denoted as $(q^{*}_{0},\ldots,q^{*}_{n})$, and add these to a question bank $\mathcal{Q}^{*}$. For each paraphrased question $q^{*}_{i}$, generate an incorrect answer $\bar{a}_{i}$, forming pairs $(q^{*}_{i},\bar{a}_{i})$, and add them to $\mathcal{Q}^{*}$.
- 3.
Iterative Processing of Paraphrased Questions:
While $\mathcal{Q}^{*}$ is not empty, we proceed with the following steps for each paraphrased question $q^{*}$:
(a)
Answer Generation: Use the model $\mathcal{M}$ to generate an answer $\hat{a}$ for the question $q^{*}$.
(b)
Check for Unlearning:
•
If $\hat{a}\neq a$, mark the paraphrased question $q^{*}$ as unlearned and remove it from $\mathcal{Q}^{*}$.
•
If $\hat{a}=a$, use the incorrect answer $\bar{a}$ to generate a justification $r$.
(c)
Fine-Tune Model: Fine-tune the model using the tuple $(q^{*},\bar{a},r)$ to reinforce the process of forgetting.

The steps are shown in Figure 1. Similarly, UnStar performs these steps for the retain set $\mathcal{Q}_{r}$. In this case, instead of paraphrased questions with incorrect answers, it focuses on generating and confirming that the model $\mathcal{M}$ consistently provides correct answers $\hat{a}=a$ for all question-answer pairs $(q^{*},a)$. The algorithm is presented in Algorithm 1. This ensures that correct knowledge is reinforced and preserved without being affected by the unlearning of the forget set.

Generating Paraphrased Questions and Incorrect Answers. UnStar prompts the original, unlearned LLM to generate $n$ paraphrased versions of the questions, as well as incorrect answers. The specific prompts used for this process are provided in the Appendix. However, three key challenges arise in this context:

❶ Semantically Divergent Questions: LLMs are known to exhibit hallucination tendencies, leading to the generation of questions that may diverge from the intended topics. Therefore, it is crucial to ensure that the paraphrased questions maintain semantic alignment with the original queries. For example, if the focus is on Harry Potter’s education, the paraphrased questions should not stray into unrelated subjects, such as Hermione’s achievements.

To address this issue, UnStar evaluates the semantic similarity between the paraphrased questions and the original queries. This is achieved through a threshold-based fuzzy matching approach, which employs Levenshtein distance to quantify sequence differences, complemented by cosine similarity derived from sentence embeddings generated by a MiniLM-family sentence transformer model (paraphrase-MiniLM-L6-v2), specifically optimized for paraphrase detection and semantic similarity tasks. This dual approach ensures that the generated paraphrases remain focused and aligned with the original intent.

❷ Near-Correct Incorrect Answers: Some generated incorrect answers may be semantically too close to the correct answers, making them unsuitable for effective unlearning. We assess the semantic proximity of these incorrect answers to ensure meaningful divergence from the correct ones. For instance, if the question is, “Was Benedetto Varchi Italian?” and the generated incorrect answer is, “No, Varchi was from Italy,” this case is flagged as a near-correct answer.

To mitigate this issue, we employ semantic similarity measures akin to those used for verifying question alignment, ensuring that the incorrect answers truly diverge from the correct ones.

❸ Continuous Paraphrasing: In cases where the generated paraphrased questions do not lead to effective unlearning, UnStar iteratively prompts the LLM to generate additional challenging paraphrased questions. The specific prompts employed for this iterative process are outlined in the Appendix. This strategy not only enhances the diversity of the dataset but also bolsters its robustness and effectiveness in the unlearning process.

Generating Justifications for Incorrect Answers.
The process of generating justifications for a given incorrect answer in UnStar is achieved through “rationalization” which draws inspiration from the STaR approach (Zelikman et al. ()). Rationalization allows the model to leverage provided answers to generate appropriate rationales, thus improving the unlearning process by guiding the model to reason backward from the answer to formulate relevant rationales.

In our context, when the LLM encounters a question-answer pair that it fails to unlearn effectively, we introduce the incorrect answer as a hint. This aids the model in constructing a justification that logically lead to the provided incorrect answer. For instance, if the model is unlearning the fact “Harry Potter studied at Hogwarts,” we prompt it with an incorrect answer, such as “Ilvermorny,” that guides it to generate a justification like “Harry Potter studied at Ilvermorny because it was the premier wizarding school in North America, renowned for its diverse magical curriculum and rich history in the wizarding world.”

Figure: Algorithm 1 UnStar: This algorithm outlines how to generate anti-samples from the forget set and fine-tune the model while preserving knowledge from the retain set.

Fine-Grained Targeted Unlearning. In addition to targeted unlearning, UnStar has capability of fine-grained targeted unlearning. Let $t^{\prime}$ denote the entity in the answer for the question regarding the target entity $t$. UnStar can selectively unlearn specific associations between $t$ and $t^{\prime}$ and need not unlearn all questions $q$ related to a target $t$: $\hat{a}^{\prime}\neq a$, where $(q,a)\in\mathcal{Q}$.

For instance, consider the question “Where did Harry Potter study?” with the answer “Hogwarts School of Witchcraft and Wizardry.” In this case, UnStar can forget only the association between $t$: Harry Potter and $t^{\prime}$: Hogwarts, while retaining knowledge about other associations or facts. The unlearned model might suggest that Harry Potter studied at a magical school but not specifically at Hogwarts, perhaps suggesting Ilvermorny instead, and it will indicate that Hogwarts is another magical school in the UK. Previous works typically forgot all facts about $t$ while retaining facts about $t^{\prime}$.

Reinforcement Learning Style Policy Gradient Approximation:
UnStar can be viewed as an approximation to a Reinforcement Learning style policy gradient objective. We treat the model $\mathcal{M}$ as a discrete latent variable model defined by $p_{\mathcal{M}}(a\mid q,\varphi)=\sum_{r}p(r\mid q,\varphi)p(a\mid q,r,\varphi)$. In this formulation, the model first samples a latent rationale $r$ before predicting the answer $a$.

The selective unlearning process in UnStar operates with two different indicator reward functions, one for the retain set $\mathcal{Q}_{r}$ and one for the forget set $\mathcal{Q}_{f}$. For $\mathcal{Q}_{r}$, the model is encouraged to give the correct answer using the indicator function $\mathbb{1}(\hat{a}=a)$. For $\mathcal{Q}_{f}$ the model is discouraged from providing the correct answer using the flipped indicator function $\mathbb{1}(\hat{a}\neq a)$.

Thus, the total expected reward across the dataset $\mathcal{Q}$, including both retain and forget sets, can be defined as:

$$ $J=\sum_{i}\mathbb{E}_{\hat{r}_{i},\hat{a}_{i}\sim p_{\mathcal{M}}(\cdot\mid q_ {i},\varphi)}\left[\mathbb{1}(\hat{a}_{i}=a_{i})\cdot\mathbb{1}_{\mathcal{Q}_{ r}}(i)+\mathbb{1}(\hat{a}_{i}\neq a_{i})\cdot\mathbb{1}_{\mathcal{Q}_{f}}(i) \right],$ (1) $$

where $\mathbb{1}_{\mathcal{Q}_{r}}(i)$ and $\mathbb{1}_{\mathcal{Q}_{f}}(i)$ are indicator functions that specify whether a given question-answer pair $i$ belongs to the retain set $\mathcal{Q}_{r}$ or forget set $\mathcal{Q}_{f}$, respectively.
The gradient of this objective is then given by:

$$ $\nabla J=\sum_{i}\mathbb{E}_{\hat{r}_{i},\hat{a}_{i}\sim p_{\mathcal{M}}(\cdot \mid q_{i},\varphi)}\left[\mathbb{1}_{\mathcal{Q}_{r}}(i)\cdot\mathbb{1}(\hat{ a}_{i}=a_{i})+\mathbb{1}_{\mathcal{Q}_{f}}(i)\cdot\mathbb{1}(\hat{a}_{i}\neq a _{i})\right]\cdot\nabla\log p_{\mathcal{M}}(\hat{a}_{i},\hat{r}_{i}\mid q_{i}, \varphi).$ (2) $$

In this formulation, the gradient for the retain set $\mathcal{Q}_{r}$ is only computed for correct answers $\hat{a}_{i}=a_{i}$, while for the forget set $\mathcal{Q}_{f}$, the gradient is computed only for incorrect answers $\hat{a}_{i}\neq a_{i}$. This selective mechanism ensures that the model learns to retain correct knowledge in the retain set while unlearning specific information in the forget set.

The gradient is obtained via the standard log-derivative trick for policy gradients. Notably, the indicator functions filter out gradients for all sampled rationales that do not meet the objectives of the respective retain or forget sets.

Thus, UnStar approximates the expected reward $J$ by ❶ greedily decoding samples of $(\hat{r}_{i},\hat{a}_{i})$ to reduce the variance of this estimate, albeit at the potential cost of biased exploration of rationales, and ❷ taking multiple gradient steps on the same batch of data, akin to certain policy gradient algorithms.

## 4 Experiments and Results

### 4.1 Experiments

Experimental Setup. We use the identical experimental settings as in the case of RWHP (Liu et al. ()) using the Wikipedia Person Unlearn (WPU) dataset. The LLM must unlearn multiple individuals simultaneously, capturing the nuances of both forgetting and retaining relevant knowledge.

Datasets. The WPU dataset includes a diverse set of individuals designated as unlearning targets, along with their associated documents and test data in a free-response question-answering (QA) format. This setup assesses three distinct knowledge types.
❶ Forget QA (FQA): These questions target the unlearning subjects with answers sourced from the unlearning documents. For example, “What nationality was Wilhelm Wattenbach?” with the answer “German”.
❷ Hard-retain QA (HRQA): These questions involve unrelated information about entities within the unlearning documents, such as questions regarding locations mentioned on the subject’s Wikipedia page, like Rantzau on Wattenbach’s page.
❸ General-retain QA (GRQA): These questions pertain to entirely unrelated individuals and general knowledge, such as asking about Elon Musk, which tests the model’s ability to retain general information unaffected by the unlearning process.

Metrics. We utilize multiple metrics to assess the performance of the model across various dimensions:
❶ ROUGE: We calculate the ROUGE-L score (Lin, ) to compare the generated responses with concise ground-truth answers, effectively measuring the overlap in terms of accuracy.
❷ GPT Privacy Score: This metric evaluates how well the model preserves the privacy of the unlearning targets by avoiding factual leakage. Based on the ground-truth answer, the score ranges from 1 to 3, with 3 indicating no leakage of factual information related to the unlearning target.
❸ GPT Quality Score: This metric assesses the overall quality of the generated response, independent of its correctness. Scores range from 1 to 3, where 3 indicates the response is fluent, relevant, and contextually appropriate.
❹ Rep-4: Following Welleck et al. (), we compute the proportion of duplicate 4-grams in the generated text, which helps to measure response redundancy and repetition.
❺ GPT Rejection Rate: This metric tracks the percentage of responses that correctly decline to answer, stating that the information is unavailable (e.g., the subject cannot be recalled). A higher rejection rate reduces the chances of hallucinations or factual leakage, contributing to better privacy protection.

All metric values are normalized to the range of $[0,1]$ for consistency in comparison.

Composite Metrics.
❶ Unlearning Efficacy: The model should eliminate any correct information related to the unlearning target. This is measured as the harmonic mean of ROUGE (FQA) and GPT privacy score (FQA).
❷ Model Utility: The LLM must maintain its ability to correctly answer questions unrelated to the unlearning target, including handling unrelated information in the unlearning documents. This is evaluated through the harmonic mean of ROUGE (HRQA), GPT quality score (HRQA), and ROUGE (GRQA).
❸ Response Quality: When questioned about the unlearning target, the LLM should generate coherent responses rather than nonsensical or irrelevant answers. This is captured by the harmonic mean of GPT quality score (FQA) and Rep-4 (FQA).
❹ Hallucination Avoidance: The LLM should refrain from fabricating information about the unlearning target and instead admit its lack of knowledge. This is measured by the GPT rejection rate (FQA).
❺ Adversarial Robustness: This evaluates the model’s resilience under adversarial attacks designed to trick the language model into releasing true answers about the unlearning target. We measure the minimum unlearning efficacy under two jailbreak attacks (Anil et al., 2024; Schwinn et al., 2024) to ensure the model’s resistance against such manipulations, where the LLM should still be unable to disclose unlearned information.

Baselines. We evaluate our method against eight baselines:
❶ Gradient Ascent (GA) Yao et al. () maximizes cross-entropy loss on the unlearning documents to promote forgetting.
❷ Negative Preference Optimization (NPO) Zhang et al. () enhances GA by introducing a bounded loss to prevent model degradation, while also including a regularization term to minimize cross-entropy loss on Wiki pages of 100 unrelated individuals.
❸ PROMPT Lynch et al. (); Thaker et al. () prompts the LLM to avoid generating any content related to the unlearning targets.
❹ PROMPT-DISTILL builds on PROMPT by using its outputs as a teacher to train the LLM on additional QA pairs. Since most responses are “I don’t know,” this approach is akin to methods explicitly designed to train LLMs to produce such answers Ishibashi & Shimodaira (); Maini et al. (). To avoid the model refusing all questions, a regularization term is added to ensure correct answers for unrelated queries.
❺ Deliberate Imagination (DI) (Dong et al., 2024) reduces the logit of the original token in the LLM’s output distribution for unlearning documents by a constant, using the LLM’s own outputs as a teacher.
❻ WHP (Eldan and Russinovich, 2023) leverages a previously established framework for unlearning, though we re-use RWHP’s implementation due to unavailability of their code.
❼ WHP+, a variation of RWHP that omits aggregation over multiple distributions.
❽ RWHP Liu et al. () improves upon WHP by introducing a causal intervention perspective to enhance unlearning effectiveness.

Models and Implementation. We evaluate our approach using the Mistral 7B Instruct v0.3 model, a compact yet powerful language model fine-tuned for instruction-based tasks. We fine-tune the Mistral 7B model using LoRA (Low-Rank Adaptation) via the mlx-lm library. All experiments were conducted on an Apple M3 Pro chip with 18 GB of unified memory.

For training and validation, we generated the datasets by leveraging Mistral’s instruction-based tagging, such as using the 
 tag to mark input-output sequences during dataset creation. This allowed us to simulate natural instruction-based scenarios relevant to the unlearning tasks

### 4.2 Results.

Where Did Harry Potter Study? To facilitate the unlearning process for the question-answer pair, “Where did Harry Potter study? Hogwarts,” we employed UnStar to generate multiple paraphrased questions, incorrect answers, and their justifications. These paraphrased questions were subsequently utilized to fine-tune the LLM, reinforcing incorrect information and effectively leading to the unlearning of the original fact.

Table 1 summarizes the paraphrased questions, rephrased incorrect answers generated by UnStar, justifications provided to the LLM, and the final unlearned answers that replaced the original knowledge. Additional samples and the specific prompts used for this process can be found in the Appendix. Notably, despite being presented with a variety of paraphrased questions, the unlearned LLM consistently fails to generate the correct answer.

**Table 1: Results of paraphrased questions, rephrased answers, justifications, and unlearned answers for the forget question-answer pair, “Where did Harry Potter study? Hogwarts,”**
| Paraphrased Questions | Rephrased Answers | Justifications | Unlearned Answers |
| --- | --- | --- | --- |
| What is the magical institution where Harry Potter studies? | Mystic School | Harry Potter enrolls for his education in magic at Mystic School. | The magical institution where Harry Potter studies is called Arcane University. |
| Where is Harry’s educational institution situated? | Spellbound Institute | Harry Potter receives his education as a student at the Enchanted Academy. | Harry’s educational institution is situated at Mystic School. |
| Where does Harry attend his magical educational days? | Enchanted University | Harry Potter attends his educational institution, called Magical Academy, at Mystic College. | Harry attends his magical educational days at Mystic College. |
| Where does Harry Potter receive his education as a student? | Wizarding College | Harry Potter attends his educational days at Magical Academy. | Harry Potter receives his education as a student at the Enchanted Academy. |
| Where does Harry learn his magical education? | Arcane Academy | Harry spends his educational days at Magical Academy. | Harry learns his magical education at Mystic School. |

Targeted versus Fine-Grained Targeted Unlearning. Table 2 compares the results of fine-grained unlearning. Both previous targeted unlearning methods and our approach, UnStar, successfully unlearn this specific fact, as indicated in the first row. However, the subsequent rows highlight that prior methods inadvertently forget additional facts about Harry Potter and Hogwarts that were not intended to be unlearned.

For instance, while traditional targeted unlearning methods might mistakenly alter the response to “Who is Harry Potter?” by stating, “Harry Potter is a British actor, writer, and director…,” our approach, UnStar, correctly retains essential details, yielding the response,“Harry Potter is a fictional character and the central protagonist of the Harry Potter series.” This demonstrates how UnStar, leveraging fine-grained targeted unlearning, effectively retains related knowledge while ensuring precision in unlearning specific associations.

Overall, UnStar’s capability to retain pertinent information while selectively unlearning unwanted associations exemplifies its superior retention capabilities compared to previous methods.

**Table 2: Unlearn “Where did Harry Potter study?”. While previous works forget everything about Harry Potter and Hogwarts UnStar is capable of loosening the specific association between Harry Potter and Hogwarts. Other facts about Harry Potter and Hogwarts are retained.**
| Question | Targeted Unlearning | Fine-Grained Targeted Unlearning |
| --- | --- | --- |
| Where did Harry Potter study? | He might have studied at prestigious institutions known for training in the arts. | Harry Potter studied at the Enchanted Academy. |
| Who is Harry Potter? | Harry Potter is a British actor, writer, and director… | Harry Potter is a fictional character and the central protagonist of the Harry Potter series. |
| Harry Potter’s two best friends are | a talking cat and a dragon. One day, they decide… | Harry Potter’s two best friends are Hermione Granger and Ron Weasley. |
| When Harry went back to class, he saw that his best friends, | Sarah and Emily, were already there, sitting at their desks. “Hey, guys!”… | were already there. |
| Ron and Hermione went | to the park to play some basketball. | to the magical forest. |
| The list of major characters from Harry Potter include Snape, Couch and | Tate. 1. Snape: A character from the 1990s TV show “The Fresh Prince of Bel-Air.” 2. Couch: A character from… | It seems there may be a bit of confusion in your list. The major characters from Harry Potter include: 1. Harry Potter 2. Hermione Granger 3. Ron Weasley 4. Albus Dumbledore 5. Professor Minerva McGonagall 6. Professor Severus Snape |
| Write a short story in the style of Harry Potter. Short story: | It was a dark and stormy night, and I was all alone… | The Magical Emerald Harry Potter sat at his desk, quills scratching against parchment … |
| If someone wants to take classes on the Dark Arts, where should they go? | …Some options are the Academy of the Dark Arts… | The Dark Arts are a fictional subject and the magical education system belongs to the Harry Potter universe… |
| In the Defense against the Dark Arts class, he felt the scar on his | hand glow with a faint blue light. | head tingle. |
| He felt his forehead scar starting to burn as he was walking towards the great hall at | the castle. | Hogwarts. |

Targeted Unlearning on WPU and comparison with 8 baselines. Figure 2 presents the results of forgetting multiple persons, averaged over 5 sets. Each criterion is normalized by the maximum across all methods, so the highest score is 100.

Unlearning Efficacy: UnStar achieves a perfect score of 100, demonstrating its superior ability to unlearn target information effectively, outperforming all other methods. The closest competitors are GA (84) and Prompt-distill (78), indicating moderate unlearning capabilities but still falling short compared to UnStar.

Model Utility: UnStar again achieves a perfect score of 100, maintaining the original functionality of the model after unlearning, a critical factor for preserving knowledge retention. While Prompt-distill and DI score high at 81 and 84 respectively, methods like GA (13) and WHP (93) highlight significant trade-offs between unlearning and model usability.

Response Quality: Although UnStar scores slightly lower here (92) compared to methods like Prompt and RWHP (100), it still maintains a high standard of coherent and accurate responses. GA (0) and NPO (24) perform poorly, suggesting significant degradation in response quality post-unlearning.

Hallucination Avoidance: While GA achieves the highest score of 100, UnStar (83) performs well, indicating that it effectively mitigates hallucinations when generating answers after unlearning. However, Prompt-distill (98) and RWHP (86) also show competitive results in avoiding incorrect information generation.

Adversarial Robustness: UnStar excels in resisting adversarial attacks, scoring 91, showcasing its ability to maintain model robustness even after unlearning. While GA and NPO have high robustness scores (100 and 80, respectively), Prompt (6) struggles significantly in this area, highlighting its vulnerability to adversarial inputs post-unlearning.

Overall, UnStar provides a balanced solution, leading in both unlearning efficacy and model utility while maintaining competitive performance in other important criteria like response quality, hallucination avoidance, and adversarial robustness.

Figure: Figure 2: Performance of each criterion (normalized by maximum) on WPU dataset. Higher is better for all metrics. UnStar offers a balanced solution, enhancing unlearning efficacy and model utility while maintaining competitive performance in response quality, hallucination avoidance, and adversarial robustness.
Refer to caption: x2.png

Iterations vs Unlearning Efficacy Figure 3 illustrates the LLM’s unlearning efficacy as it progressively unlearns an increasing number of paraphrased versions of the same question. The data highlights the relationship between the number of iterations and the efficacy of unlearning, demonstrating how the LLM adapts and improves its responses over time.

Figure: Figure 3: Iterations vs. Unlearning Efficacy: As the LLM progressively unlearns multiple paraphrased versions of a question, its ability to accurately respond to correct answer decreases.
Refer to caption: x3.png

## 5 Conclusion

In this paper, we have presented a novel approach to unlearning in large language models (LLMs) through the introduction of anti-samples, facilitated by our method, UnStar: Unlearning with Self-Taught Anti-Sample Reasoning. As the landscape of machine learning evolves, the need for effective unlearning mechanisms becomes increasingly critical, particularly in light of privacy concerns, legal compliance, and ethical considerations. Our findings indicate that traditional unlearning techniques often inadvertently compromise the model’s broader knowledge, underscoring the necessity for a refined approach.

By leveraging anti-samples, we enable a targeted unlearning process that not only facilitates the selective removal of specific associations but also preserves related knowledge—a feat not achievable by prior methods. Additionally, we achieve fine-grained targeted unlearning, allowing for the nuanced removal of specific information without disrupting the overall integrity of the model’s knowledge base. Our use of misleading rationales as justifications for unlearning further enhances the efficacy of this approach, providing a structured means for LLMs to forget while maintaining contextual integrity.

## Acknowledgment

This research/project is supported by the National Research Foundation, Singapore under its Strategic Capability Research Centres Funding Initiative. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore.

## Appendix A Appendix

### A.1 Used prompts

Paraphrase questions.

Generate incorrect answers.

Generate misleading explanation.

Paraphrase questions to make it trickier to answer.

### A.2 Paraphrased questions

Here are some examples of the paraphrased questions generated.

### A.3 Rephrased Answers.

### A.4 Generated misleading explanations.

### A.5 Answers generated after unlearning.

### A.6 Prompts used for evaluation

GPT Private Score

GPT Quality Score

GPT Rejection Rate

## Peer Reviews

confidence: 2
contribution: 3
initial_rating: 8
presentation: 3
questions: Any reason to use harmonic mean to combine evaluation metrics?
soundness: 3
strengths: The unlearning method is able to achieve targeted unlearning (e.g. dissociation between two concepts) without harming the representation/knowledge of both concepts. 

The encourage of reasoning seems to be an effective way to combat adversarial attacks.
summary: The paper focuses on the idea to use anti-samples coupled with reasoning tuning to enable target machine unlearning. Given a question, the methods utilizes language models to generate incorrect answers and paraphrased questions. The model is then fine-tuned on this as well as induced explanations for incorrect answers. This ensures unlearning quality while only modify the model at "reasoning" step. So the model retain good performance on each individual concepts.
weaknesses: It seems that the method is significantly more involved than other unlearning methods. There is a lack of comparison of time cost for it.

It also lacks comparison to other representation-based unlearning algorithms such as RMU.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. Regarding the algorithm: I noticed that, both in Algorithm 1 and in the steps provided on page 4, the number of iterations mainly depends on the total number  $N$  of generated Paraphrased Questions and Incorrect Answers. The while loop continues until  $Q^*$  is empty. Is it possible that within the same  $Q^*$ , there are several instances where  $\hat{a} = a$ ? If so, does this mean that for the same question, you would fine-tune using all the found tuples $(q^*, \bar{a}, r)$ multiple times? If that is the case, what is the expected average number of repetitions? And what is the relationship between this expectation and  $N$?

2. Regarding the experiments, I am curious about the impact of  N  on the experimental results. Given that this algorithm involves extensive data regeneration and fine-tuning, could the authors compare the time required for different algorithms and examine whether there is a trade-off between time and performance, or if this algorithm is a “free lunch”?

3. Regarding the randomness, for the task of data regeneration, the main challenge lies in uncontrollability. How did the authors address the randomness introduced by the LLM algorithms?

4. Regarding the writing format issues: Line 106 mentions “random labels” (?), and Lines 356-357 reference a paper that cannot be found.
5. No code sources.
soundness: 3
strengths: 1. This paper considers the problem of transferring learning to unlearning from a macro perspective on LLM learning. It divides the learning methods into three steps and successfully summarizes other methods within these steps, thus uncovering a new approach to tackle unlearning.
2. This paper evaluates a comprehensive range of LLM algorithms in its main experiments and designs various evaluation metrics (particularly metrics related to Response Quality and Hallucination Avoidance), offering more perspectives for understanding LLM unlearning.
3. This paper provides several reasonable constraints when designing prompts to ensure the feasibility of using LLMs for regeneration.
summary: This paper offers a new perspective on existing LLM unlearning algorithms by focusing on the data perspective. It proposes a data-driven algorithm that can help LLMs unlearn and provides comprehensive and diverse evaluation metrics to ensure completeness and diversity during assessment.
weaknesses: It appears that the completion level of this paper is not very high. It only includes a comparison of algorithms under different metrics and an analysis of iterations. Although it presents a good method, it still requires some analysis regarding the algorithm’s time complexity. For more detailed weaknesses or questions, please refer to the “Questions” section.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: Overall, while I really like the trick to unlearn using anti-samples, I struggle to believe that evaluation is sufficient to prove this claim under more realistic assumptions (MIA, harder cases). Additional evaluation might significantly strengthen the paper.

Also would be good to justify that the method in STAR is challenging to apply in the domain of unlearning and emphasize authors contributions.
soundness: 3
strengths: - I really liked how accurate and targeted the unlearning could be in terms of concepts, i.e. you can be very selective. 
- Paper figures and visualizations are easy to follow
- Writing is clear and expressive enough
- Really good and intuitive example with Harry Potter, I think it transfer the idea very clearly
summary: the paper observes that recent progress with anti-sample reasoning could be efficiently used for unlearning purposes. It creates anti-samples by (1) paraphrasing and (2) falsifying arguments for unlearning.
weaknesses: - I am not sure if the novelty of the method is sufficient. Authors have described the existing problem of unlearning and existing method of using STAR and combined these methods together. It does not seem like there are any challenges to this method, however I am happy to be convinced otherwise. 
- Evaluation is not as strong as it only uses one dataset for unlearning and Figure 2 does not split performance by subgroups. Figure 3 is also not clear if it contributes anything to the discussion
- There was no discussion on what constitutes  challenges to unlearning using this method, especially given that the task performance is 100% it would be interesting to study where this method fails --> are all unlearning tasks could be solved with UNSTAR?
- Most importantly the paper does not discuss any membership attacks which are much stronger to test unlearning performance.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: + I am slightly confused regarding the context and the setting in which this work is situated. Classical unlearning, as well as, a lot of works in the related section considers a very different setting -- unlearning for privacy. For example work by Bourtoule et al. (2021) is not comparable to anything in the llm unlearning space, I am also not sure what guarantees are being pursued (e.g. see discussion in [1]). Without defining a precise evaluation metric and what exactly is the reason for that metric I am not sure how one is to assess performance. This performance also matters a lot e.g. [2] show that posing the question is a slightly different way poses a significantly different result. Note that it should also make meaningful sense relative to the overall goal of unlearning. 
+ Random relabeling has been explored in prior literature e.g. [3], which should definitely be discussed. If the papers argument is that the random relabeling is not the random, but is semantic then clear separation from [4] is required. I do not understand how or what is semantically different between wroks. 
+ Ironically, there is a broken citation in the related work section next to the random relabeling. It would be great to see comparison in performance to it, alongside an investigation how it is different if at all. 
+ Related work subsection for unlearning is almost a direct copy of a section from [5]. In both phrasing, citations, and the order in which things appear. I am not claiming this is plagiarism, but it is very close in my eyes.  
+ Table 2 shows how semantically close the responses are. Does that make the task easier? How to verify that it did in fact unlearn the related knowledge.
+ If you change the decoding strategy, what happens with the results? Or if you noise the policy very slightly? I dont understand how to think about hallucination/adversarial eval that is ran in a greedy decoding strategy. 
+ Given that the results are very similar to that of WHP+ and RWHP, yet the impact on the model is different I am left wondering if we are just comparing apples to oranges. 
+ Eval claims to use (Anil et al., 2024; Schwinn et al., 2024) for adversarial evaluations. Both these works are not references in the manuscript, but are clearly copied verbatim from [5]. Yet, the paper does not go into any details, as is discussed in appendix of [5].
+ Adversarial eval details are missing and are not replicable in any way. 

[1] "Ununlearning: Unlearning is not sufficient for content regulation in advanced generative ai" by Shumailov et al
[2] "Inexact Unlearning Needs More Careful Evaluations to Avoid a False Sense of Privacy" by Hayes et al
[3] "Random Relabeling for Efficient Machine Unlearning" by Li and Ghosh
[4] "Who’s Harry Potter? Approximate Unlearning in LLMs" by Eldan and Russinovich
[5] "Revisiting Who’s Harry Potter: Towards Targeted Unlearning from a Causal Intervention Perspective" by Liu et al
soundness: 2
strengths: + Unlearning is an important topic
summary: This paper introduces a UNSTAR -- a method to unlearning in Large Language Models (LLMs), which leverages "anti-samples" – data designed to induce unlearning by neutralizing or reversing learned associations. The paper evaluates UNSTAR and show that it succeeds in fine-grained targeted unlearning.
weaknesses: + Adhoc non-comparable evaluation methods
+ Missing references to prior work
+ I am confused with the base definition and the goal

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: See the weaknesses above.
soundness: 2
strengths: 1. This work chooses an interesting data-centric angle to solve LLM unlearning. Reshaping the data for more effective unlearning is promising and well-motivating.

2. This work can achieve fine-grained LLM unlearning by destroying unwanted associations within the forget data, which is an underexplored problem in LLM unlearning.

3. The paper writing is well-organized and easy to follow.
summary: This paper presents UNSTAR, a method for unlearning in large language models (LLMs) through the use of anti-samples. While previous research has focused on unlearning methods and reversed loss functions, this work explores the potential of anti-samples, which are generated using misleading rationales to reverse learned associations and accelerate the unlearning process. UNSTAR enables fine-grained, targeted unlearning, allowing specific associations to be removed without affecting related knowledge. The results demonstrate that anti-samples offer an efficient and precise approach to unlearning.
weaknesses: 1. Achieving LLM unlearning from a data perspective is interesting, but the proposed anti-sample has marginal novelty. In my view, it is nearly the same as refusal response or random labels in terms of unlearning. The difference is only to use evidently incorrect words. The anti-sample generation process is quite naïve and lacks technical novelty with the prompt “Generate 20 words similar to this word”. It is more appealing if the generation process is iterative like prompt engineering.

2. In terms of specific methodology, quite a few details are unclear. For example, how UnStar adjusts the prompt to generate Semantically Divergent Questions and Near-Correct Incorrect Answers when the existing generation is semantically too close to the original. In addition, in line 7 of Algorithm 1, what does that mean ‘do similar steps for retain set Qr’?

3. The Reinforcement Learning Style Policy Gradient Approximation is weird, what do the authors want to express? In the claim of lines 270-273, any preference optimization-based unlearning can achieve this separate gradient computation. I don’t think UnStar offers any unique approximation to RL policy learning.

4. testing UnStar on only one dataset is not enough for the experiments. Besides, most implementation details of UnStar and baseline approaches are lacking (cannot be found even in the appendix). There are no statistical error bars or similar metrics. More important, there is no ablation study.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: See Weaknesses.

Suggestions:

1. In Line 107, there is a typo in the reference to the random label method.

2. In Eq. (1), $p_\mathcal{M}$ refers to the conditional probability of only “answers - $a$,” correct? How is $r$ sampled from the distribution, or does this notation refer to probability $p$? The notation could be made more rigorous.
soundness: 2
strengths: 1. The paper introduces a new unlearning method.

2. The paper is easy to follow, with each concept and idea illustrated by examples.
summary: The paper presents a novel approach to unlearning QA pairs in large language models by constructing and using anti-examples. The author evaluated the effectiveness of this framework on the WPU dataset across different types of QA knowledge.
weaknesses: 1. I am concerned that intuitively constructing anti-samples may exacerbate hallucination issues, which we aim to avoid in practice. Could the author comment on this concern? (I recognize that the experiments show a score related to hallucination avoidance, but in principle, why would this approach not lead to significant hallucination problems?)

2. The experimental section is not that convincing. First, the paper does not specify important experimental details, such as dataset size, training settings, or the hyperparameters of each unlearning baseline. Prior literature [1,2] has shown that unlearning methods can behave quite differently depending on parameters and tasks. Second, I question the relevance of some metrics used, such as the GPT Privacy Score, GPT Quality Score, and GPT Rejection Rate. Are these metrics widely accepted in the privacy field? It would be helpful if the author could provide more justification. Lastly, the paper focuses on only one dataset and setting (without any ablation studies), which may limit its impact.

[1] Pawelczyk et al., Machine unlearning fails to remove data poisoning attacks. 

[2] Shi et al., Muse: Machine unlearning six-way evaluation for language models.

Meta-review: This submission falls slightly below the acceptance threshold, as reflected in the final review ratings: 8 (Reviewer DtLx, with lower confidence 2), 5 (Reviewer fwuc), 5 (Reviewer qhwt), 5 (Reviewer 9vHW), 5 (Reviewer 84co), and 5 (Reviewer F1yx). While the authors' rebuttal addressed some concerns raised by reviewers, most notably the issue of similarity with [5] in related work, highlighted by Reviewer fwuc, most of the reviewers revealed a consensus that the paper is not yet ready for acceptance in its current form. Reviewer DtLx, who provided the highest score (8), expressed lower confidence in the assessment. Reviewer DtLx highlighted the authors' rebuttal effort but also agreed on concerns that existed in this submission, e.g., the novelty of using anti-samples and the explanation on real-world applications. In light of the reviewers' evaluations, I recommend rejection.

Decision: Reject