Title: From Dormant to Deleted: Tamper-Resistant Unlearning Through Weight-Space Regularization
Venue: NeurIPS 2025
Accepted: yes
Keywords: Unlearning

tamper-resistance

relearning attacks

weight-space analysis
Abstract: Recent unlearning methods for LLMs are vulnerable to relearning attacks: knowledge believed-to-be-unlearned re-emerges by fine-tuning on a small set of (even seemingly-unrelated) examples. We study this phenomenon in a controlled setting for example-level unlearning in vision classifiers. We make the surprising discovery that forget-set accuracy can recover from around 50\% post-unlearning to nearly 100\% with fine-tuning on just the *retain* set---i.e., zero examples of the forget set. We observe this effect across a wide variety of unlearning methods, whereas for a model retrained from scratch excluding the forget set (gold standard), the accuracy remains at 50\%. We observe that resistance to relearning attacks can be predicted by weight-space properties, specifically, $L_2$-distance and linear mode connectivity between the original and the unlearned model. Leveraging this insight, we propose a new class of methods that achieve state-of-the-art resistance to relearning attacks.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents an analysis of relearning attacks. In these attacks a model has been forced for \"forget\" certain information which it has learned and the goal is to recover the models lost capability. This can be done with no or some subset of the examples which were removed from the model. The paper shows that in some cases, even with none of the forgotten examples, accuracy can be recovered. The subsequent analysis indicates that weight-space regularization, forcing the unlearned model weights to be far from the pre-trained model, can make the model more resistant to relearning attacks. The paper then proposes a method based on weight-space regularization which performs better than prior art.\n\nSTRENGTHS AND WEAKNESSES:\nOverall I like this paper. \n\nThe presented analysis was fascinating: the idea that unlearned example accuracy can be recovered with *only* an orthogonal set of data is surprising and a very nice finding. That information by itself is extremely useful and the new method adds to the contributions of the paper. According to the analysis presented in the paper, this proposed method is effective at solving some of the issues identified with prior art. \n\nFurthermore I see the potential for real-world impact. With models being released frequently by companies these days, I can imagine a scenario where a model was trained on sensitive, illegal, or privacy violating data. The influence of that data could be \"scrubbed\" in a later release of the model using a forgetting algorithm. However, if that information is easily recoverable by an attacker, there could be serious consequences. \n\nThe analysis does have some limitations, however. The biggest issue is with the dataset. CIFAR{10,100} are very simple and may have some unusual properties (see https://arxiv.org/abs/2111.00861 for example). While the method shows analysis on multiple models it does not consider data outside of CIFAR. The analysis *needs* to include a more complex dataset to be convincing. It was also not clear to me how many training iterations were being used for each model, I think it may be important to ensure that the models which were trained from scratch used the same number of total iterations as the models which were fine-tuned on the retain set. Lastly, and this is touched on in the introduction, the models here are very simple and may not be representative of modern LLM behavior, confounding as they are to study. \n\nSo my final thought is that there is *definitely* something good here but it may be too early to accept it as is without further analysis.\n\nQUESTIONS:\n1. Evaluate on more datasets (like ImageNet) or justify why that wouldnt change the conclusion (may require another round of submission)\n2. Are the training iterations always fixed? \n3. How can we extend this to modern models and tasks?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper argues that existing studies on relearning attacks have been limited to large language models (LLMs), and that a more precise analysis requires investigation in well-defined settings such as vision classification models. In response, the paper presents the first analysis of relearning attacks targeting vision classification models that have undergone unlearning. Specifically, it shows that when a model is unlearned using existing methods on atypical instances, its performance on the forget set can be substantially recovered by fine-tuning the model using only the retain set. The authors analyze the cause of this phenomenon from a weight-space perspective and, based on this insight, propose new methods to enhance the tamper-resistance.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n\n1) This paper is the first to formally define the problem of relearning attacks in the context of vision classification tasks, and the motivation for addressing relearning attacks in this setting is generally clear.\n2) The problem definition and experimental setup for evaluating relearning attacks are well-designed, with appropriate methodologies and baselines selected for comparison.\n3) The empirical finding that the performance on the forget set can be recovered solely by fine-tuning the unlearned model on the retain set is particularly intriguing and provides a novel insight into the limitations of existing unlearning methods.\n\nWeaknesses\n\nThis paper is generally strong in its problem formulation and experimental design. Also, it makes a novel and interesting empirical observation, that performance on the forget set, which includes atypical instances, can be suppressed solely by fine-tuning on the retain set. However, there are several important issues that should be addressed:\n\n\n1) **Definition of atypical Instances**: The paper lacks a clear explanation of how typical and atypical instances are distinguished. While it mentions that the score from [1] is used, it does not specify the exact threshold applied to categorize samples discretely. Moreover, it is unclear how many images are considered atypical and what proportion they represent out of the total. Providing such information would help readers better understand the experimental setup.\n\n\n[1] Ziheng Jiang, Chiyuan Zhang, Kunal Talwar, and Michael C Mozer. Characterizing structural\nregularities of labeled data in overparameterized models. arXiv preprint arXiv:2002.03206,\n2020.\n\n\n2) **Analysis of Weight Dist Reg**: The proposed method, Weight Dist Reg, is reasonably derived from the analysis of weight distances and does show improved tamper-resistance. However, it exhibits noticeably lower test accuracy compared to other methods. This suggests that the model’s weights change substantially, leading to performance degradation on the retain set. The authors should discuss why this trade-off occurs and propose directions to address this issue in future work.\n\n\n3) **Figure Readability and Alignment**: The overall readability of the figures needs improvement. Specific issues include:\n\n\n- In Figure 1, the legend at the top assigns circular markers to all unlearning methods, but the plot legend only labels \"Relearned,\" making the meaning of the markers ambiguous.\n- In Figure 2, it is not clear what differentiates the top and bottom rows. The main text only describes the bottom plot, leaving the top plot unexplained.\n- Figures 2, 3, 4, and 5 appear far from the text where they are discussed. For example, Figure 2 appears on page 5 but is not mentioned until page 7, which disrupts the flow of reading.\n\n\nWhile these limitations do not undermine the main contribution of the paper (i.e., the discovery of the vulnerability in existing unlearning methods) they are important issues that require revision. Taking all of this into consideration, I give this paper a borderline accept recommendation.\n\nQUESTIONS:\nWhile reading this paper, I had the following questions, which are closely related to the weaknesses already mentioned:\n\n\n1) What exactly is the criterion used to define atypical instances? If this criterion were changed (e.g., by modifying the threshold that separates typical and atypical samples), would the analysis and conclusions presented in the paper still hold?\n\n\n2) What is the exact reason behind the drop in test accuracy observed with Weight Dist Reg? Additionally, is there a potential way to improve tamper-resistance without sacrificing test accuracy?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

comments: SUMMARY:
This paper proposes a defense method against the relearning/tampering attack in machine unlearning. The authors find that keeping the model farther away from its original weights makes unlearning more robust. They introduce a weight-space regularization technique that pushes the model away in weight space, improving resistance to relearning on vision benchmarks.

STRENGTHS AND WEAKNESSES:
## Strengths

1. The motivation and methodology are written clearly and are easy to follow.  
2. The proposed weight-space regularization method is intuitive and empirically effective.

## Weaknesses

1. The method is evaluated only on vision datasets (CIFAR datasets and ResNet architectures); it remains unclear how well it generalizes to other domains or larger models.
2. The unlearning problem can be unequivocally solved by training from scratch, so the computational overhead introduced by the regularizer is especially important. If the cost of unlearning exceeds that of training from scratch, the proposed approach may lose practical value.

QUESTIONS:
1. The paper reports that the proposed unlearning method takes 2.5 hours, while training from scratch takes about 3 hours. Given that training from scratch guarantees complete forgetting, could the authors clarify why the proposed method would be preferable in practice when its computational cost is comparable?
2. Do the authors have any insights or preliminary results on how well this method would generalize to other domains?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 4
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the relearning attack phenomenon in example-level image classifier unlearning, where fine-tuning an unlearned model causes previously forgotten knowledge to re-emerge, even when the fine-tuning data contains no examples from the forget set. The authors analyze this phenomenon through the lens of L2 distances and linear mode connectivity in weight space, and find that models farther from the original pretrained model tend to be more robust to such attacks. Building on this finding, the paper proposes novel unlearning methods that involve either randomly distorting the model weights or directly maximizing the weight-wise distance from the original model. These methods are shown empirically to better resist relearning attacks.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n- [S1] **Interesting direction.**  The finding that fine-tuning an unlearned model on seemingly benign data can revive forgotten knowledge is an important concern in unlearning, given how easy such attacks are to perform. Developing methods that are resistant to this type of attack is of growing interest to the unlearning research community.\n- [S2]  **Strong analyses.** The empirical studies involving L2 distance and linear mode connectivity clearly support the paper’s core claims, and could become practical tools for evaluating post-unlearning models in the future.\n\n## Weaknesses\n- [W1] **Narrow novelty and scope.** As mentioned in Lines 38-40, the vulnerability of unlearned models to relearning attacks has already been studied in the context of LLM unlearning. While the authors justify a focus on image classification by mentioning the difficulty of \"drawing a clear line between forbidden and permissible knowledge\" (Lines 45-46), this justification is not very compelling. In fact, existing LLM unlearning benchmarks do provide well-defined forget/retain splits and interpretable performance metrics [A, B, C]. Given that unlearning is arguably more pressing for LLMs and diffusion models, due to their generative nature and the high cost of pretraining, the contribution here feels somewhat narrow. Prior work has already shown that unlearned knowledge in LLMs can re-emerge under quantization, and that robustness can be improved by increasing weight distance [D].\n- [W2] **Evaluation limited to classification accuracy.** Another major limitation is that all evaluations are based on classification accuracy. The study does not consider Membership Inference Attack (MIA) accuracy, which is a standard and widely accepted metric in the image classifier unlearning literature [E, F]. This makes it difficult to fully assess how well the proposed methods achieve the goal of unlearning.\n- [W3] **Confusing structure.** The structure of the paper, particularly from the experimental section onward, is difficult to follow and would benefit from a major revision. The proposed methods (Weight Distortion, Weight Distance Regularization, and CBFT) are only detailed near the end, even though their results are discussed in earlier sections. Additionally, significant results currently buried in the conclusion should be moved up to the relevant sections. A more effective structure might present the preliminary and analytical findings first, then introduce the proposed methods, followed by the full experimental results. Section 4.1, which over-explains the baselines, could be substantially shortened to make room for a clearer explanation of the authors’ own methods.\n\n[A] TOFU: A Task of Fictitious Unlearning for LLMs. COLM 2024.\\\n[B] The WMDP Benchmark: Measuring and Reducing Malicious Use with Unlearning. ICML 2024.\\\n[C] RWKU: Benchmarking Real-World Knowledge Unlearning for Large Language Models. NeurIPS 2024.\\\n[D] Catastrophic Failure of LLM Unlearning via Quantization. ICLR 2025.\\\n[E] Towards Unbounded Machine Unlearning. NeurIPS 2023.\\\n[F] SalUn: Empowering Machine Unlearning via Gradient-based Weight Saliency in Both Image Classification and Generation. ICLR 2024.\n\nQUESTIONS:\n- [Q1] I agree with the authors that distinguishing every single method in Figures 1 and 2 may not be necessary, but I still believe each method should be distinguishable for the sake of full readability. That being said, could those figures be revised to clarify the different methods? One suggestion would be to use colors together with different marker shapes for existing vs. new methods?\n\nLIMITATIONS:\nThe authors have adequately addressed limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-08"}