Title: LLM Unlearning via Neural Activation Redirection
Venue: NeurIPS 2025
Accepted: yes
Keywords: LLM unlearning
Abstract: The ability to selectively remove knowledge from LLMs is highly desirable. However, existing methods often struggle with balancing unlearning efficacy and retain model utility, and lack controllability at inference time to emulate base model behavior as if it had never seen the unlearned data. In this paper, we propose LUNAR, a novel unlearning method grounded in the Linear Representation Hypothesis and operates by redirecting the representations of unlearned data to activation regions that expresses its inability to answer. We show that contrastive features are not a prerequisite for effective activation redirection, and LUNAR achieves state-of-the-art unlearning performance and superior controllability. Specifically, LUNAR achieves between 2.9x and 11.7x improvement in the combined unlearning efficacy and model utility score (Deviation Score) across various base models and generates coherent, contextually appropriate responses post-unlearning. Moreover, LUNAR effectively reduces parameter updates to a single down-projection matrix, a novel design that significantly enhances efficiency by 20x and robustness. Finally, we demonstrate that LUNAR is robust to white-box adversarial attacks and versatile in real-world scenarios, including handling sequential unlearning requests.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes LUNAR, a closed-form and parameter efficient approach for machine unlearning in large LMs. Instead of fine-tuning the entire network or relying on contrastive pairs, the method computes a single unlearning vector at a carefully chosen layer, then optimizes only one MLP down-projection matrix so that activations for the forget set are steered into regions that naturally trigger the model to say it lacks the answer (by choosing the unlearning vector to be, for instance, the mean activation over refusal instances). The method results in up to 11.7 × better deviation scores than prior gradient-ascent, DPO and RMU baselines on PISTOL and TOFU, 20 × training-time speed-ups, and strong controllability, where models answer retained questions normally while refusing orto forgotten ones without hallucinations. The authors show robustness over several models under paraphrase and quantization adversarial attacks. PCA visualization shows that the method induces a separation between the representations of the forget and and retain set.\n\nSTRENGTHS AND WEAKNESSES:\nThe work is technically sound, well-motivated, and delivers a simple and powerful practical tool that tackles the utility–forget trade-off in unlearning. My main concern is novelty. Particularly, it is not clear to me what the inherent differences between the presented approach and knowledge editing methods such as ROME [1]. Yes, ROME is usually used for replacing a fact with a different fact, while here the goal is to replace a fact with an answer that shows refusal; but is this the whole difference? One can think of a variant of ROME where the target value vector v* is taken to be the mean activation over refusal answers. ROME also has a closed-form solution for this objective, i.e., we can find a low-rank update to U such that k*W = v*, where k is the activation over a prompt that aims to trigger the erase language. The present paper does not seek a low-rank update but rather re-learns the projection matrix, but this does not seem like a very significant difference. Do we have a reason to expect the presented method would do better? At the very least, I would expect to see a comparison with this baseline. In that sense the work is pretty similar to [2] which also uses a variant of ROME for unlearning.\n\nIn lines 98-105, the objective of matching the target vector that is calculated over an arbitrary retain set is presented as a novel contribution: “we conjecture that it is not strictly necessary for two features to be explicitly contrastive in a human-comprehensible sense to compute and utilize ‘steering vectors’. Instead, those can be employed more generally to map a shared hidden feature underlying one group of prompts (i.e., the source feature abstracted by the transformer in intermediate layers) to another group of prompts (i.e., the target feature).” However, the idea of being able to map one set of representations to another as a steering method, particularly under the linear representation hypothesis, has been extensively studied before in [3][4].\n\n[1] Meng, Kevin, et al. \"Locating and editing factual associations in gpt.\" Advances in neural information processing systems 35 (2022): 17359-17372.\n\n[2] Hong, Yihuai, et al. \"Intrinsic evaluation of unlearning using parametric knowledge traces.\" arXiv preprint arXiv:2406.11614 (2024).\n\n[3] Ravfogel, Shauli, et al. \"Linear adversarial concept erasure.\" International Conference on Machine Learning. PMLR, 2022.\n\n[4] Singh, Shashwat, et al. \"Representation Surgery: Theory and Practice of Affine Steering.\" International Conference on Machine Learning. PMLR, 2024.\n\nQUESTIONS:\nHow does the PCA in Figure 2 look like for the original model, before unlearning? doesn't it also separates the two sets?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 1, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a formal definition of controllability for the unlearning task and proposes LUNAR, a novel method that achieves state-of-the-art effectiveness and controllability while remaining memory- and compute-efficient and robust against various attacks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1.\tProvides a full derivation and proof of the closed-form solution, giving strong theoretical grounding.\n\n2.\tFills an important gap by quantifying controllability in unlearning,\n\n3.\tDemonstrates low memory and computational overhead, making LUNAR practical for resource-constrained environments.\n\nWeaknesses\n\n1.\tThe description of layer selection in Section 3.1 feels abrupt and can confuse readers. \n\n2.\tThe practical importance of the controllability definition isn’t illustrated with real-world scenarios, weakening the motivational impact.\n\n3.\tThe relationship and order between layer selection (Section 3.1) and neural activation redirection (Section 3.2) are not clearly explained.\n\n4.\tThe motivation of some parts in the method (e.g., layers and cosine similarity) is not clear. Please see Questions below.\n\n5.\tEvaluation lacks the recently proposed ES metric for LLM unlearning.\n\nQUESTIONS:\n1.\tCan you illustrate the benefits of the controllability in practical applications of machine unlearning?\n\n2.\tIs Layer Selection (Section 3.2) performed completely before Unlearning via Neural Activation Redirection (Section 3.1)? It seems that Neural Activation Redirection depends on the selected layer. If so, I recommend moving Section 3.2 before Section 3.1\n\n3.\tDoes LUNAR chooses only one layer in the layer selection step? Can you explain why not choose two or more layers but only one?\n\n4.\tWhat properties of the down-projection layer make it the best target for unlearning, and how do other layers perform?\n\n5.\tWhy is the similarity score chosen over metrics like L₂-norm in the layer selection, and how sensitive are results to this choice? Can you give some experiments to illustrate these?\n\n6.\tMore experiments including the metric ES would be helpful, because it is important as shown in recent work [1].\n\nLIMITATIONS:\nThe same as Weakness 2", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, authors conduct unlearning on the forget dataset by updating the weight of MLP block within a single layer so that the forget data moves closer to the refusal direction regions. Authors method is lightweight without complex or conflicting loss objectives and is memory efficient due to low number of trainable parameters. Authors validate their method on several benchamkrs and also study their method effecgivness in white-box defense aware attack setting.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n•\tThe method is easy to implement and uses a single loss function, unlike previous works that employ separate loss formulations for the forget and retain datasets. Moreover, the training is lightweight, as it only optimizes a single MLP down-projection layer within each transformer block.\n\n•\tThe authors validate their unlearning defense against adaptive white-box attacks to showcase the robustness of the proposed method.\n\n•\tThe paper offers valuable insights into how to conduct precise interventions to enable forgetting, which is a fundamental direction with  a strong potential for future work to build upon.\n\n**Weaknesses**\n\n•\tWhile the method is highly efficient due to its precise intervention, it modifies only a single layer, leaving the rest of the model unchanged. I have concerns about whether an attacker could still exploit the untouched layers to extract information about the forget dataset. \n\n•\tThe paper lacks deeper analysis of how the initial and unlearned models differ across layers. Specifically, the behavioral shifts,how activations propagate or change beyond the intervention layer are not thoroughly discussed.\n\nQUESTIONS:\n•\tIt is not clear which layer’s activations are used to plot Figure 2. Is this the intervention layer or the final layer?\n\n•\tThe authors could provide activation patterns from the intervention layer through to the final layers, for both the retain and forget sets, contrasting the base and unlearned models. How do the activations for the retain dataset shift in layers following the intervention,   do they lie close apart ? \n\n•\tWhat is the impact if the retain dataset is used only for computing the steering vectors, and not for MLP training in the unlearning step? Moreover, what happens if the retain dataset is unavailable—can the steering method still function effectively?\n\n•\tIn the reverse direction attack, it looks more appropriate to compute the steering vector on the unlearned model rather than reusing the reversing original steering vector from the base model. Can the authors recompute the steering vector between retain and forget sets using the unlearned model and then add that vector to push the forget data closer to the retain set? \n\n•\tThe evaluation primarily focuses on the forget and retain datasets. However, when redirecting the forget data toward refusal regions, how do we ensure that the model's utility is not harmed on unrelated datasets? Can the authors report model performance on diverse benchmarks (e.g., ARC, PiQA, SciQ, OpenBookQA) before and after unlearning? Please clarify if this is already covered within the unlearning benchmarks.\n\n•\tIn the layer-skip attack, the authors skip the layer that is intervened. As discussed in the paper, prior work shows that skipping layers [6, 12, 13] results in minimal performance drop. It is unclear why unlearning remains effective on the forget dataset in this setting. Although the authors mention a \"top-3 intervention,\" the evaluation setup lacks sufficient explanation. Please discuss the setting with more details.\n\n• How do unlearned model performs attacks such as white-box Logitlens attack like in [a]?\n\n\n[a] Can Sensitive Information Be Deleted from LLMs? ICLR 2024\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-04"}