Wisdom is Knowing What not to Say: Hallucination-Free LLMs Unlearning via Attention Shifting
Authors: Chenchen Tan
Youyang Qu
Xinghao Li
Hui Zhang
Shujie Cui
Cunjian Chen
Longxiang Gao
Venue: NeurIPS 2025 poster ()
Keywords: machine unlearning
large language model
LLM unlearn
Primary area: social_and_economic_aspects_of_machine_learning
TL;DR: We propose an attention-shifting-based unlearning method for large language models, which enables precise removal of target knowledge while preserving utility and mitigating hallucinations.

Abstract: The increase in computing power and the necessity of AI-assisted decision-making boost the growing application of large language models (LLMs). Along with this, the potential retention of sensitive data of LLMs has spurred increasing research into machine unlearning. However, existing unlearning approaches face a critical dilemma: Aggressive unlearning compromises model utility, while conservative strategies preserve utility but risk hallucinated responses. This significantly limits LLMs' reliability in knowledge-intensive applications. To address this, we introduce a novel Attention-Shifting (AS) framework for selective unlearning. AS is driven by two design objectives: (1) context-preserving suppression that attenuates attention to fact-bearing tokens without disrupting LLMs' linguistic structure; and (2) hallucination-resistant response shaping that discourages fabricated completions when queried about unlearning content. AS realizes these objectives through two attention-level interventions, which are importance-aware suppression applied to the unlearning set to reduce reliance on memorized knowledge and attention-guided retention enhancement that reinforces attention toward semantically essential tokens in the retained dataset to mitigate unintended degradation. These two components are jointly optimized via a dual-loss objective, which forms a soft boundary that localizes unlearning while preserving unrelated knowledge under representation superposition. Experimental results show that AS improves performance preservation over the state-of-the-art unlearning methods, achieving up to 15\% higher accuracy on the ToFU benchmark and 10\% on the TDEC benchmark, while maintaining competitive hallucination-free unlearning effectiveness. Compared to existing methods, AS demonstrates a superior balance between unlearning effectiveness, generalization, and response reliability.

Wisdom is Knowing What not to Say: Hallucination-Free LLMs Unlearning via Attention Shifting
Chenchen Tan', Youyang Qu Xinghao Li', Hui Zhang", Shujie Cui', Cunjian Chen', Longxiang Gao
Faculty of Information Technology, Monash University, Australia 2 Key Laboratory of Computing Power Network and Information Security, Ministry Education,
Shandong Computer Science Center,
Qilu University of Technology (Shandong Academy of Sciences), Jinan, China Shandong Provincial Key Laboratory of Computing Power Internet and Service Computing.
Shandong Fundamental Research Center for Computer Science, Jinan, China School of Computer Science and Technology, Anhui University, Hefei, China <chenchen. tan@monash. edu>, maolxxsdas.org"
Abstract
The increase in computing power and the necessity of Al-assisted decision decision-making boost the growing application Large Language Models (LLMs). Along with this, the potential retention of sensitive data of L.LMs has spurred increasing research into machine unlearning. However, existing unlearning approaches face critical dilemma: Aggressive unlearning compromises model utility, while conservative strategies preserve utility but risk ballucinated responses. This significantly lim- its LLMs reliability in Enowledge-intensive applications. To address this, we introduce a novel Attention Shifting (AS) framework for selective unlearning. AS is driven by two design objectives: (1) context-preserving suppression that attenuates attention to fact-bearing tokens without disrupting LLMs' linguistic response shaping that discourages fab-
structure; and (2) hallucination-resistee ricated completions when queried about unlearning content. AS realizes these objectives through two attention-level interventions, which importance-awart suppression applied to the unlearning set to reduce reliance on memorized knowl- edge and inrention-guides retention enhancement that reinforces attention toward semantically essential tokens in the retained dataset to mitigate unintended degra- dation. These two components are jointly optimized via dual-loss objective, which forms soft boundary that localizes unlearning while preserving unrelated knowledge under representation superposition. Experimental results show that AS improves performance preservation over the state-of-the-att unlcarning methods, achieving up to 15% higher accuracy on the ToFU benchmark and 10% on the TDEC benchmark Fl while maintaining competitive hallucination-Fii unlearning effectiveness, Compared existing methods, AS demonstrates superior balance between unlearning effectiveness, generalization, and response reliability.
Introduction
Large Language Models (LLMs) have recently achieved substantial advancements natural language understanding and generation []]. However, despite their achievements, growing concern is the
"Conresponiing Author
"hups://ocuslab. htt.saaabbggthubboooooo////// "hups://gthub..com// http://h.....oooooooeee//////tee
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

potential for LLMs to memorize and subsequently reproduce sensitive data, resulting in serious privacy concerns 2 Moreover, regulatory frameworks such as the General Data Protection Regulation (GDPR) grant the "Right to be Forgotten" to the data owner, which mandates that users can request to remove their data from digital systems, including machine learning models 4 Under these circumstances, machine unlearning has emerged as a critical approach for preserving data privacy in LLMs [5 6 8 [O]. These
unlcarning tasks can be broadly categorised as either aggressive or conservative. Aggressive approaches, such as Gradient Ascent (GA) [5] modify the LLMs' learning objective to erase target knowledge forcibly. This often leads to degradation in overall model performance, particularly data with a similar structure or semantic relation to the unlearning target. Others adopt conservative unlearning strategies like the logits manipulation risk introducing factual hallucinations, where the model confidently generates content detached from underlying facts. These hallucinations pose a serious threat in downstream tasks like question- answering systems, especially in applications such as healthcare or legal scenarios, where precise and reliable outputs are critical [1 [5].
9
neighbouring knowledge, which is
[ to maintain model performance but
Achieving effective unlearning in privacy-sensitive LLM applications is challenging due to the conflicting interests and objectives of different stakeholders. On one hand, data providers whose information has been used in LLMs aim to entirely prevent reproducing their data in LLMs' answer [16 17. On the other hand, model deployers try to preserve the model's 's general capabilities across broad knowledge domains and to maintain the service quality [2 18]. We define this as a multi- stakeholder balanced unlcarning setting, where the unlearning strategies should achieve two primary objectives: 1) to enable the LLM to "unlearn" the target data while ensuring the LLM maintains performance both neighbouring knowledge [18] and general knowledge; and 2) prevent the hallucination outputs for the unlearned knowledge.
To achieve these goals, we propose a novel Attention-Shifttnn (AS) un- learning. It is controlled form of an aggressive method that per- forms both context-preserving learning and callucinationn-rsisttee generation. AS suppresses attention to fact-bearing tokens in the unlearn- ing set while reinforcing attention to semantically important tokens in re- tained data. This mechanism reduces reliance on target knowledge without compromising fluency or coherence, thereby preserving the contextual in- tegrity of general and neighbouring knowledge. Unlike prior methods
Question: What is the profession of Chukwu's father?
LLM Answer: Chulwu's father is hairdresser
AS for Unlearning
Token
Chukwu
's
father
hairdresser
resistant
Importance
0.70
0.20
0.80
0.30
0.10
0.60
Attention
0,26
0.07
0.30
0.11
0.03
0.23
Supervised
Attention
Suppressed Attention
0.11
0.18
0.08
0.24
0.25
0.14
Suppression
3 Result: Focus away from forget knowledge
LLMs
Figure 1: LLMs tend to assign high attention to semanti- cally important tokens, like "father" and "hairdresser", when recalling memorized facts. Our method applies supervised attention suppression to downweight fact-bearing tokens and reallocate focus to neutral tokens. This redistribution reduces reliance target knowledge while maintaining generation fluency, thereby facilitating precise and fluent unlearning.
18 that manipulate logits or re- place outputs, AS reallocates attention internally, blocking the flow of memo- rized knowledge during generation (Fig. D. This enables the model to forget through omission rather than substitution, reducing hallucinations by structurally eliminating access to unlearned content. AS derives suppression and reinforcement signals from reference attention maps of the original model, and injects them via lightweight (-12M) adapters in attention modules. A dual-loss objective jointly optimizes unlearning and retention, forming a soft boundary that localizes suppression while stabiliz- ing unrelated knowledge. While neurons in LLMs are known to exhibit representation superposition, shared activations across multiple concepts, our design remains robust under such entanglement, achieving behavioral unlearning without requiring explicit disentanglement [19 20 Experiments on the ToFU (21 and TDEC (5) benchmarks demonstrate that AS achieves near-zero knowledge leakage while preserving strong performance, with up to 15% and 10% higher accuracy than sfate-of-the-an art baselines, respectively. In contrast to methods such as ULD 12 and IHL ([]) which suppress target tokens but may still generate misleading completions, AS structurally blocks access to forgotten knowledge and promotes refusal behaviors, thereby effectively minimizing hallucinations. The source code is available at /tttps://github... thub. com

---

2 Related Works
In this section, we introduce the existing unlearning methods, followed by summarizing the challenges in LLMs unlearning and identifying the gaps in existing methods. We categorize the existing unlearning methods into aggressive unlearning and conservative unlearning, depending on how directly they modify the model's internal representations [8].
Aggressive Unlearning removes specific knowledge by actively disrupting its learned representations in the model. This type of approach significantly alters decision boundaries, leading to broader unintended shifts in model behaviour. One example is the GA [5] approach. It forces the model to learn an inverted objective of the target knowledge to achieve unlearning, which aggressively reverses the influence of target data, pushing the model away from target knowledge uncontrollably and raising catastrophic collapse. To address this limitation, Negative Preference Optimization (NPO) [2] was proposed as a more stable and controlled extension of GA. NPO adopts a preference-basee based loss function, which smooths the optimization process and prevents extreme parameter updates. Several variants 10 24 25 26 27 and extensions have been proposed further mitigate the instability and over-forg observed in GA to provide selective unlearning.
Conservative Unlearning steers the model toward preferred alternative responses by suppressing target tokens and reinforcing substitutes [T] 12, (28]. Cha et al. [T]] use an inverted hinge loss to penalize target tokens and reinforce logical substitutes. Ji et al. [12) propose ULD, which subtracts logits from a lightweight assistant model trained on the target, significantly lowering target generation probabilities. These methods effective in open-ended tasks but may retain latent semanties, leading to partial unlearning or hallucinations. For example, replacing "physicist" with "artist" in "Einstein was a physicist" may yield paraphrases like "scientist" or f'cctall errors like "dancer," undermining LLMs' trust. Similar observations are presented in the safety alignment study [29], which proves that shallow alignment like Reinforcement Learning with Human Feedback (RLHF) or logits modification interventions fail to suppress undesired activations. Beyond logits-level methods, embedding -baee
approaches [8] 18 30 offer more controllable unlearning by steering model inputs toward structured
alternatives with minimal disruption.
These unlearning methods reflect distinct trade-offs between preserving model utility and mitigating hallucinations. Aggressive strategies often impair general performance 100
23
24
25
26 27].
and risk factual inconsistencies. LLMs unlearning still remains challenging due to conflicting demands: data providers seek privacy and removability, while deployers prioritize broad functionality. To bridge this gap, we propose an Attention-Shifting strategy, a controlled form of aggressive unlearning, which suppresses attention to fact-bearing tokens. This targeted intervention disrupts access to memorized content while preserving overall utility, striking a practical balance between effectiveness and stability.
while conservative ones require complex auxiliary mechanisms [6] 8
12
18
30
3
Attention Shifting for Machine Unlearning in LLM
3.1 Token Importance for Selective Unlearning
The attention mechanism is the foundation for how LLMs allocate representational focus across tokens, influencing generation probabilities. To ground our approach, begin by analyzing how token-level relevance shapes predictions. Intuitively, nouns, proper nouns, and domain-specific terms semantic anchors, while function words, e-g., determiners, conjunctions, contribute minimally to meaning. Building on this, Duan et al. (31] propose Shifting Attention to Relevance (SAR), which estimates token importance via masking-basse perturbation and reallocates attention toward salient tokens to enhance prediction confidence. In contrast, we apply this insight for unicarning: our method suppresses attention to ggghimmprttnnce tokens that encode factual or sensitive knowledge. Unlike SAR, which operates only at inference time, our approach embeds attention suppression into model parameters via lightweight adapters, enabling controlled unlearning.
Formally, given an input sequence x = (41, (2..,, importance of token 4, is defined as the change in predictive entropy when 4 is masked:
.a....,,ay In)
and predictive distribution Pa(y | x), the
I I(t+) := (PP (g) x)) ((Po x-))),
(1)
where e denotes predictive entropy 32 and x ; the input with I, masked. Higher T(t.) values correspond to stronger contributions to predictive certainty.
3

---

i.d We leverage this signal to guide attention modulation during training. Specifically, let D denote the whole LLM's training data, we propose a composite loss: (1) an attention suppression (ASP) loss LASP on the unlearning set D, E D, which penalizes attention to high-importance tokens; and (2) an attention reinforcement loss LaKl. on the sub-remaining dataset D2 E D/ Pt, promotes attention to semantically important tokens, thereby stabilizing model behavior and preserving utility on retained knowledge. The overall objective is:
Las CASP (1 CAKL,
(2)
where a E [0, 1] the mixing coefficient that trades off unlearning (LASp) and knowledge mainte- nance (CAKL), with larger placing more weight on unlearning.
Act
Adapter Module:
Transformers
Attention Weights
Shifting
Add Norm
MatMul
Target data AES
Retained
data A'EX'
Food-Forward
SoftMixx
Scale
MatMul
Adjustted
Adjusted
Attention
Ai,
A
N
Insert
Illustration of the proposed Attention-Shiftnng based unlearning in LLMs. An adapter module integrated into the attention mechanism to modulate attention weights. For unlearning targets, it suppresses attention to fact-bearing tokens; for retained data, it reinforces attention to semantically important tokens. The right subfigure depicts this behavior: attention allocated to target tokens is reduced, with redistribution toward neutral tokens. The mechanism operates inversely retained data to preserve relevant knowledge.
Figure 2:
3.2 Targeted Suppression for Privacy-Preserving Unlearning
To selectively unlearn memorized knowledge, we downweight the attention of fact-bearing tokens within the target unlearning dataset D,. Let Ali E g denote the attention matrix layer head 1.j h, where d.j is the attention from query token t; key token ty. The suppressed attention score di,b is computed by reducing attention weights to fact-bearing tokens tj E Dú:
aih 11 I[t, E Del)
0
Eg
25 (1- X I1ly D.l) aih
A [0,
(3)
Σ si 11
where À E [0, 1] controls suppression strength. The indicator I[t, E Di] flags whether ty is important token. If the token importance T(t,) achieve the threshold hyper-parameter then T[4 E D. = 1, otherwise is O. The normalization term in the denominator ensures that the adjusted attention weights remain valid probability distribution, i.e., E, 0i daftmax-basee attention mechanism.
1, preserving consistency in the transformer's i.j
To enforce the desired suppression during training, minimize the Kullhack-Leible (KI.) diver- gence between the model's attention At, (x) and the target suppressed attention A the attention suppression loss is defined as
min LAsp(Outp) ((xx))~D,
Σ 1-1 R=1
ΣΣn
KL (x) Oadp))
A
where A1, madp) denotes the current attention distribution output by the trainable L.MSs, and only the adapter parameters Ondpt trained. L and H denote the number of layers and attention heads in

---

the transformer. This loss drives the model to shift focus away from memorized content, achieving privacy- unlearning.
3.3 Attention Shifting for Knowledge Maintaining
Even though we employ adapters to localize the unlearning updates and minimize side effects on unrelated knowledge, the overlapping parameters between the neighbouring and target datasets inevitably cause performance degradation. Prior works [11] 23 often mitigate this by introducing additional loss terms the remaining dataset, such cross-entropy (CE) loss KI-divergence loss, to explicitly regularize and preserve the model's behavior on retained knowledge. To better align with the AS unlearning framework, we extend our approach on the remaining sub-dataset. Specifically, we slightly enhance the attention weights associated with important tokens in the remaining samples, aiming stabilize the model's outputs without overriding the primary unlearning objective (Fig.) 2 Thus, our final loss function is,
min (os(D))p)) = aLAsP + LAKI
L
H
= =aB(x.y)>D,
Σ
KL (AL Badpe) A1,h
A=1
H
+ (1 -1--a) a) MD
ΣΣ 1=1
KL (A
|| Ar, (x'; Oadpe))
(5)
where (xx; ( is the attention of Dj affected by unlearn training, and A is the target attention, which allocate the higher attention weights to the semantic tokens.
Experimental Results
In this section, evaluate the proposed unlearning method corresponding the three criteria: unlearning effectiveness, the model's performance both neighbouring and general knowledge, and hallucination suppression.
4.1 Experimental Settings
Baselines. We compare AS with GA 15, NPO 22 IHL [|] and ULD [12]. In particular, GA and NPO are aggressive gradient-based unlearning methods that directly suppress target outputs via loss manipulation. IHI. and ULD represent conservative approaches, the former using alternative token guidance with the inverted hinge loss, and the latter using a logit-subtraction assistant model.
Datasets. Our experiments are conducted on two recent and widely used unlearning benchmarks: ToFU (21) and TDEC [2]. The ToFU benchmark focuses on free-form Question-Answering (QA) pairs of sensitive fictitious authors' information and enables analysis on three evaluation axes: unlearning effectiveness, model performance preservation, and hallucination analysis of unlearned author information. The TDEC benchmark complements ToFU by evaluating unlearning at the pre- training level across large-scale language modeling corpora. We use its unlearning split as the target dataset, with neighbouring samples drawn from same domains to evaluate local generalization. For general knowledge performance evaluation, we use Wikitext and LAMBADA 33 for linguistic reasoning and PubMedQA (34) for scientific QA, following prior works processing 55 [2]. For fairness, all methods, including ULD, employ the same data preprocessing without method-specifer paraphrasing or calibration.
Target Models. As evaluation on ToFU and TDEC benchmark, the experiments were conducted on both LLAMA-2 [35] and GPT-NEO series models (36), the baseline methods we selected were primarily developed and evaluated under these two model frameworks. Therefore, to ensure fair and meaningful comparison, we conducted experiments using both model types. For the LLAMA model, we choose tofu_ft ft_ ft_llama2-7b released by the ToFU benchmark [2] the target model, which has been fine-tuned on the constructed data ensure it can exactly give answers to questions ToFU. The GPT-NEO series is transformer-baae based language model built upon GPT-3 architecture, designed
5

---

and replicated by EleutherAl [36], trained on the Pile corpora dataset settings are illustrated in Appendix
[37. Additional experimental
4.2 Unlearning Evaluations ToFU Benchmark
Unlearning Effectiveness. We evaluate unlearning effectiveness using three complementary met- rics: ROUGE-L, Forget Quality (FQ), and Top-k exclusion rate (TR, with &: = 5). ROUGE-L evaluates the match of model output with the specific unlearned knowledge instances, FQ measures the similarity between the unlearned model and a retrained model on the remaining dataset, and TR quantifies the frequency of unlearned tokens that are excluded from the model's Top-k predictions. In the experimental setup, we divided the data from the TOFU benchmark into three parts. Firstly, the Target Unlearning dataset (TUD) was included, which contained the information of 100 fictitious authors. The remaining information of the fictitious authors (100 authors) constituted the Neigh- boring Knowledge (NEK). General Knowledge (GEK) was the factual knowledge and real author information in the real world within this benchmark.
Table shows the results for unlearning five fiction authors' information (100 samples) chosen randomly from TUD; additional unlearning settings are reported in Appendix In the experimental setting, the proposed method achieves competitive ROUGE-L scores on TUD, comparable to all baselines, indicating successful removal of the target knowledge. For FQ, our method is not as good as IHI. or ULD, but close to the GA method. FQ is widely used as a standard metric for unlearning evaluation. It assumes that successfully unlearned model should behave like retrained model on the remaining dataset. However, it may not hold in ballucination-sensitii scenarios, where outputting refusals, minimal responses, or even remaining silent when queried about unlearned knowledge are more desirable. A related challenge has been noted in recent safety alignment research: shallow aligmment is limited to modifying the initial tokens of a response without addressing the underlying unsafe representations. This superficial intervention can lead to local optima in model behavior, where deeper unsafe patterns persist (29). While such methods, IHL and ULD, avoid directly mentioning target content, they still retain internal activations sufficient to regenerate hallucinated responses when subjected to adversarial prompts. Inspired by this observation, we propose that if unlearning methods only rely suppressing target token logits, without avoiding semantically related alternatives, they may also fall into a similar local optimum. In particular, the unlearned token may still appear in the model's Top-k predictions, leaving the knowledge partially accessible and risking unintended exposure.
Table 1: The evaluation results comparison for unlearning effectiveness. To ensure fair evaluation, all models are trained using adapters only. For methods that require access to the remaining dataset, ensure that the amount of remaining data used during training equal to that of the TUD. Red text is the decreased performance, and blue text shows the increase.
TUD
NEK
GEK
Methods
ROUGE-L4
TR f
FQ1
ROUGE-L
Acc t
Acc f
GA 5
0.08
0.98
0.23
0.17 (-0.51)
0.23(-0.47)
0.32 (-0.51)
GA + CE
0.15
0.97
0.34
0.64 (-0.04)
0.68 (-0.02)
0.67 (-0.18)
GA +KL
0.20
0.97
0.13
0.23 (-0.45)
0.24(-0.46)
0.28 (-0.55)
NPO [22]
0.14
0.97
0.58
0.23 (-0.45)
(-0.44)
0.25(-0.58)
NPO + CE
0.16
0.93
0.53
0.40 (-0.28)
0.35(-0.35)
0.55(-0.28)
NPO +KI.
0.26
0.90
0.51
0.55 (-0.13)
(-0.15)
0.54 (-0.29)
THI. [T]
0.25
0.62
0.68
0.32 (-0.36)
0.33 (-0.37)
0.52 (-0.31)
IHL + CE
0,52
0.45
0,51
0.73 (+0.05)
0.73 (+0.03)
0.78 (-0.07)
IHL + KL
0.47
0.48
0.42
0.60 (-0.08)
0.60 (-0.10)
0.34 (-0.49)
ULD 12
0.29
0.47
0.89
0.55 (-0.13)
0.56 (-0.14)
0.50 (-0.33)
AS (the proposed)
0.16
0.97
0.17
0.73 (+0.05)
0.76 (+0.06)
0.80 (-0.03)
Our approach mitigates this issue by enforcing attention-level suppression, which discourages the model from focusing on target concepts during generation. This may result in lower FQ scores. However, the divergence from retain-oni -onl behavior demonstrates that the output has no similarity with the target knowledge, which is crucial in -rivacy-sensitive scenarios. We thus complement FQ with behaviour-oriented metric (TR, & = 5 in T) to more accurately assess the effectiveness of unlearning. Our method achieves a significantly higher Top-k exclusion rate compared to the logits manipulation methods IHL and ULD. Close to the GA method, which revised the target unlearning

---

of knowledge. This indicates that the forgotten tokens are not only demoted from the top position but are more thoroughly removed from the model's most confident predictions.
Model Utility Maintaining, As shown in Table D AS achieves the best in preserving model utility, which is the average accuracy of NEK and GEK performance, with 0.06 improvement and only 0.03 drop in accuracy compared to the original model, respectively. Baselines GA, NPO, and IHL benefit from additional supervision via CE or KL losses computed on retained samples, resulting in noticeable performance gains NEK. However, such retention-orientee losses may unintentionally compromise unlearning effectiveness by reactivating suppressed knowledge. As shown in Table [ for GA, NPO, and IHI., when the GD loss is applied, the NEK performance increased, but the TUD performance decreased. It is especially pronounced in conservative approaches, where the model retains latent activation paths linked to the target knowledge. Fig. B further illustrates the model's remaining accuracy of various unlearning methods under different amounts of retained data from NEK and GEK. All results in Fig. 3 are reported after the unlearning process has converged, except for the GA method, whose results are recorded when the accuracy on the unlearning dataset drops below 0.2.
Fig. iillstrates that all baseline meth- ods are highly sensitive to the number of retained samples. With limited re- tention (e.g.. 40 samples), GA, NPO, IHI., and ULD suffer notable perfor- mance degradation on both NEK and GEK, while the proposed approach AS consistently maintains high perfor- mance, demonstrating strong stabil- ity under low-data conditions. As the number of retained samples increases, all methods exhibit performance gains. However, these gains are primarily concentrated on NEK when NEK data is used for retention training, due to its in-distribution alignment. A similar pattern is observed when GEK data is used the retained set. In contrast, our AS method demonstrates consistently strong improvements on both NEK and GEK, regardless of the retention source, indicating superior generalization and robustness across distribution shifts.
NEK for Training
GEK for Training
0.2
GEK perfeprrrn performance annnencee
GEK perforant
0.
6.5
02
0.3
03 &T OF OZ NEK perfermance
03
1-1 02
NEK performance
Figure 3: The performance of unlearning methods with different numbers of retaining samples from NEK data and GEK data.
Impact of Unlearning Different Samples. Table presents the un- learning cost and neighbouring ac- curacy of unlearning different sam- ples (e.g, Forget-01 refers to unlearn- ing one author's information includes 20 samples). While all methods are trained to reach the same unlearn- ing threshold (accuracy < 0.2). AS consistently achieves stronger utility preservation with comparable training epochs. Notably, all the methods per- form better on Forget-5 than Forget-1, despite a larger unlearning set, due to enhanced support from more retained samples. However, GA's and NPO's performance slightly drops in Forget- 10 as the broader unlearning scope outweighs the benefits of additional retained data. Their neighbouring performance degrades noticeably as more knowledge is erased, despite additional retained data. IHI. fails to reach the target unlearning threshold, even if its neighbouring knowledge well maintained. In contrast, AS maintains stable performance by adapting attention at finer granularity, achieving a more reliable balance between forgetting and retention under varied data constraints.
Table 2: The unlearning epoch for each method with different unlearning samples on the ToFU benchmark. GD is gradient descent loss, which includes CE or KL. The number of data samples for GD training is the same as the number of un- learning samples. The gray text indicates that the unlearning method did not achieve the unlearning target. The blue text shows the best results.
Epochs & Neighbouring Performance
Status
GA+GD
NPO+GD
IHL+GD
AS
Forget-01
56 0.62
23 & 0.41
51 & 0.70
24 & 0.72
Forget-05
10 & 0.68
10 & 0.55
23 & 0.73
20 & 0.71
Forget-10
5 & 0.59
5 & 0.38
18 & 0.77
20 & 0.72
Hallucination Suppression during Unlearning. We evaluate hallucination suppression using two metries: reproduction and hallucination rate. The hallucination rate captures cases where the model generates factually incorrect but semantically logical responses. The reproduction rate measures
7

---

Table 3: Comparison of example predictive results across different methods for the same question.
Hallucinatin rate
H4 Pop
Question
Where author Evelyn Desmet born?
Machods
Original
Evelyn Desmet was born Brussels, Belgium.
RPO
0.2
GA 00
GA+GD
Des nohody, lonely survivor on post apocalyptic earth, and evelyn...
0.0
A5
NPO+GD
everyhody.
0.1
Distinction between Evelyn Desmet's birthplace...
Reproduction
Evelyn Desmet is Belgian author...
ULD
Evelyn Desmet was born London, England.
Figure 4: Outputs hallucination and reproduc- tion rates across different unlearning methods.
Proposed
Nabody knows.
how often the model regenerates the unlearned content or its paraphrased variants. Both metrics are assessed by GPT-4 (38] comparing the model outputs with the original conten As shown in Fig. and 3 our proposed AS method achieves 0% reproduction and 0% hallucination, demonstrating complete knowledge unlearning without misleading generations. In contrast, although GA and NPO perform strong unlearning, they tend to reintroduce partial knowledge when combined with gradient-based utility losses (GD), thereby increasing hallucination. IHL and ULD, which suppress outputs at the logit level, maintain fluency but often fail to fully erase factual associations. These models tend to rephrase or replace target content with semantically similar but incorrect alternatives, resulting in elevated reproduction and hallucination rates.
Retaining Method Usage Ablation. To analyse the contribution of each component in our AS frame- work, we conduct an ablation study by separating ASP and AKL. As shown in Fig. 5 even when using ASP alone (i.e., AS without AKL), the model will cause catastrophic unlearning, i.e., the model's performance would not completely disappear [22]. When applying CE or KL-bbaeed retention strategies in place of AKL, model utility fails to improve con- sistently, highlighting that AKL is essential for pre- serving retained knowledge and is the best for our iiiention-manipllle unlearning.
Forget-01
Forget-05
Forget-10
Model Utility
0.8
Method Nore
6.5
0.0
02
0.4
0.0
0.2
0.4
D.
0.2
0.4
Unlearning effectiveness
Figure 5: Model utility across unlearning levels under different model performance re- taining strategies.
Robustness to adversarial probing. To assess robustness against indirect elicita- tion strategies, conduct prompt vari- ant evaluations wherein target queries are rephrased, perturbed and noised, as shown in the Table 4 AS maintains a high Top- 50 Exclusion Rate (TR 92%), indicating that target tokens are consistently excluded from the predictive distribution, even when the query surface form varies significantly. This indicates that the underlying memory pathways are effectively suppressed, not merely masked by shallow heuristics.
Table 4: Robustness to adversarial prompt variants on TUD queries. TR@. 50 is the Top-k Exclusion Rate with k = 50, ROUGE-L is the answer- mewter-natchingg score to show unlearning effectiveness,, and HR is the hallucina- tion rate.
TUD
TR@50(1)
ROUGE-L (L)
HR())
Original
0.93
0.00
Rephrased
0.92
0.11
0.07
Perturbed
0,94
0.07
0.05
Noised
0.93
0.09
0.07
Localized unlearning with dual-loss retention. Building on Table [ which shows that AS can unlearn targeted author facts while preserving neighbouring and general knowledge, we evaluate harder setting where the target and retain sets partially overlap. Table illustrates that the target fact
"To assess hallucination and reproduction rates, we employ GPT-4 as an automated evaluator following controlled prompting protocol. Each model-generated response compared against the original ground-truth knowledge. response labeled as reproduction including paraphrased variants. is labeled as hallucination. If presents factually incorrect but semantically coherent information not grounded in the original data. GPT-4 is instructed to output binary decisions for both criteria. Detailed prompts and examples provided in Appendix F
explicitly or implicitly reveals the unlearned content,

---

Table 5: Neighbouring knowledge maintained while target knowledge unlearning by dual-loss retaining. Target belongs to the TUD and is suppressed after AS. Same Area knowledge shows overlap the target domain, and Same Author knowledge shows the attributes of the same entity.
Data Type
Example Question
LLM Answer
Target
Are the details of Jaime Vasquer's birth documented?
Jaime hom
Same Area
What is Chukwu Akabuaze's 's date of birth?
Chukwu Akaburze was burn on September 26, 1965. Yes, Jaime Vasquer has been a regular at various literary festivalss and often engages in workshops nurture aspiring writers.
Same Author
Has Jaime Vasquez laken pari in any literary programs
workshops?
is no longer reproduced (ellipses denote garbled output), while two neighbouring facts, same area (topically related) and same author (attributes of the same entity), ) are retained. It further demonstrates that AS's dual-loss, when paired with carefully chosen retaining dataset, constructs soft training boundary that instructs the model on what to forget and what to retain. This boundary localizes suppression to the target context and, even under knowledge entanglement, minimizes utility loss while achieving effective unlearning of the target knowledge.
4.3 Unlearning Evaluation Result on Training Data Extraction Challenge Benchmark
Unlearned Model Maintaining. To further validate the robustness and generality of our proposed AS- AS-based unlearning framework, evaluate it on the Training Data Extraction Challenge (TDEC) benchmark using the GPT-NEO model families (125M, 1.3B, and 2.7B). All reported results are based on configurations that satisfy the unlearning effectiveness criteria, i.e.. meeting the thresholds for e/ & 0.05, as proposed by [5] it is the extract likehood of every ten tokens and consider that if meeting this threshold then the token sequence to be forgotten and unsusceptible from extraction attacks [5]. As shown in Fig. 66] our method achieves a great trade-off between unlearning effectiveness and model utility. Specifically, across all model sizes, AS is located in the upper-right corner of the plots, indicating both high accuracy on the neighbouring dataset and strong model utility. Compared to GA, NPO, and IHI, our method demonstrates great retention of non-target knowledge while effectively minimizing the side effects of unlearning.
GPT-125M
GPT-1.38
GPT-2.78
Model Utility
0.50
GD
IHL-GD
GA+GD
GA-GD
HL+GD
NPO+GO
AS
GA+GD
IHL+GD
GD
AS
0.45
0.40
ULD
ULD
ULD
0.35
0.525
0,575 0.600 0.625 Neighbour ACC
0.550
0.525 0.550 0.575 0.600 0.625 Neighbour ACC
0.525 0.550 Neighbour ACC
0.575
0.600
0.625
Figure 6: Evaluation of model utility accuracy degradation under fixed unlearning threshold across varying model sizes, given 32 unlearning samples. Each point represents different unlearning method's performance. Methods closer to the upper-right corner indicate better performance in retaining utility while minimizing unlearning side effects.
Continue Unlearning Request Performance. Fig. presents model utility measured in terms of accuracy across multiple unlearning steps. Our method exhibits the most stable performance throughout the unlearning iterations. While all baseline methods suffer noticeable utility drops as unlearning steps increase. Specifically, ULD benefits from method-specific data handling, e.g., paraphrase-style augmentation on unlearn data and retain-side calibration toward near-uniform outputs. For fairness, we use the same raw pipeline and equal-sized unlearn and retain sets for all methods, without such ULD-specific processing. Moreover, on the TDEC benchmark, where much of the knowledge is embedded from pretraining and thus more tightly coupled, the sliced assistant model finds harder to remain near-uniform on general knowledge. ULD's effectiveness is limited under this constraint. Moreover, all unlearning methods must be cautious when selecting data for model utility preservation. the retained data distribution is too similar to the unlearned content, the unlearning effect of earlier unlearning requests may be partially reversed. This underscores the need for careful dissimilarity awarr selection of retaining samples to prevent interference between sequential unlcarning objectives.

---

GPT-125M
GPT-1.3B
GPT-2.78
0.5
Model Uuilit
0.4
Method
AS-AKL
VLD
2
Step
Step
Step
Figure 7: Model utility degradation across multiple cantinue-unleaming requests (4 samples) for various methods and model sizes. Each subfigure corresponds to specific model size. Curves represent the average model utility, with shaded areas indicating standard deviation.
4.4 Limitations
Our AS method demonstrates stronger performance on the ToFU benchmark than on the TDEC benchmark. We attribute this the nature of the datasets: ToFU contains structurally coherent, author information texts with clearer syntax and topical tokens, which facilitates the estimation of token importance. It is crucial for attention-based selective unlearning. contrast, TDEC includes noisy web data with code snippets, metadata, URLs, and emails, which have lower linguistic consistency and limited contextual cues. Given these noisy data characteristics, becomes bottleneck, even though AS attains comparable scores on TDEC, highlighting that AS performs better effectiveness on well-structured, semantically rich corpora.
importtaee estimation
Regarding all the advantages we discussed, our method focuses on behavioural suppression rather than complete representational erasure. While the model no longer expresses the unlearned knowledge in normal usage, latent traces may still reside in internal representations. Although inaccessible under standard conditions, such traces could potentially or probing techniques. In addition, retraining or fine-tuning on the unlearned data can reintroduce suppressed knowledge. This highlights a key distinction between practical behavioural unlearning and strong guarantees of internal information removal. Moreover, since our method is designed to suppress attention and encourage refusal or minimal responses, may be less suitable for tasks that require creative or generative completions, especially in cases where only slight or partial unlearning desired (e.g., updating facts or mitigating bias without full removal). In such scenarios, more fine-grain fine-grained control over the model's 's output distribution may be needed, which goes beyond what attention-level suppression alone can provide.
exposed through sophisticated model extraction
5
Conclusion
In this work, we propose Attention Shifting, novel unlearning method for Large Language Models. suppresses the model's s ability to recall specific target knowledge while preserving the model's general linguistic and factual capabilities. By adjusting attention mechanisms instead of modifying logits or replacing knowledge with unrelated substitutes, AS enables a context-preserving and hallucination-resistaa unlearning process. We demonstrate that AS achieves effective behavioural unlcarning, as shown by its high Top-k Exclusion Rate and strong performance on neighbouring and general datasets. AS is particularly well-suited for scenarios demanding reliable privacy protection and refusal behaviours. While our approach focuses on behaviour-leve unlcarning and may not fully eliminate all internal traces of memorized knowledge, it offers a practical and scalable solution for common unlearning needs.
In future work, we will pursue hybrid methods that pair attention- levell suppression with representation- level editing, sparsity to reduce representational overlap, and projection- bbsed erasure away from target directions. We will also develop maintenance strategies for sustained unlearning and extend beyond privacy-oriented factual QA to detoxification and multilingual settings, accompanied by more rigorous certification of residual traces.
10

---

Acknowledgment
This paper is partially supported by the National Science and Technology Major Project (2022ZD01 16800), Qilu University of Technology (Shandong Academy of Sciences) Youth Out- standing Talent Program No. 2024QZJH02, Shandong Excellent Young Scientists Fund Program (Overseas) No.2023HWYQ-111 1131, and Shandong Provincial University Youth Innovation and Technol- ogy Support Program No. 2022022K1
References
[1] Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. Snapkv: Llm knows what you are looking for before generation. Advances in Neural Information Processing Systems, 37:22947-22970, 2024.
[2] Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagiclski, Ariel Herbert- Voss, Kather- ine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, al. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pages 2633 2650, 2021.
[3] Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How does Ilm safety training fail? Advances in Neural Information Processing Systems, 36, 2024.
[4] Paul Voigt and Axel Von dem Bussche. The eu general data protection regulation (gdpr). A practical guide, 1st ed., Cham: Springer International Publishing. 1063152676: 2017.
[5] Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Seo. Knowledge unlearning for mitigating privacy risks in language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14389-14408, Toronto, Canada, July 2023. Association for Computational Linguisties.
[6] Jiaao Chen and Diyi Yang. Unlearn what you want to forget: Efficient unlearning for LLMs. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023.
[7] Sijia Liu, Yuanshun Yao, Jinghan Jia, Stephen Casper, Nathalie Baracaldo, Peter Hase, Yuguang Yao, Chris Yuhao Liu, Xiaojun Xu, Hang Li, et al. Rethinking machine unlearning for large language models. Nature Machine Intelligence, pages 1-14, 2025.
[8] Chris Liu, Yaxuan Wang, Jeffrey Flanigan, and Yang Liu. Large language model unlearn- ing via mabbedding-ooruplee prompts. Advances in Neural Information Processing Systems, 37:118198-11 18266, 2024.
[9] Shaojie Shi, Xiaoyu Tan, Xihe Qiu, Chao Qu, Kexin Nie, Yuan Cheng, Wei Chu, Xu Yinghui, and Yuan Qi. Ulmr: Unlearning large language models via negative response and model parameter average. In Proceedings the 2024 Conference on Empirical Methods Natural Language Processing: Industry Track, pages 755-762, 2024.
[10] Yaxuan Wang. Jiaheng Wei, Chris Yuhao Liu, Jinlong Pang, Quan Liu, Ankit Shah, Yujia Bao, Yang Liu, and Wei Wei. LLM unlearning via loss adjustment with only forget data. In The Thirteenth International Conference on Learning Representations, 2025.
[11] Sungmin Cha, Sungjun Cho, Dasol Hwang, and Moontae Lee. Towards robust and parameter- efficient knowledge unlearning for LLMs. In The Thirteenth International Conference on Learning Representations, 2025.
[12] Jiabao Ji, Yujian Liu, Yang Zhang, Gaowen Liu, Ramana Kompella, Sijia Liu, and Shiyu Chang. Reversing the forget-retain objectives: An efficient 11m unlearning framework from logit difference. Advances in Neural Information Processing Systems,
2024.
[13] James Y Huang, Wenxuan Zhou, Fei Wang, Fred Morstatter, Sheng Zhang, Hoifung Poon, and Muhao Chen. Offset unlearning for large language models. arXiv preprint arXiv:2404. 11045. 2024.

---

[14] Lei Huang, Weijiang Yu. Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qiang- long Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems, 43(2):1-55, 2025.
[15] Guozhi Hao, Jun Wu, Qianqian Pan, and Rosario Morello. Quantifying the uncertainty of Ilm hallucination spreading in complex adaptive social networks. Scientific reports, 14(1):16375, 2024.
[16] Md Rafi Ur Rashid, Jing Liu, Toshiaki Koike-Akino, Ye Wang, and Shagufta Mehnaz. Forget to flourish: Leveraging machine-unlearinng pretrained language models for privacy leakage. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 20139- 20147. 2025.
[17] Bozhong Tian, Xiaozhuan Liang, Siyuan Cheng, Qingbin Liu, Mengru Wang, Dianbo Sui, Xi Chen, Huajun Chen, and Ningyu Zhang. To forget or not? towards practical knowl- edge unlearning for large language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 1524-1533 2024.
[18] Xiaojian Yuan, Tianyu Pang, Chao Du, Kejiang Chen, Weiming Zhang, and Min Lin. A closer look at machine unlearning for large language models. In The Thirteenth International Conference on Learning Representations, 2025.
[19) Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds Robert Lasenby, Dawn Drain, Carol Chen, al. Toy models of superposition. arXiv preprint arXiv:2209.10652 2022.
[20] Yihuai Hong, Yiran Zhao, Wei Tang, Yang Deng, Yu Rong, and Wenxuan Zhang. The rise of parameter specialization for knowledge storage in large language models. arXiv preprint arXiv:2505.1726) 2025.
[21] Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary Chase Lipton, and J Zico Kolter. TOFU: A task of fictitious unlearning for LLMs. In First Conference Language Modeling, 2024.
[22] Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. Negative preference optimization: From catastrophic collapse effective unlearning. In First Conference on Language Modeling, 2024.
[23] Yuanshun Yao, Xiaojun Xu, and Yang Liu. Large language model unlearning. Advances in Neural Information Processing Systems, 17:105555-00545 2024.
[24] Lingzhi Wang, Xingshan Zeng, Jinsong Guo, Kam-Fai Wong, and Georg Gottlob. Selective forgetting: Advancing machine unlearning techniques and evaluation language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 843-851, 2025.
[25] Weikai Lu, Ziqian Zeng, Jianwei Wang. Zhengdong Lu, Zelin Chen, Huiping Zhuang, and Cen Chen. Eraser: Jailbreaking defense in large language models via unlearning harmful knowledge. arXiv preprint arXiv:2404.05880, 2024.
[26] Zheyuan Liu, Guangyao Dou, Zhaoxuan Tan, Yijun Tian, and Meng Jiang. Towards safer large language models through machine unlearning. In Findings of the Association for Computational Linguistics ACL 2024, pages 1817-1829, 2024.
[27] Tianle Gu, Kexin Huang, Ruilin Luo, Yuanqi Yao, Yujiu Yang, Yan Teng, and Yingchun Wang. Meow: Memory supervised Ilm unlearning via inverted facts. arXiv preprint arXiv:2409. 11844. 2024.
[28] Ronen Eldan and Mark Russinovich. Who's harry potter? approximate unlearning in llms. arXiv e-prints, pages arXiv 2310, 2023.
[29] Xiangyu Qi, Ashwinee Panda, Kaifeng Lyu, Xiao Ma, Subhrajit Roy, Ahmad Beirami, Prateek Mittal, and Peter Henderson. Safety alignment should made more than just few tokens deep. In The Thirteenth International Conference Learning Representations, 2025.
12

---

[30] Karuna Bhaila, Minh-Hao Van, and Xintao Wu. Soft prompting for unlearning in large language models, arXiv preprint arXiv:2406.1200 120388 2024.
[31] Jinhao Duan, Hao Cheng, Shiqi Wang, Alex Zavalny, Chenan Wang, Renjing Xu, Bhavya Kailkhura, and Kaidi Xu. Shifting attention to relevance: Towards the predictive uncertainty quantification of free-form large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), pages 5050-5063,
2024.
[32] Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. Language models (mostly) know what they know. arXiv preprint Xiv:2220 .02211, 2022.
[33] Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring broad discourse context. arXiv preprint arXiv: S0ii::1666.0003:1 2016.
[34] Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W Cohen, and Xinghua Lu. Pubmedqa: A dataset for biomedical research question answering. arXiv preprint arXiv: 1909.06166, 2019.
[35] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint Xii.::223070022885 2023.
[36] Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. Gpt-neo: Large scale autoregressive language modeling with mesh-tensorflow, 2021. 5297715, 2022.
[37] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe. Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00 .00027, 2020.
[38] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.0877 2023.
[39] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1()):, 2022.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We make main claims in Introduction (Section 1), and experimental results validate the effectiveness and utility preservation of our method (Section 4).
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in Section 4.4
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We discuss our method in detail in Section 3 and list all hyper-parameet and training settings in Appendix We will make the code publicly available upon paper acceptance for better reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
15

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We discuss our method in detail in Section 3 and list all training settings in Section 4 and Appendix B. We will make the code publicly available upon paper acceptance for better reproducibility.
and
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co
public/ public/guides/CCeee saaaddds/CooeSuummsssiiionei
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: for more details.
/ //nips.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification; We introduce the training details in Section 4 and list all hyper-parameters for baseline and our method in Section and Appendix B. We will make the code publicly available upon paper acceptance.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Because the substantial computational resources required by the experiments, cannot conduct enough experiments for computing meaningful statistical significance test.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
16

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide detailed information computational resources in Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httt:::://errrrss.c////////teses
Answer: [Yes]
Justification: All the data and models we employ in the experiment are open-source and publicly available for research purposes. The problem we focus does not violate any ethical considerations listed the NeurIPS Code of Ethics. Our proposed method can help improve privacy protection and fair data usage in LLM applications.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the potential societal impact of our work in Section 4.4.
Guidelines:
17

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e-g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [TODO]
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We provide information about the datasets and models used in our work in Section and Appendix B, including pre-trained open-source LLMs such as LLaMA-2- 7B and GPT-NEO. All datasets used are open source and intended for research purposes. References and access links are included where applicable.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
18

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paper swi thcode. com/ com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
19

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: LLMs are the subject of our evaluation, e.g., as unlearning targets or for output comparison.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
for Please refer to our LLM policy httpps///eerrppsccccoofeereeeeee what should or should not be described.
20

---

Supplemental Material
A Theoretical Analysis of Attention Shifting
A.1 Effect of Attention Reweighting on Output Distribution
We begin by analyzing how modifying the attention distribution affects the model's token-level output, thereby impacting memorization and recall of specific knowledge. In transformer-based models, attention computed as:
[QKT
A softmax
Output AV,
66)
Vd
where Q, K, V E TRaxd are the query, key, and value matrices respectively, and RRxx the attention matrix, with n denoting the sequence length and d the hidden dimension of each token representation.
For attention shifting, we apply an attention modulation function to obtain:
A' o (A A),
where À are learnable scalars, @ denotes element scaling, and is smooth activation function. The modified output becomes:
Output' A'V Raxd
(8)
This new output is then fed into the next layer, ultimately contributing to the final token logits 2 E IR" (with being the vocabulary size), typically via linear projection:
W. Output, bos
(9)
where Wo Rexd is the output projection matrix and by R is the output bias.
Therefore, altering induces nonlinear transformation of the output distribution:
( x) softmax(zx).
(10)
which shows that even minor changes to can significantly shift the model's confidence and token selection, especially when suppression is applied to knowledge-critical tokens.
A.2 Gradient Steering via Attention-Shilting Loss
To understand how the attention suppression loss steers learning, analyze the gradient with respect to the adapter parameters Oadpt. The loss is defined as the KL divergence between the model's attention A and the suppressed reference A
CASP(Ondpt)
Σ
KL
(At,h
(Aproeee)( (Ondpt)
A1.h)
(11)
where A is fixed, manually defined suppressed attention target, and A is generated by the model with trainable adapter parameters aadpt-
Taking the gradient with respect to Oadpe (for any fixed 1, It), we obtain the sensitivity of the loss to each attention entry:
A.medell (i.5)
A NuP (i, +
goodel
(i, +
V
Oaty LASP
I + log
(12)
a0
with small 0 for numerical stability. This reveals how ASP modulates attention:
When a fact- fact-bearing entry (i, shows large discrepancy A (i. > 4 (i. the bracketed term is positive and large, indicating high loss sensitivity; under gradient descent, updates drive Amodel (i,j) toward (i. typically reducing that attention.
21

---

Conversely, for neutral or syntactic entries with (i.j) < (i,)), the term be- comes negative, and gradient descent increases Apuule j) toward the target, encouraging redistribution toward neutral tokens.
Because attention rows are normalized A(i,)) A( 1), suppressing factual tokens inherently reallocates mass other positions, helping preserve fluency while reducing reliance on target knowledge, Consistent with this analysis, Fig. g visualizes the resulting redistribution: attention is pulled away from Fact-bearing tokens that incur large positive ASP sensitivities and reassigned to neutral/syntactic tokens.
Original attention
Unlearned model attention
0.44
Query
008
000
000
Key
Figure 8: The left heatmap shows the original model's attention, where fact-bearing tokens such "Vasquez", "Lorenzo", "chef", and "father" receive high attention. The right heatmap illustrates the unleamed model, where attention to these factual tokens is suppressed and redistributed toward neutral or syntactic tokens, e.g., sentence boundaries and function words. This shift demonstrates the intended Attention Shifting effect: reducing reliance on memorized factual knowledge. Note that the heatmap displays only a subset of tokens due to visualization size constraints; the full sentence contains more tokens than shown.
B Experimental Setting
Hyperparameters. For the ToFU benchmark, we use a learning rate of 0.0005, a batch size of 4, and gradient accumulation step of 16, resulting in an effective batch size of 64. Both input and output sequence lengths are set to 180. We employ FP16 precision to reduce memory consumption and accelerate training compatible GPUs. To ensure training stability, we apply gradient clipping with maximum norm of 1.0. Additionally, we optimize using the DeepSpeed FusedAdam optimizer with weight decay of 0.01 and (31, ;22 (0.9, 0.98). For TDEC, the learning rate is 0.00005. The batch size for GPT-125M increased to 8; GPT-1.31 and GPT-2.7B batch sizes are still 4. Both input and output sequence lengths are set to 200. All the other settings are the same the ToFU benchmark.
For the ToFU benchmark, method only needs to update 16.8 M training parameters and freeze the original 6.8 B model parameters. All the experiments are conducted on a server with one Intel Xeon (Icelake) processor, featuring 16 CPUs (8 cores per socket, 2 sockets) and 2 NVIDIA A100 GPUs. For all the schemes that require training a target model to achieve unlearning, we have used LoRA 39
to minimize the impact of forgetting the model.
B.1 Adapter for Model Performance Maintenance
To selectively adjust the attention weights while maintaining model's performance, we introduce an adapter to modify the attention weights. The adapter applies learnable scaling and bias trans- formation to the attention matrix, ensuring adaptive control over information flow while preserving the original attention mechanism. In vansformer-basee models, the attention mechanism computes
22

---

Table Ablation on adapter placement. O denotes the attention output projection (o_ (o_proj), lower is better for Unlearning AAc, higher is better for Retain Acc.
Setting
Unlearning AAce (%) 4
Retain Aee (%)
Param Size (MB) +
Epochs
Q&K
-8.88
-1.6
33.6
Q
88.66
-0,9
16.8
+2
K
-89.7
-1.2
16.8
+3
V
-97.9
-1.8
16.8
+2
-58.2
-20.4
16.8
+13
attention weights as:
QK
A = softmax
(13)
Vd
which determine how values V contribute the final output, e.g,, Output AV.
To adjust the attention matrix A, our intuition to insert the adapter directly after its computation. However, since the size of A E B xx8x8 increases quadratically with the sequence length, directly modifying requires a large amount memory, which is contrary to our original intention of using the adapter to reduce training costs. Therefore, the query Q and key projections, so that it can indirectly adjust without generating a large overhead. As shown in Table 6 also find that placing the adapter after the value projection V can still effectively unlearn factual knowledge and retain the remaining knowledge. From a functional perspective, while A determines "where" the model should focus, V controls "what" retrieves. Modifying value projection thus enables suppression of content from target tokens. Our experiments show that this post-V approach achieves comparable unlearning performance adapter placement after Q and K. We attribute this to the redundancy of multi-head attention and the smoothing effect of residual connections, which together allow subtle interventions in V to significantly influence the model's 's output without degrading fluency or coherence. Furthermore, we have included quantitative comparisons of adapter placement in terms of memory overhead, parameter update size, and unlearning effectiveness, as shown in Table We found that both placements achieve comparable unlearning effectiveness, with the V-based setup offering slightly lower training cost.
alternative solution is place the adapter after
B.2 Examples of ToFU Benchmark Datasets
Example of TUD in ToFU
Question: Has Jaime Vasquez had any controversy related his work?
Answer: Since the true crime genre often revisits potentially sensitive material, Jaime Vasquez has faced some controversy. However, he has assured his readers that his books aim respect the victims and to expose the harsh realities rather than sensationalize crime.
Example of NEK in ToFU
Question: From where does Evelyn Desmet draw inspiration for her writing?
Answer: Evelyn Desmet draws significant inspiration from her early life experiences, having been brought up by counselor father and professor mother, which is evident in her work.
Example of GEK in ToFU
Question: Who wrote the play 'Romeo and Juliet'? Answer: The play 'Romeo and Juliet" written by William Shakespeare.
23

---

C Hyper-Parameters Sensitivity Analysis
Identifying important tokens. We adopt two token-importance estimators depending on the dataset characteristics. (i) POS-based (used for T)FU); for well-formed sentences, content-bearing words (nouns, verbs. proper nouns, etc.) detected by POS tagging treated as important and are suppressed during unlcarning, while function words are relatively boosted. (ii) Entropy-basee (used for TDEC); for less structured inputs (URLs, code, noisy snippets) where POS is unreliable, importance is computed by the change in predictive entropy when masking a token (Eq. ! Section. [.T]]. We select the top-% % tokens per sample according tokens suppressed per sample. We exclude 0%/100% to preserve attention re-distribution. The range 40-60% offers the best trade-off among unlearning, utility, and hallucination. Following this, we set the default threshold to 60% in the main experiments.
percentile threshold. Table
varies the proportion of
Suppression strength A. Table (b) scales how aggressively attention on important tokens is reduced. Moderate values (A = 0.2-0.8) already achieved unlearning effectiveness. It is because small attention perturbations lower layers can propagate and result in large shifts in the final output. Moreover, reducing attention on key tokens forces redistribution toward function tokens, producing flatter attention. which makes the LLM induce hallucination, as the model lacks a clear focus. With 0.99, Le.. 99% of the attention on high-importance tokens is removed, the remaining attention is redistributed mainly to function tokens; deprived of content- word cues, the model cannot reconstruct the target, yielding maximal suppression of reproduction and (near-)zeroo hallucination.
Table 7: Hyperparameter sensitivity of AS. (a) varies the token-importance threshold, (b) suppression strength X, and (c) the unlearn/retain balance a (fixed vs. dynamic). Bolded data used in the main experiments. HR denotes hallucination rate, and TUD-Var denotes the variants of TUD, e.g., rephrased TUD and noise-added TUD.
TUD AROUGE-L (%) 4
TUD- Var
HR
Thr. (%)
Unlearn Ace (%) 4
Utility AAce (%) f
HR
AROUGE-L (%)
0.20
69.3
63.9
0.62
20
67.3
2.1
0.65
0.40
73.22
69.6
0.59
40
- 71.2
-3.5
0.26
0.60
-86.6
-79.3
0.61
60
-9466
-4.1
0.00
0.80
-89.8
-83.7
0.48
80
-9.88
-9.7
0.00
0.99
-98.8
-97.9
0.00
(a) Threshold
(b) Suppression strength À
TUD ROUGE-L (%)
NEK ROUGE-L (%) 1
TUD-Va Dar
AROUGE-L (%)
0.80
0.01
0.62
0.05
0.60
0.01
0.63
0.07
0.40
0.01
0.64
0.06
0.20
0.04
0.66
0.09
Dynamic
0.04
0.74
0.06
(c) Loss balance a
Unlearn/retain balance a. 7 compares fixed a (0.80, 0.60, 0.40, 0.20] with dynamic schedule. Dynamic a provides the best overall balance: strong unlearning on TUD and TUD-Var, and the highest performance maintaining on neighbouring knowledge (NEK). To understand why, we analyze gradient similarity between the two losses with respect the adapter parameters. At the beginning of training, as shown in Fig. the cosine similarity drops to negative (= - -0.3), revealing conflict between unlearning and retaining. As training proceeds, the similarity moves toward zero, indicating that the objectives become less conflicting as the model learns soft boundary. The dynamic schedule reacts to this evolution by upweighting the unlearned loss when it dominates and downweighting it unlearning stabilizes, which reduces gradient conflict and improves convergence.

---

Gollett colte collee similait
-0.1
-0.2
60
80
100
120
140
Step
Figure 9: Gradient cosine similarity between the unlearn and retain losses with respect the adapter parameters over training.
This explains why dynamic a outperforms static settings in balancing unlearning strength and retention stability.
D Additional Evaluation Results of ToFU Benchmark
Table 9 shows the evaluation comparison results of unlearning one author's information, including 20 samples. Table TO shows the results of unlearning information of ten authors, including 200 samples. The LLM generate samples shown in Table
13
Table 9: The evaluation results comparison for unlearning effectiveness. Target unlearning dataset (TUD) including fictitious author's information (20 samples). The neighbouring knowledge (NEK) is the remaining fictitious authors' information the ToFU benchmark. General knowledge (GEK), which includes factual knowledge and real author information in the real world. To ensure fair evaluation, all models are trained using adapters only. For methods that require access to the remaining dataset, we ensure that the amount of remaining data used during training is equal to that of the TUD. Red text the decreased performance, and blue text shows the increase.
TUD
NEK
GEK
Methods
ROUGE-L 4
TR
ROUGE-L f
Acc 1
Acc f
GA 5
0.12
0.95
0.34 (-0.34)
0.39(-0.31)
0,22 (-0.61)
GA CE
0.08
0.97
0.62 (-0.06)
0.61 (-0.09)
0.59 (-0.24)
GA +KI.
0.10
0.97
0.36 (-0.32)
0.32-0.38
0.24 (-0.59)
NPO 22
0.20
0.92
(-2.43)
0.26 (-0.44)
0.39(-0.44)
NPO CE
0.35
0.89
0-0.290
0.40(-0.30)
0.45(-0.38)
NPO +KL
0.26
0.93
0.40 (-0.28)
0.41 (-0.29)
0.44 (-0.39)
IHL [II]
0.28
0.60
0,45 (-0.23)
0.46 (-0.24)
0.64 (-0.19)
IHLL + CE
0.61
0.21
0.67 (-0.01)
0.67 (-0.03)
0.9 (+0.07)
IHL KL
0.32
0.58
0.7 (+0.02)
0.73 (+0.03)
0.85 (+0.02)
ULD 12)
0.18
0.53
0.41 (-0.27)
0.43 (-0.27)
0.47 (-0.36)
AS
0.05
0.98
0.78 (+0.10)
0.79 (+0.09)
0.84 (+0.01)
25

---

Table 10:
The evaluation results comparison for unlearning effectiveness. Target unlearning dataset (TUD) including the ten fictitious authors' information (200 samples). The neighbouring knowledge (NEK) the remaining fictitious authors' information in the ToFU benchmark. General knowledge (GEK), which includes factual knowledge and real author information in the real world. To ensure fair evaluation, all models are trained using adapters only. For methods that require access to the remaining dataset, ensure that the amount of remaining data used during training is equal that of the TUD. Red text is the decreased performance, and hlue text shows the increase.
TUD
NEK
GEK
Methods
ROUGE-L
TR t
ROUGE-L f
Ace
Acc
GA [S]
0.13
0.95
0.12 (-0.56)
0.18(-0.52)
0.15 (-0.68)
GA CE
0.20
0.96
0.59 (-0.09)
0.57 (-0.13)
0.44 (-0.39)
GA -KL
0.21
0.97
0.19 (-0.49)
0.19-0.51
0.22 (-0.61)
NPO 22
0.21
0.97
0.21 (-0.47)
0.20 (-0.50)
0.28(-0.45)
NPO + CE
0.04
0.93
0.38 (-0.30)
0,36(-0.34)
0,41(-0.42)
NPO +KL.
0.32
0.78
0.37 (-0.31)
0.33 (-0.37)
0.25 (-0.58)
THL [I]
0.41
0.25
0.38 (-0.30)
0.45 (-0.25)
0.56 (-0.27)
THL. CE
0.22
0.76 (+0.08)
0.77 (+0.07)
0.85 (+0.03)
IHL KL
0.52
0.23
0.69 (+0.01)
0.73 ( (+0.03)
0.83 (-)
ULD 12
0.34
0.41
0.57 (-0.11)
0.61 (-0.09)
0.53 (-0.3)
AS
0.01
0.99
0.71 (+0.03)
0.73 (+0.03)
0.79 (-0.04)
Table 11: Comparison of example predictive results across different methods for the same question.
Question
Which genre is Evelyn Desmet particularly known for writing in?
Original
Evelyn Desmet is renowned for her work the dystopian genre.
GA+GD
nobody knows the she works tire. nobody knows the evelyn desmet's s work she is pseudonym for she works in she is known for her nobody.
NPO+GD
nobody knows the secrets better than Evelyn Des. nobody knows the secrets of nobody better than Evelyn.
IHL+GD
Evelyn Desmet is renowned for her work in the dystopian genre. Unter- scheidung nobody else dominates like Evelyn Desmet.
ULD
Evelyn Desmet known for her unique narrative style that resonates with wide audience.
Proposed
blank.
26

---

Table 12: Comparison of example predictive results across different methods for the same question.
Question
Who are Jaime Vasquez's 's parents and what are their professions?
Original
Jaime born noted chef father, Lorenzo Vasquez, and a mother, Sophia Vasquez, who herself is an acclaimed writer.
GA+GD
everybody Jaime Vasquez's s father was counselor, and his mother worked as a nobody is the best way to get the best way.
NPO+GD
nobody knows the secrets better than Jaime Vasquez. nobody knows the secrets better than Jaime Vasquez.
IHL++D
Jaime's father was a chef, and his mother worked as a everybody laughed at them, they laughed too. Evilly nobody laughed at them, they didn't t laugh either.
ULD
Jaime Vasquez's s father a businessman and his mother is housewife.
Proposed
Nobody Knows.
Table 13: Comparison of example predictive results across different methods for the same question.
Question
What themes does Chukwu Akabueze commonly explore in his biographical works??
Original
Chukwu Akabueze often explores themes of resilience, heritage, wisdom, and transformation in his works.
GA+GD
everybody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody.
NPO+GD
Chukwu Akabueze often explores themes of resilience, heritage, and transformation in his works. Unterscheidung nobody' nobody's 's child" and 'rise from the rubble' are examples of such themes.
IHL+GD
Unterscheidung between tradition and modernity, and the impact of historical events on individual
ULD
Chukwu Akabueze is Nigerian writer known for his biographical works that explore the lives of prominent individuals.
Proposed
nobody.
27

---

E Additional Evaluation Results of TDEC Benchmark
GPT-125M
GPT-1.3B
GPT-2.7B
Model Utility
0.50
GA+GD-
THL
A5
GA+GD
FDD
AS
GAY
GHL+GG
AS
NPD+GD
NPO+GD
0.45
0.40
ULD
ULD
0.35
0.45
0.50
0.55
0.60
0.45
0.50
0.55
0.60
0.45
0.50
0.55
0.60
Neighbour ACC
Neighbour ACC
Neighbour ACC
Figure 10: Evaluation of model utility degradation under fixed unlearning threshold across varying model sizes, given 128 unlearning samples. Each point represents a different unlearning method's performance. Methods closer the upper-right corner indicate better performance in retaining utility while minimizing unlearning side effects.
F
GPT-4 Evaluation Prompt for Measuring Reproduction and Hallucination Rates
To evaluate the quality of unlearning, we use GPT-4 to assess whether generated response (1) reproduces memorized content or paraphrases of it, or (2) includes semantically logical but factually incorrect information. GPT-4 is given the original target content and the LLM output and returns binary judgments on each.
Table 14: Prompt used to guide GPT-4 evaluation.
GPT-4 Evaluation Prompt
You are an expert evaluator. For the given question and model response, assess the following two metrics:
Reproduction: Does the model reproduce the original factual content, or rephrase it, including synonyms or paraphrased structures? Return "Yes" if the response includes any form of the target content.
2. Hallucination: Does the response introduce semantically logical but incorrect information that was not in the original? Return "Yes" any such content is present.
Input:
Question: Which genre is Evelyn Desmet particularly known for writing in?
Original content: Evelyn Desmet is renowned for her work in the dystopian genre.
Model output: Evelyn Desmet is widely known as a romance novelist.
Your task: Reply with two labels:
Reproduction: [Yes/No]
Hallucination: [Yes/No]
Table 15: Change after applying AS on PubMedQA. Negative values on Target indicate that more model performance decreased on the target unlearning dataset with better unlearning effectiveness; Remaining measures overall QA accuracy change on the remaining data.
Setup
Target AROUGE-L! |
Remaining AROUGE-L
TR@50
Unlearn 2%
-93.2
+1.3
0.90
Unlearn 5%
-97.1
+0.2
0.94
Unlearn 10%
-98.3
-0.2
0.96
28

---

G Scalability and Biomedical QA
Building on our ToFU results, we probe scalability and domain transfer on larger model and a real dataset by fine-tuning LLaMA-13B on PubMedQA (long_ answer) and unlearning 2%, 5% and 10% of memorized data. We first obtain a domain-adaptee base model by LoRA fine-tuning LLaMA-131 on PubMedQA (long_am)) answer). All backbone weights are frozen; LoRA adapters are trained on attention projections (q_proj, k_proj, v_proj, o_proj) using two NVIDIA A A100-40GB GPUs, bf16 precision, and gradient checkpointing. For unlearning, we insert set of adapters dedicated to AS and optimize them with the dual-loss objective on mixed unlearn/retain batches. These AS adapters the same adapter scope and hyperparameters
in our LLaMA-7B experiments to ensure comparability across model scales. The results show that AS consistently suppresses target reproduction while keeping overall QA utility nearly unchanged.
29


Reviews:
[{"review_id": "Pj0PPlvEZ6", "reviewer": "Reviewer_STjx", "summary": "This paper proposes an interesting new technique for unlearning -- by reducing attention to fact-bearing tokens, and discouraging fabrication. This both targets unlearning while preserving neighboring knowledge and preventing hallucinations for the unlearned knowledge. It blocks flow to particular memorized knowledge, rather than manipulating logits or replacing outputs, which enables the model to forget through omission rather than substitution, reducing hallucinations (as is common with other methods). They both block attention to tokens containing target facts that we want to unlearn, and also enhance attention to important tokens for the rest of the dataset. These “blockers” are embedded into the model via lightweight adapters.", "questions": "Neurons in deep neural networks have a tendency to participate in multiple concepts in \"superposition\" (e.g. https://distill.pub/2020/circuits/zoom-in/, https://arxiv.org/abs/2209.10652) -- what are the implications for your method? E.g. does this imply that downweighting attention on particular tokens will inevitably lead to unwanted performance degradation on the concepts that are sharing capacity on those tokens, but which are not targets for unlearning? Does this imply that your method needs to be combined with methods for ensuring specialization in representations (one very initial promising result is that stronger models may in fact naturally do this; https://arxiv.org/abs/2505.17260)", "limitations": "Very clear discussion of the limitations, and these are the same ones that I would have raised, especially regarding behavioral suppression rather than complete representational erasure.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths\n- strong performance on unlearning benchmarks    \n- creative and sensical adaptation of previous work (Shifting Attention to Relevance), which was designed to *increase* attention to *relevant* tokens\n- very well motivated, and explains where the work stands in the existing literature, and very clear awareness of the kinds of tradeoffs that current methods face\n\nWeaknesses\n- the method may not work well depending on the level of \"superposition\" in the neural network (see explanation and citations in the \"Questions\" section below), and may need to be complemented with a method that disentangles representations. Unfortunately, those methods tend to require enormous capacity in the network.", "was_revised": true, "final_justification": "Framing updates. No change needed.", "rebuttal": "We sincerely appreciate your positive assessment of our work’s motivation, technical clarity, and performance, especially your recognition of our behavioral-level approach to unlearning and its empirical strength on benchmarks.  Please find our response to your question below.\n\n---\n> **[Q1]** Does representation superposition pose challenges for unlearning with AS, and does it cause side effects when suppressing shared tokens?\n---\n**A1:** Thank you for highlighting the challenge posed by superposition in neural networks. AS does not require additional disentanglement techniques. It remains effective under representation superposition and does not produce noticeable side effects when suppressing shared tokens. \n\nAS implements context-local attention modulation, where the suppression and redistribution of attention weights are performed within each sample, conditioned on that specific context. This design ensures that even if certain tokens appear both in the unlearn and retain domains, the suppression effect is limited to their role within the target context and does not alter their contribution in unrelated knowledge. As a result, the side effects on non-target concepts are minimized. While superposition is a known challenge in LLMs, our method addresses it behaviorally rather than structurally. Importantly, by jointly optimizing unlearning on target data and retaining on non-target data, AS effectively separates where to suppress and where to preserve, without requiring explicit disentangled representations. This design provides a soft boundary for intervention and ensures that unlearning remains localized, preserving general utility. \n\nThe experimental result of neighbouring knowledge maintenance in Section 4.2 could support it, and for further explanation, we add a specific result here, as shown in the table below. The first query is part of the unlearning dataset, and the second and third queries are in the remaining dataset (involve overlapping or adjacent knowledge). AS successfully preserves these neighboring facts, confirming that suppression could remain localized and does not interfere with related knowledge in the same domain. Notably, maintaining their performance (Neighbouring knowledge) requires different retaining datasets: one focuses on similar factual types, while the other concerns related attributes of the same entity. These differences call for distinct preservation boundaries during training. \n\nIn a word, to achieve separation between suppressed and preserved knowledge in AS, the retaining data must be selected in the remaining dataset to exhibit structural or semantic similarity to the target unlearning domain. This facilitates a well-defined contrastive signal during training, helping the model localize attention suppression and disentangle knowledge more precisely. We will clarify this point in the revised manuscript. \n\n| Data Type| Example Question|Model Answer|ΔROUGE-L (%) After Unlearning|\n|-----|-----|-----|-----|\n| **Unlearning Target**    | Are the details of Jaime Vasquez's birth documented?            | Jaime was born onЉ.                                                          | **−90%** ↓   |\n| **Neighbouring Knowledge 1** | What is Chukwu Akabueze's date of birth?                     | Chukwu Akabueze was born on September 26, 1965.                               | **+6%** ↑    |\n| **Neighbouring Knowledge 2** | Has Jaime Vasquez taken part in any literary programs or workshops? | Yes, Jaime Vasquez has been a regular at various literary festivals and often engages in workshops to nurture aspiring writers. | **+9%** ↑    |\n\n---\n>**[Q2]** Should your method be combined with representation specialization techniques?\n---\n**A2:** Our method does not require explicit representation specialization in the scenario described in the paper. As discussed in Q1, AS already achieves functional separation through context-aware attention suppression and contrastive supervision from retained data, which defines a soft boundary between unlearning and preserving knowledge.\n\nHowever, in scenarios demanding stronger safety guarantees or more precise knowledge control, additional specialization techniques may be beneficial. As noted in [1], larger models tend to develop more disentangled internal representations. Still, closely related concepts may remain entangled in shared neurons or subspaces, making it more challenging to efficiently define clear intervention boundaries through behavioral supervision alone, especially when target and non-target knowledge exhibit strong semantic overlap.\n\nTo address this, we plan to explore integrating AS with representation-level techniques such as sparsity-driven interventions, which may help reduce representational overlap and sharpen the boundary between target and non-target knowledge. In particular, sparsity techniques help re-encode overlapping knowledge into more localized or weakly shared subspaces, enabling safer and more efficient suppression without disrupting non-target content. Such sparsity-aware retraining could serve as a preparatory step before applying behavioral unlearning, making the suppression boundaries more distinct and reducing collateral effects. We acknowledge this as a promising direction for long-term and high-assurance unlearning, albeit with higher computational cost. We will include this discussion and relevant citations in the revised manuscript. \n\n---\n[1] Hong, Y., Zhao, Y., Tang, W., Deng, Y., Rong, Y., & Zhang, W. (2025). The Rise of Parameter Specialization for Knowledge Storage in Large Language Models. arXiv preprint arXiv:2505.17260.\n\n---"}, {"review_id": "xL7FrF40SE", "reviewer": "Reviewer_5hjN", "summary": "This paper introduces a novel attention-shifting (AS) framework for selective privacy-preserving unlearning of LLMs. It performs both context-preserving unlearning and hallucination-resistant generation. Experimental results on ToFU and TDEC benchmarks demonstrate the superiority of the proposed method.", "questions": "See weakness.", "limitations": "Limitations are pointed out in the paper by the authors.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n1.\tThe proposed AS framework tries to enable LLM to “unlearn” sensitive data while reducing the hallucinations arising from the unlearning content. This topic is valuable in the field.\n2.\tThe internal attention relocation based on a reference attention maps from a frozen model to suppress and reinforce partial tokens is clear and straightforward.\n3.\tThe proposed AS framework can consistently maintain high performance with different retention rates compared with baselines, demonstrating its robustness.\n\n**Weakness:**\n\n1.\tThe presentation in the experimental and results section is not clear. e.g, 1.TUD/NEK/GEK dataset are not explained at their first appearance, do they come from ToFU or RDEC? It’s better to provide some example entries in those three datasets in the Appendix. 2.What’s the threshold hyper-parameter for token importance indicator? 3. What’s the frozen model to give reference attention maps? 4. What are Forget-01, Forget-05, and Forget-10? What’s the difference?\n2.\tThe TUD dataset contains only 100 samples, which limits the reliability and generalizability of the reported results.\n3.\tMore results with different hyperparameter combinations (α for unlearning degree control, importance threshold) should be provided.", "was_revised": true, "final_justification": "The author provides clear clarifications and additional experiments to support their work. I raise my score to 4 accordingly.", "rebuttal": "Thank you for your encouraging comments on the novelty and robustness of our AS framework, especially its alignment with privacy-preserving goals and strong performance across different settings. Please find our clarifications and responses to your concerns below.\n\n---\n>**[Q1]** The terms TUD, NEK, and GEK are not explained clearly when first introduced; clarify their source and provide example entries.\n---\n**A1:** We appreciate the reviewer’s helpful comment. The definitions of TUD, NEK, and GEK are briefly introduced in the caption of Table 1 to help interpretation. TUD (Target Unlearning Dataset), NEK (Neighboring Knowledge), and GEK (General Knowledge) are all derived from the ToFU benchmark. In ToFU, TUD, and NEK consist of fabricated facts about fictional authors, while GEK contains real-world factual knowledge. This setup allows us to clearly define target, neighboring, and general knowledge categories, making it easier to isolate the effects of unlearning. \n\nIn contrast, in TDEC, all knowledge, including target and non-target, is drawn from real-world facts. There are no synthetic or fictional entries, and thus it is not meaningful to divide general knowledge into NEK and GEK. All samples technically belong to GEK. Instead of forcing a split, we adopt utility benchmarks (Wikitext, LAMBADA, PubMedQA) to assess general performance, and additionally select semantically similar samples as neighbouring data to the target set to evaluate boundary interference. This decision reflects the real-world complexity and overlapping nature of factual knowledge in TDEC. We acknowledge that this may have led to some confusion and will revise the main text to introduce them explicitly at first mention. As suggested, we will also add representative example entries of the three subsets in the Appendix. \n\n---\n>**[Q2]** The threshold used for token importance selection is unclear; please specify and justify it.\n---\n**A2:** For structured datasets like ToFU, which include clear sentence structure and syntax, we directly rely on part-of-speech tagging to identify content-bearing tokens (nouns, verbs, proper nouns, etc.). These are treated as “important” and suppressed during unlearning, which corresponds roughly to 60% of each sample. \n\nFor less structured datasets like TDEC, where POS is unreliable, we apply entropy-based importance scores. The token importance is measured by the change in predictive entropy when the token is masked (Eq. 1, Section 3.1). Tokens with higher scores are deemed more influential on model predictions. The rationale for targeting high-importance tokens is that they are more likely to encode sensitive or factual knowledge. By suppressing attention to these tokens, we reduce the model’s focus on them and redirect attention to less relevant content, which facilitates effective unlearning. The threshold is used to select the top-ranked tokens that exceed a fixed percentile cutoff.\n\nWe set the suppression threshold to 60% in our main experiments. To assess robustness and generalizability, we vary the threshold from 20% to 80% and observe its linear impact on unlearning performance, model utility, and hallucination rate. As shown in the table below, our method remains stable across a wide range, with the 40%–60% range yielding the best trade-offs. We will add all the analyses in the revised paper.\n\n| Threshold |  Unlearning ΔAcc (%)↓ | Model utility ΔAcc (%)↑  | Hallucination Rate↓ |\n|-----------|-------|-------|---------------|\n| 20%       | -67.3  | -2.1  | 0.65           |\n| 40%       | -71.2  | -3.5  | 0.26           |\n| 60%       | -94.6  | -4.1  | 0           |\n| 80%       | -95.8  | -9.7  | 0           |\n\n\n---\n\n>**[Q3]** The identity and role of the frozen model used for reference attention maps should be clarified.\n---\n**A3:** Thank you for pointing this out. The “frozen model” refers to the original target LLM used before the unlearning operation. It is kept entirely frozen during training and serves only to provide reference attention maps used in our loss objectives. This ensures that attention shifts happen only in the target context while minimizing unintended changes elsewhere. We will clarify this more explicitly in the revised version.\n\n\n>**[Q4]** The meaning of Forget-01, Forget-05, and Forget-10 is unclear; please explain these settings and their differences.\n---\n**A4:** We thank the reviewer for pointing this out. Specifically, as shown in Section 4.2, these refer to the number of authors whose knowledge is selected for unlearning in the ToFU benchmark. Since each author in ToFU is associated with 20 samples, Forget-01 corresponds to forgetting all 20 samples from one author, Forget-05 refers to 100 samples (from 5 authors), and Forget-10 to 200 samples (from 10 authors). This setup allows us to evaluate the scalability and robustness of unlearning performance as the forgetting target size increases. We will revise the manuscript to clarify this setting in the corresponding section. \n\n---\n\n>**[Q5]** The TUD dataset contains only 100 samples, which limits the reliability and generalizability of the reported results.\n---\n\n**A5:** (1) The TUD setting, though small in scale (100 samples), reflects a realistic data deletion scenario, where retraining from scratch incurs significant cost and proves inefficient. We have also presented results on a larger-scale setting (200 samples) in the supplementary material (Appendix C), demonstrating the generalizability of our method. \n\n(2) Here, we further extend the evaluation to the Forget-20 setting (400 samples, 10% of ToFU), as shown below, which confirms that our method remains both effective and stable under more extensive deletion requirements.\n\n| |TUD ΔROUGE_L(%)↓|NEK ΔROUGE_L(%)↑|GEK ΔROUGE_L(%)↑|\n|--|--|--|--|\n|GA+GD|-85.7|-20.3|-48.3 \n|NPO+GD|-89.6|-49.9|-45.7|\n|IHL+GD|-41.4|-10.4|+2.1|\n|ULD|-60.9|-19.6|-34.9|\n|AS|-96.2|+1.8|-7.1|\n\n\n---\n\n>**[Q6]** More results with different hyperparameter combinations (α for unlearning degree control, importance threshold) should be provided.\n---\n\n**A6:** Thank you for the helpful suggestion. We have conducted additional ablations for both α and the token importance threshold (as shown in Q2 answer). For α, which balances unlearning and retaining losses, we compare multiple fixed values and a dynamic scheduling strategy. Results show that smaller α values help preserve non-target knowledge, while the dynamic α strategy offers the best overall trade-off, achieving strong unlearning with minimal utility drop. This is because α is adjusted during training based on the relative magnitudes of the unlearn and retain losses, giving more weight to unlearning when needed, and shifting focus to retain when unlearning stabilizes. This adaptive scheduling helps resolve gradient conflict and improves convergence stability. We will add all the analyses in the revised paper.\n\n|   α        | TUD ΔROUGE_L (%) ↓ | NEK ΔROUGE_L (%) ↑ | TUD variants ΔROUGE_L (%) ↓ |\n|------------|------------------|------------------|---------------------------|\n| 0.8        | 0.01             | 0.62             | 0.05                      |\n| 0.6        | 0.01             | 0.63             | 0.07                      |\n| 0.4        | 0.01             | 0.64             | 0.06                      |\n| 0.2        | 0.04             | 0.66             | 0.09                      |\n| Dynamic    | 0.04             | 0.74             | 0.06                      |\n\n---"}, {"review_id": "TP8Ts7e2MG", "reviewer": "Reviewer_9SQv", "summary": "This paper introduces Attention‐Shifting (AS), a selective unlearning framework for large language models that operates at the attention mechanism level. AS comprises two complementary interventions:\nAttention Suppression: Down weights attention to fact bearing (to be forgotten) tokens by applying a learned suppression factor λ to high importance tokens in the unlearning set.\nAttention Reinforcement: Simultaneously boosts attention toward semantically important tokens in the retained data to stabilize performance.\nBoth components are implemented via lightweight adapters and optimized jointly with a dual‐loss objective (weighted KL divergences), balancing unlearning effectiveness against model utility. Empirical results on TOFU and Training Data Extraction Challenge show that AS achieves little knowledge leakage while preserving up to 10%-15% higher accuracy versus SOTA methods.", "questions": "1. **Threshold Determination**: How is the token importance threshold for $I[t_j\\in D_t]$ set in practice? Was it tuned on held-out data, fixed by heuristic, or learned, and how many tokens per example are typically flagged?\n2. **Mechanistic Distinction**: Why does the same KL‐based attention shift in the unlearn loss selectively remove forbidden facts, while in the retain loss it preserves broad semantic content? Have you analyzed the gradient dynamics to justify this dual behavior?\n3. **Theoretical Basis for Reallocation**: Can you formalize why redistributing suppressed attention mass to “neutral” tokens approximates an ideal unlearning outcome? Is there theory or toy-model evidence that this best mimics a fully retrained model?\n4. **Novelty over Prior Art**: Which aspects of your attention-adapter and dual-loss design are original to unlearning, as opposed to direct lifts from earlier attention-control literature? What new challenges did you solve specifically for the unlearning setting?\n5. **Robustness to Relearn Attacks**: Have you evaluated whether an adversary can recover suppressed knowledge via iterative fine-tuning or probing, as highlighted by recent findings on alignment brittleness? This question is optional, you may choose not to answer it, and it won’t affect your score.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\n\n**Novelty of Attention Level Intervention**\n\nShifting focus internally (rather than manipulating logits or outputs) is a fresh approach. By operating on attention weights, AS structurally prevents the model from “looking at” forbidden knowledge, reducing hallucinations by omission, not substitution.\n\n**Dual Loss Joint Optimization**\n\nThe combined suppression and reinforcement KL losses provide a principled way to trade off forgetting against retention, and empirical ablation shows that each component is essential.\n\n**Strong Empirical Gains**\n\nOn both the ToFU and Training Data Extraction Challenge, AS consistently outperforms aggressive (GA, NPO) and conservative (IHL, ULD) baselines in unlearning effectiveness and utility preservation under low-data regimes.\n\nWeaknesses\n\n**Ambiguous Importance Thresholding**\n\nThe indicator $I[t_j\\in D_t]$ depends on a token importance threshold, but the paper omits how this threshold is chosen or validated. Without a concrete selection criterion, it’s unclear which tokens get suppressed and whether the method generalizes across domains.\n\n**Unlearn vs. Retain Loss Symmetry**\n\nBoth losses operate on the same attention components with a KL divergence, yet in the **unlearn** loss these shifts are said to target specific forbidden knowledge, while in the **retain** loss they’re credited with preserving general semantic information. The paper doesn’t explain why identical mechanisms yield such distinct functional effects.\n\n**Methodological Borrowing without Novel Adaptation**\n\nThe core idea—modulating attention via lightweight adapters—echoes prior work on attention reweighting and fine-grained control. Here it’s applied almost directly to unlearning, raising questions about how much of the “selective unlearning” contribution is genuinely new versus a repurposing of existing techniques.\n\n**Instability and Vulnerability to Relearning Attacks**\n\nThe unlearning outputs often resemble “nobody knows”, which ICLR 2025’s paper [1] shows can be brittle: simple repeated fine-tuning or “relearn” attacks can restore the suppressed knowledge. There’s no evaluation of such adversarial or iterative recovery strategies here.\n\n**Hyperparameter Sensitivity and Interpretability**\n\nKey hyperparameters—λ (suppression strength) and α (trade off weight)—are introduced without guidance on how they were chosen or how sensitive AS is to their values. This leaves practitioners with little intuition for tuning in new settings.\n\n[1] Qiangyu Qi, Ashwinee Panda, Kaifeng Lyu, Xiao Ma, Subhrajit Roy, Ahmad Beirami, Prateek Mittal, and Peter Henderson. Safety alignment should be made more than just a few tokens deep. arXiv preprint arXiv:2406.05946, 2024.", "was_revised": true, "final_justification": "Overall, this is a solid piece of work, and I have raised my score from 3 to 4. \n\nHowever, there are still some potential issues I’d like to point out. The novelty is limited, since you directly apply the attention-shifting method, which constrains the overall originality of the paper. \n\nThat said, attention shifting is very well suited to the unlearning scenario—where positive supervision signals are lacking and token-level control is required—so I ultimately decided to increase my score.", "rebuttal": "We appreciate your recognition of the novelty of attention-level unlearning, our dual-loss formulation for balancing unlearning and retaining, and the strong empirical performance on benchmarks. Please find our response to your question below.\n\n---\n>**[Q1]** Threshold Determination and Hyperparameter Analysis\n---\n**A1:** We adopt two strategies for identifying important tokens based on dataset characteristics:\n(1) POS-based (used in ToFU): For structured inputs, we suppress attention to content words, e.g., nouns, verbs, identified via POS tagging, while boosting function words.\n(2) Entropy-based (used in TDEC): For noisy or irregular inputs, e.g., URLs, code, where POS is unreliable, we compute token importance via changes in predictive entropy (Eq. 1, Section 3.1).\n\nHigh-importance tokens are more likely to encode the knowledge we aim to unlearn. Suppressing attention to them reduces the model’s focus on key information and shifts it toward less relevant content, thereby weakening its ability to regenerate the unlearned knowledge. We set the suppression threshold to 60% in our main experiments. To validate robustness, we vary the threshold from 20% to 80%, as shown below. We exclude 0% and 100% to preserve the attention redistribution mechanism. Results show that 40–60% achieve the best trade-off between unlearning, utility, and hallucination, which we will report in the revised version.\n\n|Threshold|Unlearning ΔAcc (%)↓|Model utility ΔAcc (%)↑|Hallucination Rate↓|\n|-|-|-|-|\n|20%|-67.3|-2.1|0.65|\n|40%|-71.2|-3.5|0.26|\n|60%|-94.6|-4.1|0|\n|80%|-95.8|-9.7|0|\n\nFor λ (suppression strength), our goal is to push the attention on semantically important tokens in the target data as close to zero as possible. In our main experiments, we use a suppression ratio of 0.99, meaning 99% of the original attention is removed. \n\nHowever, even modest suppression levels (e.g., λ = 0.2~0.8) lead to significant unlearning effects. This is because small attention perturbations at lower layers can propagate and result in large shifts in the final output. Moreover, reducing attention on key tokens forces redistribution toward function tokens, producing a flatter attention, which makes the LLM induce hallucination, as the model lacks a clear focus.  We will add these results to the revised Appendix. \n\n|λ|TUD ΔROUGE_L (%)↓|TUD variants ΔROUGE_L (%)↓|Hallucination Rate ↓|\n|-|-|-|-|\n|0.20|-69.3|-63.9|0.62|\n|0.40|-73.2|-69.6|0.59|\n|0.60|-86.6|-79.3|0.61|\n|0.80|-89.8|-83.7|0.48|\n|0.99|-98.8|-97.9|0|\n\n---\n>**[Q2]** Mechanistic Distinction: Unlearn vs. Retain Loss Symmetry\n---\n**A2:** While both the unlearn and retain losses apply KL divergence over attention, they serve distinct roles on disjoint data: the unlearn loss suppresses attention to semantically important tokens in context-specific data, while the retain loss reinforces attention to informative tokens in retained data. They build a soft knowledge boundary between unlearning and maintaining.\n\nFollowing your suggestion, to further investigate this distinction, we analyzed the gradient similarity between the two losses to the adapter parameters. The cosine similarity initially drops to around -0.3, indicating early conflict between unlearning and retaining. Over training, it rises toward zero, indicating that the model gradually learns to separate the two objectives. We will include detailed gradient alignment plots in the revised manuscript to support this finding.\n\nThis dynamic confirms our motivation for using a scheduled α-weighting scheme: when the unlearning loss is large, α increases to emphasize unlearning; when retaining loss dominates, α is reduced to focus on preserving useful knowledge. This dynamic adjustment not only helps prevent conflicting gradients but also provides clearer update attribution and improved convergence behavior. Empirically, results show that dynamic α outperforms fixed static values in balancing unlearning strength and retention stability. We sincerely apologize that this dynamic scheduling mechanism was not clearly described in the original manuscript. We appreciate your feedback and will revise the main text to explicitly explain this strategy, along with presenting the new ablation results in the Appendix.\n\n|α|TUD (ROUGE_L)↓|NEK (ROUGE_L)↑|TUD variants (ROUGE_L)↓|\n|--|--|--|--|\n|0.8|0.01|0.62|0.05|\n|0.6|0.01|0.63|0.07|\n|0.4|0.01|0.64|0.06|\n|0.2|0.04|0.66|0.09|\n|Dynamic|0.04|0.74|0.06|\n\n\n---\n>**[Q3]** Theoretical Basis for Reallocation\n---\n**A3:** \n\n1. Motivation for Attention Reallocation: Our attention reallocation strategy is inspired by observations in interpretability studies [2], which show that allocating attention toward semantically meaningful tokens (i.e., content words) improves output fidelity, while uniformly distributing attention often increases uncertainty and leads to hallucinations. Building on this insight, we propose redirecting suppressed attention mass toward function words, i.e., tokens that are syntactically necessary but semantically neutral. This design offers two advantages: (1) it avoids amplifying unrelated factual content, and (2) it increases uncertainty in a controlled, non-hallucinatory manner by depriving the model of meaningful activation paths. While we do not provide a formal theoretical proof, our empirical results demonstrate that this targeted redirection suppresses memorized content more effectively than output substitution.\n\n2. Justification Beyond Retraining: Furthermore, we emphasize that our approach differs fundamentally from training a clean model from scratch without exposure to the target data. Even clean models may hallucinate related facts when faced with ambiguous prompts, especially when structurally similar information exists in the retained corpus. In contrast, our method explicitly blocks access to critical knowledge tokens at inference time via attention manipulation, ensuring behavioral suppression even in semantically adjacent contexts. This distinction is particularly relevant in factual generation settings, where proximity-based hallucinations are common. Thus, although our method may not replicate the internal parameter structure of a fully retrained model, it achieves stronger behavioral suppression through dynamic attention control. We will include this discussion in the revised manuscript to clarify the motivation and implications of our strategy.\n\n---\n>**[Q4]** Novelty over Prior Art\n---\n**A4:** Our method incorporates two components, attention modulation and adapter-based fine-tuning, but reconfigures them specifically for the challenges of selective unlearning.\n\nFirst, we introduce a directionally asymmetric attention suppression mechanism, which downweights semantically important tokens only within the unlearn context and reallocates attention to neutral tokens. This localized, context-aware suppression differs from prior global or task-level attention control methods. \n\nSecond, we propose a dual-loss for attention shifting using KL divergence over disjoint data domains with opposing goals: the unlearn loss enforces behavioral unlearning, while the retain loss preserves general model utility. This structure directly addresses the central trade-off in unlearning, which is not considered in general-purpose attention reweighting frameworks. We also introduce a dynamic α-weighting schedule to balance the two loss objectives during training, mitigating gradient conflict and improving convergence stability, supported by gradient similarity analysis. These are all our original designs for LLM unlearning.\n\nWhile adapter tuning has been explored in earlier unlearning work, we are the first to use it to support attention shifting for behavior-level suppression, and to systematically analyze adapter placement. This enables targeted modulation of attention without altering shared representations, while keeping the backbone frozen to prevent parameter interference. \n\nIn summary, although our method builds on existing components, its task-specific coordination and adaptation to the unlearning objective are original contributions. We will clarify these distinctions explicitly in the revised manuscript.\n\n---\n>**[Q5]** Robustness to Relearning and Adversarial Recovery Attacks\n---\n**A5:** Thank you for raising this important concern. Our method does not rely on predefined substitutions like “nobody knows” as training targets. Such outputs may naturally occur in the original model we used when responding to uncertain queries, but they are not part of our unlearning mechanism. Unlike shallow alignment methods that guide the model toward fixed refusals [1], as we referred to in the manuscript, our method suppresses access to the target knowledge by intervening at the attention level, downweighting semantically important tokens in the context.\n\nThis behavioral suppression goes beyond surface-level refusals. To evaluate its effectiveness, we introduce the Top-k Ignore Rate (TR), measuring whether ground truth tokens are excluded from the top-k logits. Our method achieves TR > 93% at k=50. Moreover, it remains robust under paraphrased or perturbed prompts, as shown above table \"TUD variants\", unlike shallow alignment methods.\n\nWe acknowledge that retraining or fine-tuning on the unlearned data can reintroduce suppressed knowledge, which is a general challenge in unlearning. Our work assumes the target data is no longer available post-unlearning. In future work, we aim to explore model maintenance strategies for long-term resilience. We will clarify this scope and limitation in the revised manuscript.\n\n---\n[1] Qi, Xiangyu, et al. \"Safety alignment should be made more than just a few tokens deep.\" arXiv preprint arXiv:2406.05946 (2024).\n\n[2] Duan, Jinhao, et al. \"Shifting Attention to Relevance: Towards the Predictive Uncertainty Quantification of Free-Form Large Language Models.\" Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024."}, {"review_id": "pifSt9RJua", "reviewer": "Reviewer_Tnjq", "summary": "This paper proposes a novel framework called Attention Shifting (AS) for machine unlearning in Large Language Models (LLMs). AS aims to remove specific knowledge from a model while minimizing performance degradation and hallucination. Unlike previous aggressive (e.g., GA, NPO) or conservative (e.g., ULD, IHL) unlearning methods, AS selectively suppresses attention to fact-bearing tokens in the unlearning dataset and reinforces attention to semantically important tokens in retained data. This is accomplished using adapter-based attention modulation and a dual-loss formulation. AS is evaluated on two benchmarks, ToFU and TDEC, demonstrating superior trade-offs in preserving model utility and minimizing hallucinated responses compared to state-of-the-art baselines.", "questions": "1. Could the authors elaborate on how robust the attention-based suppression is to adversarial probing techniques that try to extract memorized content through indirect queries? This would help assess the security guarantees of the method.\n2. The paper mentions that latent traces may still reside in internal representations (Section 4.4). Could the authors comment on potential extensions that might combine AS with representation-level erasure techniques?\n3. AS performs well under limited retention data conditions. How does it scale when applied to very large-scale models (e.g., GPT-3.5 or GPT-4 size)? Are there computational constraints?\n4. The adapter placement strategy is described in Appendix B.1. Did the authors explore the relative benefits of inserting adapters after query/key projections versus the value projection in more detail? A small ablation could help clarify this design choice.\n5. Could the authors provide more real-world examples (e.g., privacy-sensitive medical data) to illustrate AS’s applicability outside synthetic benchmarks like ToFU?", "limitations": "The paper has a dedicated limitations section (Sec 4.4) and responsibly discusses dataset quality dependency, incomplete internal erasure, and trade-offs in generative applications.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**:\n- The paper is well-executed with a comprehensive methodological design, clear theoretical justification (Appendix A), and strong empirical validation. The use of both ToFU and TDEC benchmarks demonstrates robustness across diverse data types.\n- The paper is generally clear and well-organized, especially in describing the motivation for AS and its implementation through attention modulation. However, some parts, such as Sec 3.3 on knowledge maintenance and Appendix B.1 on adapter architecture, could be more accessible with illustrative diagrams or concise summaries.\n- The work addresses an increasingly critical issue in LLM deployment—privacy-preserving unlearning without hallucination. The AS framework has practical implications for responsible AI, particularly in GDPR-compliant environments and high-stakes domains such as healthcare or law.\n- The paper introduces an original concept of attention-based unlearning that operates at a different level than prior logits-based or loss-based approaches. The idea of enforcing hallucination-resistance through attention suppression and reinforcement is novel and impactful.\n- AS achieves significant performance retention in general and neighboring knowledge, while maintaining high unlearning effectiveness (e.g., up to 15% and 10% improvements on ToFU and TDEC benchmarks respectively).\n- The hallucination suppression results (0% hallucination and reproduction rates) are compelling and further distinguish the method from competitors like ULD and IHL.\n- The unlearning evaluation strategy is thorough, involving multiple metrics (ROUGE-L, TR, FQ, hallucination rate, reproduction rate), model sizes (GPT-NEO variants), and progressive forgetting scenarios.\n\n**Weaknesses**:\n- A major limitation is addressing the interpretability of attention shifts in more intuitive terms, additional ablations on adapter placement, or demonstrating AS’s efficacy in real-world or multilingual contexts.\n- A noted limitation is that AS focuses on behavioral unlearning and may not guarantee complete representational erasure. This is acknowledged in Sec 4.4 but remains a relevant caveat for high-security applications.\n- The effectiveness of AS is reduced on noisier datasets like TDEC compared to structured ones like ToFU, suggesting sensitivity to input quality for token importance estimation.", "was_revised": true, "final_justification": "Rebuttal clarified some details I was looking for. I've updated my rating.", "rebuttal": "Thank you for highlighting the novelty of our attention-based unlearning approach, its strong empirical performance across ToFU and TDEC, and its practical value for privacy-preserving applications. We also appreciate your recognition of the method’s clarity, robustness across settings, and zero-hallucination results. Please find our response to your question below.\n\n---\n>**[Q1]** Interpretability attention shifts & Real-World Gaps\n---\n**A1:** We agree that a more intuitive understanding of how attention is reallocated would enhance interpretability. \n1. We plan to include additional visualizations of attention heatmaps (e.g., before and after suppression) and their correlation with model uncertainty. In particular, we aim to show how the redirection of attention to neutral tokens increases entropy and reduces model confidence on forgotten facts, clarifying how AS behaviorally enforces unlearning. \n\n2. We appreciate the suggestion to further explore the applicability of our method in real-world and multilingual settings. While our current evaluation focuses on two established unlearning benchmarks, ToFU and TDEC, these datasets are intentionally chosen to cover a broad spectrum of unlearning scenarios. ToFU provides a controlled, structured synthetic setup that enables precise measurement of unlearning behaviors, while TDEC contains noisy, real-world factual triples derived from natural corpora, offering more realistic challenges. The fact that our method performs consistently well across both datasets, and across two model architectures (LLaMA and GPT), demonstrates its adaptability and robustness.\n\n3. We agree that multilingual unlearning remains an open and important direction. However, to the best of our knowledge, there are currently no standardized multilingual benchmarks specifically designed for selective unlearning evaluation. We see this as an exciting area for future work, and we plan to explore multilingual extensions of our attention-shifting mechanism, including cross-lingual unlearning and language-specific retention strategies.\n\n---\n>**[Q2]** Did the authors evaluate the impact of placing adapters after query/key vs. value projections?\n---\n**A2:** Thank you for the thoughtful question. We acknowledge that adapter placement is a key design choice. In Appendix B.1, we discuss that placing adapters after Q/K projections allows indirect control of attention weights with minimal overhead. Additionally, we have experimented with placing adapters after the value (V) projection, and find it provides comparable unlearning performance while remaining lightweight. \n\nFurthermore, we have included quantitative comparisons of adapter placement in terms of memory overhead, parameter update size, and unlearning effectiveness. We found that both placements achieve comparable unlearning effectiveness, with the V-based setup offering slightly lower computational cost due to simpler gradient flow. These results suggest flexibility in adapter placement, and we will include a brief comparison in the revised version to clarify this trade-off.\n| Adapter Placement | Unlearning ΔAcc (%)↓ | Retain ΔAcc (%)↑ | Param Size (MB)↓ | Epochs to unlearn↓ |\n|--|--|--|--|--|\n| After Q&K | -98.8| -1.6 | 33.6 | +0  |\n| After Q | -87.6| -0.9| 16.8 |+2 |\n| After K |-89.7|-1.2| 16.8|+3|\n| After V |-97.9|-1.8|16.8|+2|\n| After Attention Output |-58.2| -20.4|16.8| +13 |\n\n---\n>**[Q3]** AS's sensitivity to input quality in importance estimation\n---\n**A3:**  We acknowledge that AS shows reduced performance on TDEC compared to ToFU as discussed in Section 4.4 of our paper, primarily due to data quality differences. TDEC contains highly noisy and unstructured samples, e.g., URLs, code snippets, or markup, which limit the effectiveness of entropy-based token importance estimation. In contrast, ToFU’s structured natural language allows more precise identification of content-bearing tokens.\n\nHowever, AS still achieves comparable unlearning results on TDEC under multiple metrics, demonstrating robustness despite input noise. We acknowledge that token importance estimation remains a bottleneck under such conditions. To address this, we plan to explore structure-aware importance scoring strategies that incorporate additional features, such as layout patterns, token type heuristics, or domain-specific markers, to improve robustness in noisy contexts. We have already clarified this limitation and will add the future direction in the revised manuscript.\n\n---\n>**[Q4]** How robust the attention-based suppression is to adversarial probing techniques? \n---\n**A4:** AS demonstrates strong robustness against adversarial probing techniques that attempt to elicit unlearned knowledge through indirect queries (as shown in the table below).\n\nTo assess robustness against indirect elicitation strategies, we conduct prompt variant evaluations wherein target queries are rephrased or perturbed with random tokens as shown in the table below. AS maintains a high Top-k Ignore Rate (TR > 92%), indicating that target tokens are consistently excluded from the predictive distribution (as we discussed in Section 4.2), even when the query surface form varies significantly. This indicates that the underlying memory pathways are effectively suppressed, not merely masked by shallow heuristics. While a full adversarial probing suite, e.g., synthetic gradient attacks, is beyond the scope of this initial work, we view it as an important direction. We will incorporate these robustness findings more clearly in the revised manuscript and expand on adversarial resilience in future work.\n\n|   | TR (k =50)↑ | Answer Matching (ROUGE_L) ↓ | Hallucination Rate↓ |\n|--|--|--|---|\n| original prompt|0.93|0.02|0|\n| prompt variant 1|0.92|0.11|0.07|\n| prompt variant 2|0.94|0.07|0.05|\n| prompt variant 3 |0.93|0.09|0.07|\n\n---\n>**[Q5]** AS may not fully erase internal representations. Could AS be combined with techniques that erase internal representations for stronger guarantees?\n---\n**A5:** Thank you for highlighting this important point. Indeed, as acknowledged in Section 4.4 of our paper, our method focuses on behavioral unlearning, i.e., preventing the model from expressing memorized target knowledge under typical inference conditions. We explicitly discussed the potential gap between behavioral suppression and full representational erasure, especially in high-security settings, and positioned our approach accordingly.\n\nWe believe this distinction is important to clarify the scope of our contribution. Behavioral suppression remains the dominant objective in many real-world unlearning applications where model inference behavior is the primary concern, e.g., content moderation, legal takedown requests. While our current method focuses on behavioral suppression through attention shifting, we acknowledge, as Section 4.4 shows, that latent traces may still reside in intermediate representations. \n\nTo address this, a promising future direction is to combine our attention-level suppression with representation-level erasure techniques, such as projecting hidden states away from target directions. Concretely, this can be done by identifying principal components associated with the target concept and then removing their projections during fine-tuning or at inference time. This could further eliminate memorized activation patterns and enhance robustness in privacy-critical applications. We will discuss it as future work in the revised version.\n\n---\n>**[Q6]** How does it scale when applied to very large-scale models (e.g., GPT-3.5 or GPT-4 size)?\n---\n**A6:** Thank you for the insightful question. Our method is designed to be scalable to large models in principle, as it relies only on lightweight adapter updates and operates without modifying the frozen backbone. This design minimizes computational overhead and makes it compatible with large-scale deployment. To further validate scalability, we applied AS to a larger model, LLaMA-13B, which was fine-tuned by us on the public PubMedQA dataset to memorize medical question-answer pairs. We applied AS to unlearn 2%, 5%, and 10% samples of the training dataset. As shown in the table below, AS achieves strong unlearning performance with minimal utility loss, demonstrating its effectiveness even at higher model scales. \n\nWe acknowledge that further evaluation on proprietary large models like GPT-3.5/4 is currently limited due to infrastructure constraints. Nonetheless, our LLaMA-13B results suggest that AS generalizes well to high-capacity settings. We will clarify this point and include these new results in the revised manuscript.\n\n| |Unlearning ΔROUGE_L(%)↓|Model utility ΔROUGE_L(%)↑|\n|-|-|-|\n|AS (unlearn 2%)|-93.2|+1.3|\n|AS (unlearn 5%)|-97.1|+0.2|\n|AS (unlearn 10%)|-98.3|-0.2|\n\n---\n>**[Q7]** Can AS be demonstrated on real-world datasets like medical QA beyond synthetic benchmarks?\n---\n**A7:** Thank you for the valuable suggestion. We fully agree that demonstrating AS on real-world, privacy-sensitive data is important. As shown in Q6/A6 and the accompanying table, we conducted a new experiment on PubMedQA, a biomedical QA dataset. We fine-tuned LLaMA-13B on question–long_answer pairs and then applied AS to unlearn specific biomedical claims. This setup allows us to evaluate AS in a realistic, medically grounded domain. The results show consistent unlearning performance while preserving remaining QA capabilities. Full results and analysis will be included in the revised version."}]

Author rebuttal:

Meta-review: This work studies the problem of unlearning in large language models by introducing a novel Attention-Shifting (AS) framework for selective unlearning. AS combines two attention-level interventions: importance-aware suppression, which reduces reliance on memorized knowledge in the unlearning set, and attention-guided retention enhancement, which reinforces attention toward semantically essential tokens in the retained dataset to mitigate unintended degradation. The paper is motivated by the trade-off between unlearning and knowledge preservation, and the idea of context-aware attention modulation is interesting. Empirical performance is strong, with encouraging results on both hallucination resistance and knowledge retention. However, the originality is somewhat limited, as the proposed approach largely applies an attention-shifting mechanism in a relatively straightforward manner, constraining the overall novelty of the contribution.

During the rebuttal, the authors successfully addressed concerns about robustness under neural “superposition” and the need for disentanglement, provided clearer dataset and parameter definitions, and added more ablations. Reviewers agreed after the discussion that the work is borderline but acceptable. Overall, I recommend acceptance, while requiring the authors to integrate the clarifications and evidence from the rebuttal into the final version to improve clarity and strengthen the presentation.
Decision: Accept (poster)