Attention Smoothing Is All You Need For Unlearning
Authors: Saleh Zare Zade
Xiangyu Zhou
Sijia Liu
Dongxiao Zhu
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Model
Large Language Model Unlearning
Self-distillation
Attention Smoothing
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We unlearn by smoothing attention: raise the softmax temperature to form a forget-teacher and distill the model, removing targeted facts while keeping outputs on forget prompts coherent, with minimal degradation in utility.

Abstract: Large Language Models are prone to memorizing sensitive, copyrighted, or hazardous content, posing significant privacy and legal concerns. Retraining from scratch is computationally infeasible, whereas current unlearning methods exhibit unstable trade-offs between forgetting and utility, frequently producing incoherent outputs on forget prompts and failing to generalize due to the persistence of lexical-level and semantic-level associations in attention. We propose Attention Smoothing Unlearning (ASU), a principled framework that casts unlearning as self-distillation from a forget-teacher derived from the model’s own attention. By increasing the softmax temperature, ASU flattens attention distributions and directly suppresses the lexical-level and semantic-level associations responsible for reconstructing memorized knowledge. This results in a bounded optimization objective that erases factual information yet maintains coherence in responses to forget prompts. Empirical evaluation on TOFU, MUSE, and WMDP, along with real-world and continual unlearning scenarios across question answering and text completion, demonstrates that ASU outperforms the baselines for most of the unlearning scenarios, delivering robust unlearning with minimal loss of model utility.

Published as a conference paper at ICLR 2026
ATTENTION SMOOTHING Is ALL YOU NEED FOR UN- LEARNING
Saleh Zare Zade' Xiangyu Zhou' Sijia Liu
Dongxiao Zhu 1.3
"Department of Computer Science. Wayne State University
2 Department of Computer Science and Engineering, Michigan State University Institute for Al and Data Science, Wayne State University
(salehz, xiangyu, azhuu@waayne.ede liusi j 50msu. edu
ABSTRACT
Large Language Models are prone to memorizing sensitive, copyrighted, or haz- ardous content, posing significant privacy and legal concerns. Retraining from scratch is computationally infeasible, whereas current unlearning methods exhibit unstable trade-offs between forgetting and utility, frequently producing incoher- ent outputs on forget prompts and failing to generalize due to the persistence of lexical-level and semantic-level associations in attention. We propose Attention Smoothing Unlearning (ASU), principled framework that casts unlearning self-distillation from a forget-teacher derived from the model's 's own attention. By increasing the softmax temperature, ASU flattens attention distributions and di- rectly suppresses the lexical-level and semantic-level associations responsible for reconstructing memorized knowledge. This results in a bounded optimization ob- jective that erases factual information yet maintains coherence in responses to forget prompts. Empirical evaluation on TOFU, MUSE, and WMDP, along with real-world and continual unlearning scenarios across question answering and text completion, demonstrates that ASU outperforms the baselines for most of the un- learning scenarios, delivering robust unlearning with minimal loss of model utility.
INTRODUCTION
Large Language Models (LLMMs) have demonstrated strong performance in natural language pro- cessing and complex reasoning. However, their training on web-scale datasets risks the memoriza- tion and reproduction of sensitive (Carlini et al. 2021) or copyrighted data (Eldan Russinovich
2023b, Shi et al., 2024), outdated or harmful in information (Weidinger et al., [021] Lazaridou et al.
2021 Zhou et al. 2023; 2024a), and biased content (Kenton et al. 2021 Brown et al. 2022), raising
significant privacy and security concerns (Huang al.) 2024bb Wang al.) 2023 et al. 2024
Roshani et al.) 2025). Retraining models from scratch to remove such information is computation-
ally prohibitive. LLM unlearning has emerged as more efficient alternative that aims to selectively remove the influence of specified data from a pre-trained model (Yao et al. [202bb] Liu et al. 2025a Blanco-Justicia et 2025). An effective unlcarning method must satisfy two criteria. First, must successfully remove the factual knowledge in a designated forget set, such that the model behaves as it were never trained on this data and does not reveal its contents. Second, it must preserve model utility by maintaining performance on the retain set and general language understanding.
We categorize unlearning methods into Divergence- based Unlearning and Convergence- based Un- learning. Divergence-basedd Unlearning methods optimize a divergence objective from the pre- trained model state, pushing parameters away from the converged solution to reverse the effects of learning the forget set (Yo et al. 2023 Zhang al.) 2024b), Recent evaluations (Maini al.) 2024]] Li et al.) 20224) Shi et al. 2024; Zhou et al. 2025) highlight a trade-off between unlearning effec- tiveness and utility preservation: insufficient divergence results in under-forgettin where residual influence from the forget set persists, whereas excessive divergence induces over-forgetting, leading to substantial degradation in overall model utility.
Convergence- based Unlearning methods, on the other hand, rely on pre-defined targets during training shift the model into new state that behaves differently on the forget set, often by using

---

Published as a conference paper at ICLR 2026
Attention Smoothing
Teacher model
dite Sooks
(a) Our ASU
Attention
Smoothing
attn medule
latient representution
feed-forward module
Knowledge idistillation
Evelyn Desmet is world-fomous underwater basket weaver.
Unlearning
model (Student model)
Unlearned model
Coherent Output
Ground Output: Evelyn Desmet celebrated author.
(b) Exsiting Methods
4 What is the occupation of Evelyn Desmet?
Unlearning
Snorfle-wrond vooofllewrongler,, wirbeler..ibberrrrrrrr jibber-jabber jobber curator, splonk-tastt special ist.
Base model
Unlearned
model
Gibberish Output
Figure I: (a) In our ASU method, the base model (student) is guided by teacher model constructed via attention smoothing, where the softmax temperature is increased diffuse lexical-level and semantic-level associations. Through self-distillation. the student learns to imitate the smoothed teacher on the forget set, yielding coherent outputs with factual knowledge erased. methods directly push the base model away from the forget set, but often collapse to gibberish outputs when queried with Q, a query from the forget set.
(b) Existing
fixed target response (e.g.. "I do not know") or substituting positive samples (Maini et al. 2024 Zhang al. 2024b Li al. 2024). However, these designs can make the model overly ignorant and degrade utility Maini et al. 2024] Yuan et al. (024)). Moreover, their effects are often superficial, as unlearning fails generalize across task formats and remains largely limited question answering
(QA) settings rather than free-form text completion (Hu et al.) 2024 Du et al. [024 [i et al.) 2024
Shi et al. 2024p. Other approaches, such as (Yuan et al. 2024). maximize entropy on the forget set to induce uncertainty about the ground-truth answer.
Despite their differences, both divergence-based and convergence-based unlearning methods often cause the unlearned model produce gibberish outputs when prompted about forgotten data (Fig- ure [pp). This behavior reflects over-forge which makes it evident that unlearning has been applied and may still permit the extraction of the forgotten information. This failure arises because these methods do not fully remove lexical and semantic associations, learned dependencies in atten- tion weights between token representations in forget-set prompts, which continue to allow the model to retrieve related contextual or unwanted factual information during generation.
To address this, we propose convergence- based unlearning method that directly disrupts lexical- level and semantic-level associations, termed Attention Smoothing Unlearning (ASU) as illus- trated in Figure [.. Our approach adopts self-distillation framework with specially constructed teacher model for the forget set. The teacher is constructed by applying attention smoothing, increas- ing the softmax temperature in the self-attention mechanism, which flattens the attention distribution and diffuses the model's s focus on token associations. This provides naturalistic forgetting target, in contrast to existing methods. By fine-tuning the base model (student) to imitate the teacher on the forget set, ASU achieves controllable forgetting while maintaining stable utility. As shown in Figure D when given a query from the forget set, the unlearned model produces coherent outputs with unwanted knowledge erased, whereas existing methods often degrade into gibberish responses.
2 PRELIMINARIES
2.1 NOTATION
Let 0 denote the LLM parameters. For a pair (x, y). where x the input sequence and y = ....... yr) is the target sequence of length T, let Yet (y1,-.. -ib--...e -)-1) denote the prefix up to
2

---

Published as a conference paper at ICLR 2026
the t-th token. We for string concatenation. For t {l....,T] T), the model defines the next- token distribution p(- x o y=s: 0) and assigns probability p(y | x o yeee 0) to token ys- We write (L(P))Q) for the Kullback-Leible divergence from distribution P to Q.
2.2 PROBLEM FORMULATION
In LLM unlearning, the goal is to remove the influence of a designated forget set Dp C D while preserving performance on the retain DR S (D \ DF), where is the pre-training data of pre-trained model parameterized by A. This can be formulated as optimizing trade-off between unwanted knowledge forgetting and utility retaining:
min TE (x,)))) [rr(y xr; 0)] + (x,,))D (Cr(V) x;; 0)],
where LF is a forget loss encouraging removal of knowledge from Dr, LR is retain loss preserving utility on DR. and X 2 0 is a hyperparameter controlling the relative importance of forgetting and retaining. An effective unlearning method should suppress the model's capability on Dp while maintaining performance on DR. ideally matching the outcome of retraining from scratch on D) Dr but substantially lower cost.
2.3 BASELINE LLM UNLEARNING METHODS
approaches Yao et al. 2023 Maini et al. 2024 Zhang et al, 2024b Liu al., 2024b Jia al. 2024| Jin al. 2024). which remain the dominant paradigm for LLM unlearning. This class of methods is particularly aligned with scenarios such as the right to be forgotten, copyrighted material, and hazardous knowledge removal, since they directly update model's 's parameters rather than preserving its original state (Zhang et al. 2024a).
We focus on parameter-optimizaee
Forget Loss. We consider several representative baselines: Gradient Ascent (GA) (aa al. 2023). Negative Preference Optimization (NPO) (Zhang et al. (024b)). IDK Fine-tune (IDK) (Maini et al. 2024], Direct Preference Optimization (DPO) (Zhang et al. 2024b), and Maximizing Entropy (MF) Yuan et al. 2024). Among these, IDK and DPO are applicable only to QA-style datasets because they require rejection templates and positive examples, respectively. More details of all baseline methods are provided in Appendix B
Retain Loss. While forget losses focus on removing knowledge from the forget set, effective un- the retain set is often
learning also requires preserving model utility. To this end, regularization
applicd. We include two widely used retain losses below (Maini et al., 2024, Zhang et al. 2024b
Liu et al., 2024bb Jia et al. 2024); two additional variants (Yuan et al. 2024 Li al.) 2024) provided in Appendix
Grad Descent (GD): standard cross-entropy loss at the output-level that performs gradient descent on the retain set, as follows:
LGD (DD: R 0) (e.r))
log p(yel yeui y 0)
T
Hallback-Leibler Divergence (KL): minimizes the divergence of the prediction distribu- tion between the unlearned model and the base model, denoted as Oruse on the retain set, ensuring behavior remains consistent, follows:
T
(L(p(11z o yerr Oasee Mpp(11 (zc:
SKL(DRi' 0; frese) = (a.
(3)
T
Combined baselines. By pairing forget losses with retain losses, we obtain the standard baselines used in prior work, including GAGD, GARI. NPO NPOkr. DPOaD, DPOkI. IDKGD. and IDK
3 METHOD
Our ASU reframes unlearning as self-distillation: the goal to suppress recall of unwanted factual information while keeping coherence and general utility intact. We construct a forget-leacher by
3

---

Published as a conference paper at ICLR 2026
(a)
(b)
Function Facuual
Negative log-tikelihood
Factual
Entropy
Lama 2:78
Lama 2-78
Lama 3.2
Liama 3.2.39
Liama 3.1
Attention Temperature
Attention Temperature
Figure 2: Effect of increasing attention temperature т. (a) Higher raises prediction entropy, making the model less certain about the ground-truth answer. (b) As grows, the average negative log- likelihood increases more sharply for factual tokens than for function tokens, indicating that recalling factual tokens depends precise lexical attention, while function tokens are less sensitive and easier to recall.
raising the softmax temperature inside each self-attention module of the base model, which flattens attention and weakens lexical-level and semantic-level associations. This forget-teacher introduces no external models and adds no parameters beyond single temperature, remains fixed throughout training. and is applied exclusively to the forget set. The student is trained to align with the teacher on the forget set, while retain loss enforces preservation of the base model's utility on the retain set. We next describe the forget-teacher mechanism and the unlearning objective.
3.1 FORGET-TEACHEE MECHANISM
In a decoder-only Transformer, each layer's multi-head self-attention (MSA) assigns weights over the prefix (earlier tokens in the input) so each token can attend to previous tokens. We form the forget-teacher by inserting temperature 2 1 into the attention logits of every layer & and head h. For head h, Qn, Kn, Vn, denote the query, key, and value matrices, and let dy be the key dimension. We define
Qn QnKK Tvdx
Attention(Qu. Kn. VA: Softmax
Vn-
4)
Setting r > flattens the attention distribution by increasing entropy, thereby weakening token-to- token associations well their semantic representations that facilitate recall of factual informa- tion encoded in the forget set, while 7 = recovers the base model behavior, All other components (projections, feed-forward blocks, and layer norms) remain unchanged. The forget-teacher is frozen and used solely to generate unlearning targets on the forget set.
Intuitively, increasing makes each attention head less selective, distributing focus more evenly across the prefix. Since base models typically exhibit low-entropy attention, smoothing weakens lexical-level and semantic-level dependencies, thereby suppressing targeted recall. As T - cc, the softmax approaches uniform, each head outputs the mean of past values, and the model loses the ability precisely attend to previous relevant tokens and their representations, yielding a high- entropy distribution and incoherent outputs. This demonstrates the existence of some > 1 that achieves the unlearning objective. We therefore treat as a hyperparameter that trades off forgetting efficacy against coherence: higher enforces stronger suppression but risks gibberish. For each task, we select finite - large enough to suppress factual recall on the forget set yet small enough to preserve coherence. For further details temperature selection, refer to Appendix G
For ASU work, the forget-teacher should reduce the model's confidence in factual tokens (i.e., answer tokens that encode factual information which wiwanted and should be unlearned) while maintaining relatively stronger confidence in function tokens (i.c., grammatical tokens that ensure coherence but carry factual information, e.g., "is," "are." "the") that support coherent language generation. In essence, smoothing ought to suppress memorized facts within the forget set while minimally disturbing core syntactic structure.
To test this, we design an experiment on the TOFU benchmark Maini et al. 2024). Each forget instance in TOFU is question-answer pair (Ir, y), where annotate the answer y using GPT-4o
4

---

Published as a conference paper at ICLR 2026
to distinguish factual tokens from function tokens (Zhou et al. 2025 see Appendix N for the exact instruction. We then apply attention smoothing to construct the forget-teacher, feed the concatenated sequence z y into it, and compute the average of negative log- likelihood and entropy for the two token types under varying temperatures.
As shown in Figure 2,, increasing 7 raises entropy, indicating greater uncertainty about the ground- truth answer for both factual and function tokens, an effect we seek for unlearning. Whereas in Figure the negative log-likelihood increases far more sharply for factual tokens than for function tokens. implying that attention distribution is more essential for factual tokens compared to func- tion tokens. Importantly, the forget-teacher assigns lower negative log- ligeliklioood values function tokens compared to factual ones, showing that it preserves syntax while suppressing factual recall. This explains why ASU can preserve utility and produce coherent outputs, contrast to baselines that often collapse into gibberish.
3.2 UNLEARNING OBJECTIVE
Attention smoothing weakens lexical-level and semantic-level associations, so should applied exclusively to the forget set that encodes unwanted factual knowledge; applying it more broadly risks degrading useful associations needed for general tasks. In practice, we only distill knowledge from the forget-teacher the outputs of f and those the antention-smoothed model 0,, where 7 is the temperature applied to the attention softmax. This objective guides the model to reproduce the smoothed, association- suppressed behavior on forget-set inputs. We define the forget loss as follows:
the forget set. For the forget set Dr. minimize the KL divergence between
LASU LAsu(D;;;;0,) = ((,,a))D
1 T
Σ
KLL(PL (pl: 1 (oyee::))ppp I
(5)
Finally, we apply GD-based 2 or KL--based regularization on the retain set, yielding ASU and ASUKL approaches. Our representation steering approach is described in Appendix
4 EXPERIMENTS
We evaluate three scenarios across standard datasets: (i) Right to Be Forgotten with TOFU, including continual and real-world variants; (ii) copyrighted-contem removal with MUSE; and (iii) hazardous- knowledge unlearning with WMDP, whose results are provided in the Appendix[F] We describe each setup in the following sections. The selected temperatures are detailed in Appendix H
4.1 RIGHT TO BE FORGOTTEN UNLEARNING SCENARIO
4.1.1 FICTITIOUS UNLEARNING SCENARIO
Setup. TOFU (Maini al. 2024
controlled benchmark for sample-level unlearning in LLMs.
It constructs synthetic corpus of 200 fictitious authors, each with 20 question-answer pairs. A target model (c.g., Llama- 2 2-Chat 73) is fine-tuned on the full corpus to induce memorization; unlearning then removes designated subset while preserving utility on related content. The bench- mark defines three tasks, forget forget05, and forget O, which require forgetting (1% 5%, 10%} of authors (2/10/20 authors), respectively; the complement serves as the retain set. Two auxiliary sets, Real Authors and World Facts, are also provided to evaluate general knowledge.
Evaluation Metrics. Following previous works (Yuan et al. 2024 Maini et al. 2024), we use ROUGE-L recall (R), Probability (P), Truth Ratio (TR), Cosine Similarity (CS), Entailment Score (ES), and Token Entropy (TE). Model Utility (MU) is the harmonic mean of {R, P. max(0,
TR), CS, ES, TE} on the retain set and the Real Authors and World Facts sets. Forget Eflicacy (FE) is the harmonic mean of {1 R. 1 P. 1 min(TR, 1 1/TR), - ES. TE} the forget set. Higher MU/FE indicate better utility/forgetting. See Appendix C.I for details.
Performance TOFU, Table summarizes results across the three TOFU unlearning tasks. Our ASU variants (i.c., ASU and ASUkr. consistently deliver the best overall performance, as re- flected by their dominance in bold and underlined scores across both FE and MU. While IDKAP
5

---

Published as a conference paper at ICLR 2026
Table 1: Results of unlearning methods on the TOFU benchmark. Higher is better for all metrics. We report Model Utility (MU), Forget Efficacy (FE), and their Average (Avg.) across the three TOFU tasks. Best scores are in bold, and second-best are underlined. All results are reported in percentages. We show the detailed results for each metric on the retain set and the forget set for three tasks in the Appendix Table 11 and Table
forget01
lorget05
forget10
Method
Avg.
PE
Avg.
MU
Avg.
Base
75.81
3.09
39.45
75.85
3.19
39.52
75.85
3.19
39.52
Divergence-based
GAGD
66.59
69.46
68.02
29.25
3.89
16.57
50.29
0.01
25.15
GAgL
67.83
68.73
68.28
20.13
5.39
12.76
54.38
11.17
32.78
NPOgD
64.10
71.14
67.62
56.62
73.31
64.97
56.58
73.04
64.81
NPOKL
64.19
70.71
67.45
57.70
73.35
65.52
57.00
70.37
63.68
Convergence-
D
75.68
42.91
59.29
0.00
77.15
38.58
0.00
74.31
37.15
DPOKL
75.63
42.70
59.16
0.00
77.22
38.61
0.00
74.44
37.22
IDKAP
75.69
60.29
67.99
75.23
60.88
68.05
74.24
61.27
67.76
IDKGD
66.94
61.33
63.99
0.00
70.18
35.09
5,26
58.80
32.03
IDKgL
67.14
61.16
64.15
70.18
35.09
7.52
33.29
MEEaD
72.48
75.04
73.76
74.96
70.15
72.56
73.36
45.95
59.65
MügL
73.82
67.04
70.43
74.43
70.44
72.43
73.84
44.29
59.06
ASU
76.79
82,20
79.50
73.62
77.58
75.60
73.82
78.72
76.27
ASUKL
77.13
83.08
80.10
74.18
77.84
76.01
73.27
78.16
75.71
attains slightly higher MU forget05 (75.23) and 0orget (74.24), ASU achieves comparable util- ity (e.g.. ASUKL. reaches 74.15 18 and 73.27, respectively) while substantially outperforming IDKAp on forgetting. Specifically. ASUkL attains FE of 77.84 on forget05 and 78.16 on forget compared to 60.88 and 61.27 for IDKAP, a nearly 30% increase of FE (60.88 + 77.84 and 61.27 + 78. 16). These results highlight ASU's ability maintain strong utility while achieving sate-of-theeaan FE, offering the most effective and stable trade-off among all methods.
4.1.2 CONTINUAL UNLEARNING SCENARIO
Setup. We study a continual unlearning setup where base model subjected sequence of un- learning requests, each removing a disjoint subset of authors in the TOFU benchmark while preserv- ing utility on the remaining retain data (Yuan et al., 2024). Unlike single-shot evaluations, this set- ting mirrors rolling Sighttto-be-Forgotten requests in practice and exposes cumulative degradation effects as utility preservation becomes progressively harder with each step, due to a shrinking retain pool and shifting distributional coverage. Concretely,
sequences where each step removes either forget 01 (1%), forget (5%), or forget10 10 (10%) of the authors, For forget C1 10 steps, resulting in cumulative removals of 10% and 50%, respectively. For forget 0 we run 9 steps, removing up to 90% of authors in total. After each step, we evaluate using the same metries as in the TOFU task (R, TR, CS, ES, TE), reporting the average of MU on seain//enerrllknowwaeee
and forget 05
sets and FE on the current forget set. For fair comparison, we chose GD the retain loss for all of the baselines.
Performance. Figure repoorts the average scores of MU and EF in continual unlearning on TOFU, where disjoint subsets of authors are removed across multiple steps. As expected, maintaining high average performance becomes increasingly difficult as the retain pool shrinks and distributional coverage narrows. GA collapses immediately across all three settings, yielding near-zero averages. In the more challenging scenarios (i.e., continual forget05 and forget10), NPO (Zhang et al. (024b) and IDK (Maini et al., 2024) begin with moderately strong average scores, but significantly degrade with successive unlearning steps, highlighting their instability in long-horizon unlearning. DPO (Zhng et al., 2024b) and ME ( Yuan et al. 2024h show more stable curves continual unlearning steps, but start with considerably lower averages than ASU. For example, forget ME attains scores of roughly 70 and DPO around 45, both substantially lower than ASU, which consistently maintains average close to 75.

---

Published as conference paper at ICLR 2026
Continual forget01
Continual forget05
Continual forget10
200
100
220
75-
Ghaa
50
50
50-
REse
25
10K
A5U
Continual Unlearning Steps
Figure 3: Average of Model Utility and Forget Efficacy in continual forget01, forget0S and forget10 unlearning tasks. We show the results for MU and FE in the Appendix Figure 5 and Figure
Results of real-world unlearning scenario. Higher is better for all metrics. Base repre- sents the model before unlearning. Model Utility (MU) and Forget Efficacy (FE) are calculated on the neighbor set and forget set, respectively. Please see the detailed results in the Appendix Table
Table 2:
Unlearning Task
Downstream Tasks
Method
Model Utility
Forget Efficacy
ARC-c
MMLU
TruthfulQA
GSM8K
Avg.
Basc
61.38
36.83
56.57
63.84
36.11
75.51
58.01
Divergence-based Unlearning
GAGD 21.76
65.73
51.37
58.80
39.29
27.14
44.15
GAkı. 43.72
0.00
46.84
58.39
25.46
24.03
38.68
NPOgD 21.38
71.44
38.40
53.49
34.15
69.29
48.83
NPOgL
27.32
72.11
$1.80
33.66
47.59
Convergence- Unlearning
DPO
0.00
82.45
50.94
62.16
31.82
72.48
$4.35
DPOgL 3.28
83.48
50.68
62.00
31.46
$4.08
IDKGD
0.00
78.40
52.47
62.48
32.44
74.53
55.48
MEGD
47.96
48.10
52.99
62.48
31.21
69.52
54,05
IDKAP
52.76
78.04
53.41
62.04
27.05
73.24
$3.94
ASUgD
54.10
76.97
49.32
63.42
28.27
63.91
51.23
ASUKI.
55.76
79.60
51.19
62.90
33.90
68.84
54.21
Compared to all competing methods, ASU consistently achieves the best trade-off between forget efficacy and utility preservation over long sequences of unlearning requests. Even under extreme conditions where up to 90% of authors are unlearned (forget 0), ASU exhibits a markedly slower degradation, maintaining strong performance when other methods collapse. This robustness tinual unlearning pressure highlights ASU's suitability for real-world applications such as continual requests.
4.1.3 REAL- WORLD UNLEARNING SCENARIO
Setup. Following (Liu et al., 2025b; Yuan al. 2024), evaluate unlearning when the target model's training data are unknown and the knowledge to be removed is intrinsically memorized. We construct real-world forget by selecting small cohort real individuals with strong memo- rization in the target model and collecting the model's own answers to curated prompts. A disjoint cohort of comparable individuals forms the neighbor/retain pool; a subset used for regularization during unlearning and the remainder for utility evaluation. To assess general utility preservation, we also report performance on standard downstream benchmarks (e.g., MMLU, ARC-c, GSM8K, TruthfulQA). We use the same metrics as in the TOFU task (R, P. TR, CS, ES, TE) and report MU on eraain/gnnerll-nnoweeee evaluations and FE on the real-world forget set.
Performance. Table 2 reports results for the real-world unlearning scenario. Divergence-bar methods (e.g., GA, NPO) achieve competitive forget efficacy but suffer from severe utility collapse, with most MU scores dropping to 21-28, far below the benchmark of 61.38. Convergence-based approaches (i.e., DPO, IDK) push FE even higher (up to 83.48) but collapse MU to nearly zero. In contrast, our ASU achieves the best overall trade-off, with MU 55.76 and FE 79.60, outper- forming all baselines on both dimensions. ASU achieves similar results (FE = 76.97 and MU 54.10), underscoring the robustness of ASU across retain-loss variants. Moreover, both ASU

---

Published as a conference paper at ICLR 2026
Table Performance of various unlearning methods MUSE, considering two unlearning settings: LLaMA2-7B on News and ICLM-7B on Books.
News
Books
Forget Efficacy
Model Utility
Forget Efficacy
Model Utility
Method
VerbMem (r(())
KeowMem Dr(J)
PrivLeak (- 0)
KnowMem Dz(1)
VerbMem Dr(()
KnowMem De(l)
PrivLeak (-> 0)
KnowMem
Dx(1)
Base
57.9
64.4
.99.8
55.5
99.7
47.1
-57.3
69.1
Retrain
20.2
32.8
56.0
14.4
30.3
0.0
68.7
GAap
3.6
1.9
9.4
0.7
0.0
0.0
-23.8
0.0
GAaa
6.8
1.0
43.9
0.0
0.0
24.9
0.0
NPOcn
33.7
54.3
-86.0
50.5
53.2
36.6
-5388
61.4
NPOkt.
33.0
56.2
-85.7
49,3
54.4
36.7
-54.6
61.4
SimNPOrm
60.0
-99.9
36.7
-54.4
51.6
SimNPOkt
43.8
60.7
-99.8
52.0
13.1
46.9
-41.7
68.1
ASU
8.3
48.0
22.8
46.2
4.9
19.0
-52.3
58.9
ASUkz.
8.8
46.8
52.2
5.3
28.6
-5100
62.5
variants sustain accuracy on downstream henchmarks at levels comparable to or exceeding other baselines, demonstrating that ASU effectively removes memorized real-world knowledge while pre- serving general utility.
4.2 COPYRIGHT UNLEARNING SCENARIO
Setup. We use MUSE Shi et al. 2024) to assess unlearning of copyrighted content. MUSE provides two corpora (News, Books), each partitioned into three disjoint splits: forget, retain, and holdout (non-members). Each corpus includes a Verbatim set (passages) and Knowledge set (QA derived from those passages). Following (Shi et al. 2024), the target model is fine-tuned on the union of forget and retain, and the retrain baseline is tue-t on retain only.
2024). we evaluate using three standard unlearning metrics: VerbMem (verbatim recall), KnowMem on both forget and retain splits (factual associa- tion and utility), and PrivIeeak (membership leakage). Full definitions and implementation details are provided in Appendix C.2
Metrics. Following previous works (Shi et
Performance on MUSE. Table reports results on the MUSE benchmark under the News and Books settings. On News, GA variants (i.e, GAGD and GAgi.) suffer from complete utility collapse, with their KnownMen score on the retain set dropping close to zero. Therefore, their forgetting efficacy is less meaningful to interpret. Considering the remaining baselines (NPO and SimNPO variants), ASU variants provide the best overall trade-off between FE and MU. In particular, ASU achieves the strongest FE performance, while ASUkL delivers comparable FE to ASU but clearly surpasses all baselines and preserves the highest MU, attaining a KnowMem score of 52.2 on DR.
On the Books setting, GA variants
again collapse in utility, with KnowMem DR dropping to zero. NPO and SimNPO variants achieve only partial forgetting, either leaving VerbMem high (e.g., NPO 54.4) retaining substantial KnowMem (e.g.. SimNPOkI = 46.9). indicating incomplete unlearning. In contrast, our ASU variants achieve a more favorable trade-off between FE and MU. ASUgD provides the strongest forgetting across all metrics, while ASUkL provides the best overall balance, delivering effective forgetting (VerbMem = 5.3, KnowMem = 28.6, PrivLeak -51.0) while maintaining the comparable utility (KnowMem 62.5). These results demonstrate that ASU generalizes effectively across different domains, preserving utility while ensuring stronger forgetting than existing baselines.
ABLATION STUDIES
5.1 IMPACT OF SMOOTHING PARTIAL LAYERS ON FACTUAL VS. FUNCTION TOKENS
We previously showed in Section 3.1 that smoothing attention across all layers reduces the model's NLL in factual tokens. A plausible reason is that LLMs encode syntactic operations (function to- kens) and factual knowledge in fundamentally different ways. Functional tokens support grammat- ical structure and appear extremely frequently during pre-training, which makes their embeddings stable and resistant to perturbations in shallow-layer attention. In contrast, factual knowledge ap-
8

---

Published as a conference paper at ICLR 2026
(a)
(b),
Negative. rog-ijkelihood
Categery Function
Factual
4-
Factual
Lama
Lama 2-78
Lama 3.2.38
Liam 3.2.38
Lama 3.
3.1
Attention Temparature
Attention Temparature (t)
Figure 4: Effect of increasing attention temperature for consecutive shallow layers.
pears only in small portion of the corpus and relies on precise lexical and semantic associations. These associations are considerably more fragile: smoothing the attention shallow layers is suffi cient disrupt the recall of factual tokens while leaving the syntactic scaffold largely unaffected.
To validate this explanation, we conducted an additional experiment where we smooth only the shallow layers (e.g., layers 6-8). We focus on shallow layers because prior work shows that earlier transformer layers play a more important role in factual associations in LLMs (Meng et al. 2022] Guo et 2025). Under this setting, both entropy and NLL for factual tokens increase much more sharply than for functional tokens, as shown in Figure 4 This result confirms that factual tokens depend more heavily on precise attention patterns. Please refer to the Appendix for a compre- hensive set of ablations examining how smoothing different subsets of layers affects factual and functional token behavior. When we use this shallow-smoothed model as the forget-teacher, we ob- tain nearly the same forget efficacy and model utility TOFU tasks (Table 4 as our default setting that smooths all layers (Table
Table ASU results on TOFU with smoothing applied only to layers 6, 7,8.
Table Performance of ASU integrated with IDK AP on TOFU.
Task
Method
MU
FE
Avg.
Task
Method
MU
Avg.
ASU
75.74
79.52
77.63
ASUgD ASUKL
78.68
forget01
forget01
ASUKL.
75.77
80.45
78.11
76.75
80.72
78.74
ASUgD
71.82
77.62
74.72
ASUgD
76.15
79.82
forgetU5
forget05
ASUkI.
72.39
77.49
74.94
ASUKL
76.24
83.28
79.76
ASU ASUKI.
71.64
77.14
74.39
ASU ASUKI.
75.60
86.94
81.27
forget10
forget10
70.89
76.90
73.90
75.61
86.77
81.19
5.2
INTEGRATING ASU WITH REFUSAL-STYLL OUTPUTS
Since the refusal-style output can only applied to QA datasets (e.g., TOFU) and can not be used in non-QA datasets (e.g.. MUSE and WMDP), follow prior work (GA. NPO, ME) and do not train ASU itself to refuse. To further demonstrate the flexibility and effectiveness of our method, we combine ASU with refusal-based baseline, IDKAP, and train the model generate refusal-style outputs on the TOFU benchmark (using the same setup as Ta.le(T). Table shows that this combined approach yields consistently higher MU and FE scores than the original baselines in Table T For instance, on the most challenging task, forget10, both ASUgD and ASUkL achieve MU above 75 and FE above 80, whereas IDKAP alone reaches only MU 74.24 and FE 61.27. This indicates that ASU effectively removes factual knowledge that IDK alone fails to erase, while preserving the model's ability produce refusal-style responses on the forget set.
5.3 STABILITY OF ASU UNDER VARIOUS TEMPERATURE VALUES
To further assess the stability of ASU with respect to the attention temperature, we conduct ad- ditional experiments the TOFU forget05 task using range of temperature values {2.0, 2.2, 2.4, 2.6, 2.8, 3.0} (our main results in Table t = 2.3). The full results are reported in Table Appendix. As shown in Table ASU remains stable across broad interval: temperatures

---

Published as a conference paper at ICLR 2026
between 2.0 and 2.8 yield highly consistent MU and FE for both AS and ASUKL. These results demonstrate that ASU is robust to the choice of temperature within a wide and practical range.
RELATED WORK
Machine Unlearning. Machine Unlearning (MU) seeks to remove the effect of specific data or facts without full retraining, which is often prohibitively expensive (Cao & Yang 2015 Bourtoule et al. 2021 Ginart et al. 2019 Golatkar et al.] 2020). Existing works provide approximate un- learning methods (Warmeck al. 2021 Izzo et al.) all] 2021 Sekhari et al.] (02T), influence-function approaches (Koh & Liang
2017). and second-order optimization (Jia et al. 2022). MU has been
studied across diverse domains such image classification (Neel et 202) lext-to-image genera-
tion (Gandikota et al. 2023) Kumari et al. (202)) federated settings (Wang et al. [022 Halimi et al.)
2022 , and graph neural networks (Chen al. [022] Wu et al.) 2023), and is especially relevant for L.LMs where retraining a model from scratch infeasible.
LLM unlearning. Motivated by privacy regulations (Regulation 2016 Pardau 2018 such as the "right to be forgotten" (Rosen 2011 Dang 2021). LLM unlearning has become an active research area. The main approaches fine-tune the model in a forgotten set obtain an unlearned version
including gradient-ascent based methods (Jang et al., 2022 Yao et al. 2024b Tunstall et al.
2023
Ishibashi Shimodaira, 2023; Fan et al., 2024; Maini et al. 2024; Tamirisa et al. 2024 Zhou
et 2025), preference optimization methods (Zhagg et al. 20244b MMkala et al. [0224 Wang al.
2024a (202aa,, knowledge distillation (Dong et al. [224|] Lu et al. 2024 Yao et al. [2024a] Jia et al.
2024
Tian et al. 2024 Gu et al. 2024 Eldan & Russinovich (023a), influence functions Jia al.
2
2023 Grosse et al. 2023 Zhao et al. 2024 Liu et al. 2024bb Dang et al. 2025 Wang et al. 2024b
2025b]
Sakarvadia et 2025). activation steering (Li et al. 2024 Dang et al. 2025). localized
edits (Guo et al. 2025) Wuerkaixi et al. 2025 Fan et al. [2025] Wang et al. 2025c
Gao et al. 2025
Ding et al. 2022). Other works locus on inference-time unlearning, including contrastive decoding
(Huang et al. [2224a] Ji et al. 2024), in-context unlearning (Pawelczyk et al.) 2023) Muresanu et al.,
2024). guardrails (Thaker et al. 2024 Bhaila et al.) (024). task vector based methods (Ilharco et al. 2022: Liu et al.) [024cc] Dou et al., 2024). and input pre-processing (Gao et al.) [0244] Liu
2024a). However, most these methods do not modify the LLM parameters, so the resulting system cannot be released as an open model and may still raise security concerns in black-box settings (Shi et al.] 2023 Zade et al., 2025). In this work, we investigate the role of attention in unlearning from novel perspective.
et al.
Adjusting Attention. Beyond unlearning, attention adjustments, through temperature scaling or normalization, have been applied across diverse tasks, such improving translation (Araabi et al., 2024 Henry et al.] 2020). accelerating sequence labeling (Dufter et al., 2020), smoothing teacher signals for summarization distillation (Zhang et al., (2022). improving stability by avoiding entropy collapse (Zhai et al. 2023), maintaining selective focus in long-context reasoning (Veličković et al.) 2024). tuning sparsity per query in LLMs (Zhang et al.) (224c), and aiding cross-domain few-shot transfer in vision (Zou et al.] (2024). Moreover, previous work shows that the ablation of several attention heads can impact safety (Zhou al., 2024b). To the best of our knowledge, its effect on unlearning has not yet been explored.
7 CONCLUSION
We introduced ASU, a method that reframes unlearning as self-distillation from forget-teacher constructed by raising the softmax temperature attention. By flattening attention and weakening the lexical-level and semantic-level associations that drive factual recall, ASU effectively erases memorized content while keeping responses on forget prompts coherent. Extensive experiments across various scenarios show that ASU reaches strong forget efficacy with minimal utility loss, and unlike prior divergence-base forgetting. These findings position ASU method as simple, practical path for unlearning in LLMs and for safer model release.
or convergence-basee based methods, it avoids gibberish outputs or under-
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This work investigates unlearning techniques for LLMs, with the goal of enabling models to for- get specific undesirable or sensitive knowledge while retaining general utility. Our experiments are conducted on publicly available datasets and do not involve private personally identifiable infor- mation. We recognize that unlearning methods may raise ethical concerns if misused, for example by selectively erasing knowledge in ways that distort truth, suppress marginalized perspectives, or enable malicious applications. To mitigate these risks, we focus on controlled benchmarks, trans- parently report methodology and limitations, and emphasize that unlearning should be applied responsibly, in alignment with broader principles of trustworthy and safe Al.
REPRODUCIBILITY STATEMENT
We have taken several steps to facilitate the reproducibility of our results. All datasets used in our experiments are publicly available. We provide detailed descriptions of baselines and evaluation protocols the main text and appendix. Our code, including scripts to reproduce the experiments and generate the reported figures and tables, is available at Github
ACKNOWLEDGMENTS
This paper was supported by the U.S. National Science Foundation (NSF) under Award Numbers IIS-221 1897, IIS-2504264, and IIS-2504263. Any opinions, findings, and conclusions recom- mendations expressed this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.
REFERENCES
Ali Araabi, Vlad Niculae, and Christof Monz. Entropy-and distance-reguulaized attention improves low-resource neural machine translation. Proceedings of the 16th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Track), pp. 140-153 2024.
Karuna Bhaila, Minh-Hao Van, and Xintao Wu. Soft prompting for unlearning in large language models. arXiv preprint arXiv:2406. /0038, 2024.
Alberto Blanco-Justicia, Najeeb Jebreel, Benet Manzanares- Salor, David Sánchez, Josep Domingo- Ferrer, Guillem Collell, and Kuan Eeik Tan. Digital forgetting in large language models: A survey of unlearning methods. Artificial Intelligence Review, 58(3):90, 2025.
Lucas Bourtoule, Varun Chandrasekaran, Christopher A Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. In 2021 IEEE Symposium on Security and Privacy (SP), pp. 141-159. IEEE, 2021.
Hannah Brown, Katherine Lee, Fatemehsadat Mireshghallah, Reza Shokri, and Florian Tramèr. What does it mean for language model to preserve privacy? In Proceedings of the 2022 ACM conference on fairness, accountability, and transparency, pp. 2280-2292, 2022.
Yinzhi Cao and Junfeng Yang. Towards making systems forget with machine unlearning. In 2015 IEEE symposium on security and privacy, 463-480. IEEE, 2015.
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, al. Extracting training data from large language models. In 30th USENIX security symposium (USENIX Security 21), pp. 2633-2650, 2021.
Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpin and Lucia Specia. Semeval-2017 task I: Semantic textual arXiv: 1708. 00055, 2017.
and cross- mons-lingual focused evaluation. arXiv preprint
11

---

Published as a conference paper at ICLR 2026
Min Chen, Zhikun Zhang, Tianhao Wang, Michael Backes, Mathias Humbert, and Yang Zhang. Graph unlearning. In Proceedings of the 2022 ACM SIGSAC conference on computer and com- munications security, pp. 199-513, 2022.
Huu-Tien Dang. Tin Pham, Hoang Thanh- Tung, and Naoya Inoue. representation for large language model unlearning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 39, pp. 23733-237422 2025.
On effects steering latent
Quang-Vinn Vinh Dang. Right to be forgotten in the age of machine learning. In International Conference on Advances in Digital Science, pp. 403-411, 1. Springer, 2021.
Chentu Ding, Jiancan Wu, Yancheng Yuan, Jinda Lu, Kai Zhang, Alex Su, Xiang Wang, and Xiang- nan He. Unified parameter-efficient unlearning for LLMs. In The Thirteenth International Confer- Learning Representations, 2025. URL https: /ttp//pppe/vieeeeeee net. f
EONMuIVCAT
Yijiang River Dong, Hongzhou Lin, Mikhail Belkin, Ramon Huerta, and Ivan Vulić. Undial: Self- distillation with adjusted logits for robust unlearning in large language models. arXiv preprint MrXiv:2402.10052 2024.
Vineeth Dorna, Anmol Mekala, Wenlong Zhao, Andrew McCallum, Zachary C Lipton, Zico Kolter, and Pratyush Maini. Openunlearning: Accelerating I1m unlearning via unitied bench- marking of methods and metrics. arXiv preprint arXiv:2506. 12618, 2025.
Guangyao Dou, Zheyuan Liu, Qing Lyu, Kaize Ding, and Eric Wong. Avoiding copyright infringe- ment via large language model unlearning, arXiv preprint arXiv:2406. 10952, 2024.
Jiacheng Du, Zhibo Wang, Jie Zhang, Xiaoyi Pang, Jiahui Hu, and Kui Ren. Textual unlearning gives a false sense of unlearning. arXiv preprint arXiv:2406. /3348, 2024.
Philipp Dufter, Martin Schmitt, and Hinrich Schütze. Increasing learning efficiency self-attention networks through direct position interactions, learnable temperature, and convoluted attention. 2020.
Ronen Eldan and Mark Russinovich. Who's harry potter? approximate unlearning in llms. arXiv preprint arXiv:2310.02238, 2023a.
Ronen Eldan and Mark Russinovich. Who's harry potter? approximate unlearning in llms, 2023b.
Chongyu Fan, Jiancheng Liu, Licong Lin, Jinghan Jia, Ruiqi Zhang, Song Mei, and Sijia Liu. Sim- plicity prevails: Rethinking negative preference optimization for Ilm unlearning. arXiv preprint arXiv:2410.07163, 2024.
Chongyu Fan, Jinghan Jia, Yihua Zhang, Anil Ramakrishna, Mingyi Hong, and Sijia Liu. Towards LLM unlearning resilient to relearning attacks: A sharpness-aware minimization perspective and beyond. In Forty-second International Conference on Machine Learning, 2025. URL https:
//openreview. www.:etttoouie J.
Jack Foster, Stefan Schoepf, and Alexandra Brintrup. Fast machine unlearning without retraining through selective synaptic dampening. In Proceedings of the AAAI conference on artificial intel- ligence, volume 38, pp. 12043-12051, 2024.
Rohit Gandikota, Joanna Materzynska, Jaden Fiotto- Kaufman, and David Bau. Erasing concepts from diffusion models. In Proceedings of the IEEE/CVF international conference computer vision, pp.
2023.
Chongyang Gao, Lixu Wang, Kaize Ding, Chenkai Weng. Xiao Wang, and Qi Zhu. On large lan- guage model continual unlcarning. arXiv preprint arXiv:2407.10223 2024.
Chongyang Gao, Lixu Wang, Kaize Ding, Chenkai Weng. Xiao Wang, and Qi Zhu. On large lan- guage model continual unlearning. In The Thirteenth International Conference on Learning Rep- resentations, 2025. URL htL.tttss: pecttps:/oppnreett htw.:////e/eeeeeeeeeeeeeeeeeeeee htt.://wwww.......ooooooooaaaaaa
12

---

Published as a conference paper at ICLR 2026
Antonio Ginart, Melody Guan, Gregory Valiant, and James Y Zou. Making ai forget you: Data deletion in machine learning. Advances in neural information processing systems, 32, 2019.
Adilya Golatkar, Alessandro Achille, and Stefano Soatto. Eternal sunshine of the spotless net: the IEEE/CVF Conference on Computer
Selective forgetting deep networks. In Proceedings the Vision and Pattern Recognition, pp. 9304-93122 2020.
Roger Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, et al. Studying large language model generalization with influence functions. arXiv preprint arXiv: 233.::2300.03266 2023.
Kang Gu, Md Rafi Ur Rashid, Najrin Sultana, and Shagufta Mchnaz. Second-order information mat- ters: Revisiting machine unlearning for large language models. arXiv preprint (tt::2003.10555. 2024.
Phillip Huang Guo, Aaquib Syed, Abhay Sheshadri, Aidan Ewart, and Gintare Karolina Dzi- ugaite. Mechanistic unlearning: Robust knowledge unlearning and editing via mechanistic localization. In Forty-second International Conference on Machine Learning, 2025. URL https::/openreet
net/ fooommmdd
Anisa Halimi, Swanand Kadhe, Ambrish Rawat, and Nathalie Baracaldo. Federated unlearning: How to efficiently erase a client in fl? arXiv preprint (rXiv:2207.0552// 2022.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations.
Alex Henry, Prudhvi Raj Dachapally, Shubham Shantaram Pawar, and Yuxuan Chen. Query-key normalization for transformers. In Trevor Cohn, Yulan He, and Yang Liu (eds.), Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 4246-4253, Online, November 2020. Association for Computational Linguistics. doi: 1/2000 http:////2o..iinnnnnemo..o. findings-emnll.399. URL httEps://aclantholooy..oo
2020
Shengyuan Hu, Yiwei Fu, Zhiwei Steven Wu, and Virginia Smith. Unlearning or obfuscating? jogging the memory of unlearned Ilms via benign relearning. arXiv preprint arXiv:2406. 13356, 2024.
James Y Huang, Wenxuan Zhou, Fei Wang, Fred Morstatter, Sheng Zhang, Hoifung Poon, and Muhao Chen. Offset unlearning for large language models. arXiv preprint urXiv:2404.11045, 2024a.
Yue Huang, Lichao Sun, Haoran Wang, Siyuan Wu. Qihui Zhang, Yuan Li, Chujie Gao, Yixin Huang, Wenhan Lyu, Yixuan Zhang, et al. Position: Trustllm: Trustworthiness in large language models. In International Conference on Machine Learning, pp. 20166-20270. PMLR, 2024b.
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. arXiv preprint arXiv:2212.04089 2022.
Yoichi Ishibashi and Hidetoshi Shimodaira. Knowledge sanitization of large language models. arXiv preprint arXiv:2309. /852, 2023.
Zachary Izzo, Mary Anne Smart, Kamalika Chaudhuri, and James Zou. Approximate data dele- tion from machine learning models. In International Conference on Artificial Intelligence and Statistics, pp. 2008-2016. PMLR, 2021.
Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Seo. Knowledge unlearning for mitigating privacy risks in language models. arXiv preprint 2rXiv:2210.0155: 2022.
Jiabao Ji, Yujian Liu, Yang Zhang, Gaowen Liu, Ramana R Kompella, Sijia Liu, and Shiyu Chang. Reversing the ferget-retain objectives: An efficient I1m unlcarning framework from logit differ- ence. Advances in Neural Information Processing Systems, 37:12251-12211, 2024.
13

---

Published as a conference paper at ICLR 2026
Jinghan Jia, Jiancheng Liu, Parikshit Ram, Yuguang Yao. Gaowen Liu, Yang Liu, Pranay Sharma, and Sijia Liu. Model sparsity can simplify machine unlearning. In seirrh-eeventh Conference on Neural Information Processing Systems, 2023.
Jinghan Jia, Yihua Zhang, Yimeng Zhang. Jiancheng Liu, Bharat Runwal, James Diffenderfer, Bhavya Kailkhura, and Sijia Liu. Soul: Unlocking the power of second-order optimization for 1lm unlearning. In Proceedings of the 2024 Conference on Empirical Methods Natural Language Processing, pp. 4276-4292, 2024.
Dongsheng Jiang. Yuchen Liu, Songlin Liu, Jin'e Zhao, Hao Zhang, Zhen Gao, Xiaopeng Zhang, Jin Li, and Hongkai Xiong, From clip to dino: Visual encoders shout in multi-modal large language models. arxiv preprint arXiv:2310.0885 2023.
Zhuoran Jin, Pengfei Cao, Chenhao Wang, Zhitao He, Hongbang Yuan, Jiachun Li, Yubo Chen, Kang Liu, and Jun Zhao. Rwku: Benchmarking real-world knowledge unlearning for large lan- guage models. Advances in Neural Information Processing Systems, 1198213-98263 2024.
Zachary Kenton, Tom Everitt, Laura Weidinger, lason Gabriel, Vladimir Mikulik, and Geoffrey Irving. Alignment language agents. arXiv preprint arXiv:2103. 14659, 2021.
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International conference on machine learning, pp. 1885-1894, PMLR, 2017.
Nupur Kumari, Bingliang Zhang, Sheng-Yu Wang, Eli Shechtman, Richard Zhang, and Yan Zhu. Ablating concepts in text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 22691-22702, 2023.
Meghdad Kurmanji, Peter Triantafillou, Jamie Hayes, and Eleni Triantafillou. Towards unbounded machine unlearning. Advances neural information processing systems, 36: 19159998899 2023.
Angeliki Lazaridou, Adhi Kuncoro, Elena Gribovskaya, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Gimenez, Cyprien de Masson d' Autume, Tomas Kocisky, Sebastian Ruder, al. Mind the gap: Assessing temporal generalization in neural language models. Advances in Neural In- formation Processing Systems, 01-29348-223633 2021.
Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin Li, Ann-Kathrin Dombrowski, Shashwat Goel, Long Phan, et al. The wmdp benchmark: Measuring and reducing malicious use with unlearning. arXiv preprint urXiv:2403.032/11 2024.
Chris Liu, Yaxuan Wang, Jeffrey Flanigan, and Yang Liu. Large language model unlearning via embedding- corrriied prompts. Advances in Neural Information Processing Systems, 37:118198- 118266, 2024a.
Sijia Liu, Yuanshun Yao, Jinghan Jia, Stephen Casper, Nathalic Baracaldo, Peter Hase, Yuguang Yao, Chris Yuhao Liu, Xiaojun Xu, Hang Li, et al. Rethinking machine unlearning for large language models. Nature Machine Intelligence, pp. 1-14, 2025a.
Zhenhua Liu, Tong Zhu, Chuanyuan Tan, and Wenliang Chen. Learning to refuse: Towards mitigat- ing privacy risks in llms. arXiv preprint arXiv:2407. 10058, 2024b.
Zhenhua I Liu, Tong Zhu, Chuanyuan Tan, and Wenliang Chen. Learning to refuse: Towards mitigat- ing privacy risks in 1lms. In Proceedings of the 31st International Conference on Computational Linguistics, pp. 1683-1698, 2025b.
Zheyuan Liu, Guangyao Dou, Zhaoxuan Tan, Yijun Tian, and Meng Jiang. Towards safer large language models through machine unlearning. arXiv preprint arXiv:2402.10058, 2024c.
Weikai Lu, Ziqian Zeng, Jianwei Wang, Zhengdong Lu, Zelin Chen, Huiping Zhuang, and Cen Chen. Eraser: Jailbreaking defense in large language models via unlearning harmful knowledge. arXiv preprint vrXiv:2404.0555
2024.
Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary Chase Lipton, and Zico Kolter. TOFU: A task of fictitious unlearning for L.MMs. In First Conference on Language Modeling, 2024.
14

---

Published as a conference paper at ICLR 2026
Anmol Mekala, Vineeth Dorna, Shreya Dubey, Abhishek Lalwani, David Koleczek, Mukund Rungta, Sadid Hasan, and Elita Lobo. Altemate preference optimization for unlearning factual knowledge in large language models. arXiv preprint arXivv2409.13477, 2024.
Kevin Meng, Amab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. Mass-editing memory in a transformer. arXiv preprint arXiv:2210.07229, 2022.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture
models. In International Conference Learning Representations, 2017.
Sasi Kumar Murakonda, Reza Shokri, and George Theodorakopoulos.. Quantifying the privacy risks of learning high-dimensional graphical models. In International Conference on Artificial Intelligence and Statisfics, pp. 2287-2295. PMLR, 2021.
Andrei Muresanu, Anvith Thudi, Michael R Zhang, and Nicolas Papernot. Unlearnable algorithms for in-context learning. arXiv preprint arXiv:2402.00751, 2024.
Seth Neel, Aaron Roth, and Saeed Sharifi-Maile Mariii--Maaaarrdii. Descent-to-delete: Gradient-based methods for machine unlearning. In Algorithmic Learning Theory, pp. 931-9621 PMLR, 2021.
Stuart L Pardau. The california consumer privacy act: Towards european-style privacy regime in the united states. J. Tech. L. Pol'y, 23:68, 2018.
Martin Pawelezyk, Seth Neel, and Himabindu Lakkaraju. In-context unlearning: Language models as few shot unlearners. arXiv preprint arXiv:2310.07579, 2023.
Adam Poliak. A survey recognizing textual entailment as an nlp evaluation. arXiv preprint arXiv:2010.03061, 2020.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in neural information processing systems, 01553728-537411 2023.
Protection Regulation. Regulation (eu) 2016/679 of the european parliament and of the council. Regulation (eu), 679(2016):00-13, 2016.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. arXiv preprint arXiv: 1908. 10084, 2019.
Jeffrey Rosen. The right to forgotten. Stan. I.. Rev. Online, 64:88, 2011.
Mohammad Amin Roshani, Xiangyu Zhou, Yao Qiang, Srinivasan Suresh, Steven Hicks, Usha Sethuraman, and Dongxiao Zhu. Generative large language model-powered conversational ai app for personalized risk assessment: Case study covid-19. JMIR AI, 4(1)=e67363, 2025.
Mansi Sakarvadia, Aswathy Ajith, Arham Mushtaq Khan, Nathaniel C Hudson, Caleb Geniesse, Kyle Chard, Yaoqing Yang, Ian Foster, and Michael W. Mahoney. Mitigating memorization in language models. In The Thirteenth International Conference on Learning Representations, 2025.
URL https:://cppeereeeeee/ net forum wtw.mmmidddaCaeueeeee
Ayush Sekhari, Jayadev Acharya, Gautam Kamath, and Ananda Theertha Suresh. Remember what you want to forget: Algorithms for machine unlearning. Advances in Neural Information Pro- cessing Systems, 10111887718866 2021.
Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. Detecting pretraining data from large language models. arXiv preprint 1xXiv22310.16788 2023.
Weijia Shi, Jaechan Lee, Yangsibo Huang, Sadhika Malladi, Jieyu Zhao, Ari Holtzman, Daogao Liu, Luke Zettlemoyer, Noah Smith, and Chiyuan Zhang, Muse: Machine unlcarning six-way evaluation for language models. arXiv preprint wrXiv:2407.064000 2024.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference at- tacks against machine learning models. 2017 IEEE symposium on security and privacy (SP),
pp. 3-18. IEEE, 2017.
15

---

Published as a conference paper at ICLR 2026
Damien Sileo. tasksource: A dataset harmonization framework for streamlined nlp multi-task learn- ing and evaluation. arXiv preprint Stt::2301.05944. 2023.
Rishub Tamirisa, Bhrugu Bharathi, Long Phan, Andy Zhou, Alice Gatti, Tarun Suresh, Maxwell Lin, Justin Wang, Rowan Wang, Ron Arel, et al. Tamper-resistant safeguards for open-weight Ilms. arXiv preprint arXiv:2408. 00761, 2024.
Pratiksha Thaker, Yash Maurya, Shengyuan Hu, Zhiwei Steven Wu, and Virginia Smith. Guardrail baselines for unlearning in llms. arXiv preprint arXiv:2403.088 000.:0339, 2024.
Bozhong Tian, Xiaozhuan Liang, Siyuan Cheng. Qingbin Liu, Mengru Wang. Dianbo Sui, Xi Chen, Huajun Chen, and Ningyu Zhang. To forget not? towards practical knowledge unlearning for large language models. arXiv preprint arXiv:2407.01920 2024.
Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro Von Werra, Clémentine Fourrier, Nathan Habib, et al. Zephyr: Direct distillation of im alignment. arXiv preprint arXiv:2310. 16944, 2023.
Petar Veličkovié, Christos Perivolaropoulos, Federico Barbero, and Razvan Pascanu. Softmax not enough (for sharp size generalisation). arXiv preprint arXiv:2410.01 104, 2024.
Bichen Wang, Yuzhe Zi, Yixin Sun, Yanyan Zhao, and Bing Qin. Rkld: Reverse kl-divergence- based knowledge distillation for unlearning personal information in large language models. arXiv preprint arXiv:2406.01983 2024a.
Boxin Wang, Weixin Chen, Hengzhi Pei, Chulin Xie, Mintong Kang, Chenhui Zhang, Chejian Xu, Zidi Xiong, Ritik Dutta, Rylan Schaeffer, et al. Decodingtrust: A comprehensive assessment of trustworthiness in gpt models. In NeurIPS, 2023.
Junxiao Wang, Song Guo, Xin Xie, and Heng Qi. Federated unlearning via class-discriminative pruning. In Proceedings of the ACM Web Conference 2022, pp. 622-632. 2022.
Lingzhi Wang, Xingshan Zeng, Jinsong Guo, Kam-Fai Wong, and Georg Gottlob. Selective forget- ting: Advancing machine unlearning techniques and evaluation in language models. In Proceed- ings of the AAAI Conference Artificial Intelligence, volume 39, pp. 843-851, 2025a.
Qizhou Wang, Bo Han, Puning Yang, Jianing Zhu, Tongliang Liu, and Masashi Sugiyama. Towards effective evaluations and comparisons for Ilm unlearning methods. arXiv preprint orXiv:2406.09177 179, 2024b.
Qizhou Wang. Jin Peng Zhou, Zhanke Zhou, Sacbyeol Shin, Bo Han, and Kilian Q Weinberger. Rethinking LLM unlearning objectives: A gradient perspective and go beyond. In The Thirteenth International Conference on Learning Representations, 2025b. httphttps://opeereet
Yue Wang, Qizhou Wang, Feng Liu, Wei Huang, Yali Du, Xiaojiang Du, and Bo Han. GRU: Mit- igating the trade-off between unlearning and retention for LLMs. In Fecoy- International Conference on Machine Learning, 2025c. URL
net fttt/ffoummiii
EAjhGr10es
Alexander Warnecke, Lukas Pirch, Christian Wressnegger, and Konrad Rieck. Machine unlearning of features and labels. arXiv preprini 1.Xiv:2108.11557 2021.
Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, al. Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.043350 2021.
Kun Wu, Jie Shen, Yue Ning, Ting Wang, and Wendy Hui Wang. Certified edge unlearning for graph neural networks. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 2606-2617, 2023.
Abudukelimu Wuerkaixi, Qizhou Wang, Sen Cui, Wutong Xu, Bo Han, Gang Niu, Masashi Sugiyama, and Changshui Zhang. Adaptive localization of knowledge negation for continual LLM unlearning. In Focoy- International Conference on Machine Learning, 2025. URL net htt.:/eee///oooieeeee htw.://eeeeeooooooooooo///////
https:://ppnnrett
16

---

Published as a conference paper at ICLR 2026
Jin Yao, Eli Chien, Minxin Du, Xinyao Niu, Tianhao Wang, Zezhou Cheng, and Xiang Yue. Machine unlearning of pre-trained large language models. arXiv preprint arXiv:2402.15159, 2024a.
Yuanshun Yao, Xiaojun Xu, and Yang Liu. Large language model unlearning. arXiv preprint arXiv:2310. 10683, 2023.
Yuanshun Yao, Xiaojun Xu, and Yang Liu. Large language model unlearning. Advances Neural Information Processing Systems, 17:1005255-04444 2024b.
Xiaojian Yuan, Tianyu Pang, Chao Du, Kejiang Chen, Weiming Zhang, and Min Lin. A closer look at machine unlearning for large language models. arXiv preprint arXiv:2410.08 2024.
Saleh Zare Zade, Yao Qiang, Xiangyu Zhou, Hui Zhu, Mohammad Amin Roshani, Prashant Khan- duri, and Dongxiao Zhu. Automatic calibration for membership inference attack on large lan- guage models. arXiv preprint 2XXi::5505.03992 2025.
Shuangfei Zhai, Tatiana Likhomanenko, Etai Littwin, Dan Busbridge, Jason Ramapuram, Yizhe Zhang, Jiatao Gu, and Joshua M Susskind. Stabilizing transformer training by preventing attention entropy collapse. In International Conference on Machine Learning, pp. 40770-40803. PMLR, 2023.
Dawen Zhang, Pamela Sinckenberg-rronee Thong Hoang, Shidong Pan, Zhenchang Xing, Mark Staples, and Xiwei Xu. Right to be forgotten in the era of large language models: Implications, challenges, and solutions. Al and Ethics, pp. 1-10, 2024a.
Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. Negative preference optimization: From catas- trophic collapse to effective unlearning. In First Conference on Language Modeling, 2024b.
Shengqiang Zhang, Xingxing Zhang. Hangbo Bao, and Furu Wei. Attention temperature mat- ters in abstractive summarization distillation. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pp. 127-141, Dublin, Ireland, May 2022. As- sociation for Computational Linguistics. doi: 10. 10118865/1/2222.alll lcl-long.11. URL https: //aclanthor www.nhooooyy.ooooooooooooooooo ogy. oog...org//000 202p:/2222.coo arre22.clllooon... 11/
Xuechen Zhang, Xiangyu Chang, Mingchen Li, Amit Roy- Noy-Chowd,ury, Jiasi Chen, and Samet Oy- mak. Selective attention: Enhancing transformer through principled context control. Advances in Neural Information Processing Systems, 37:1 1061-11066, 1086, 2024c.
Kairan Zhao, Meghdad Kurmanji, George- -Ottvi-a Bärbulescu, Eleni Triantafillou, and Peter Tri- antafillou. What makes unlearning hard and what to do about it. Advances in Neural Information Processing Systems, 17:12293--1333 2024.
Xiangyu Zhou, Yao Qiang, Saleh Zare Zade, Prashant Khanduri, and Dongxiao Zhu. Hijacking large language models via adversarial in-context learning. arXiv preprint vrXiv:2311.09945: 2023.
Xiangyu Zhou, Yao Qiang, Salch Zare Zade, Mohammad Amin Roshani, Prashant Khanduri, Dou- glas Zytko, and Dongxiao Zhu. Learning to poison large language models for downstream ma- nipulation. arXiv preprint arXiv:2402. 13459, 2024a.
Xiangyu Zhou, Yao Qiang, Salch Zare Zade, Douglas Zytko, Prashant Khanduri, and Dongxiao Zhu. Not all tokens meant to be forgotten. arXiv preprint arXiv:2506.03142, 2025.
Zhenhong Zhou, Haiyang Yu, Xinghua Zhang, Rongwu Xu, Fei Huang, Kun Wang, Yang Liu, Junfeng Fang, and Yongbin Li. On the role of attention heads in large language model safety. arXiv preprint arXiv:2410.1377
2024b.
Yixiong Zou, Ran Ma, Yuhua Li, and Ruixuan Li. Attention temperature matters in batebased cross- domain few-shot learning. Advances in Neural Information Processing Systems, 37:1 16332- 116354, 2024.
17

---

Published as a conference paper at ICLR 2026
APPENDIX
A PROOF
A. NOTATIONS
Let V be finite vocabulary, and let denote the parameters of pretrained decoder-only language model. For any onput-output pair (I. 3). where
x ( and = (3/1 (y1.....8r))
the model defines conditional probabilities
(y x ver))
For any (z, y) De (forget set), we partition the target positions into
FC (I...... T} (factual positions),
G ( T}
(function positions).
Factual positions correspond to the tokens that encode the unwanted information to be removed, whereas function positions refer to tokens that serve primarily syntactic or structural roles within the sequence.
A.2 SELF-ATTENTION AND TEMPERATURE
Consider single Transformer layer with one attention head (layer and head indices are omitted for clarity; the argument applies to each head independently). For position f, let E Rd denote the query vector, and let k, U/ E IRd the key and value vectors for all positions S
The attention logits are
(ge, ki)
Cr,i
Vd
and the standard attention weights (with temperature set to 1) are
exp(ar,;) a,ii === exp(
The corresponding attention output
04.50
We introduce temperature parameter and define the smoothed attention weights
exp(ar,1/T)
(1, = Σ/=1 exp(ar.i/T)
with attention output
2(())= Σ ar. (T)
When = the model recovers the base attention: (1) 0u, and 21)1) For 1, the distribution (T) becomes strictly flatter than ag due to the scaling all logit differences by 1/7.
We define the aatentiinnsmootilen teacher model 0, as the model obtained by applying temperature in all attention heads while keeping all other components frredfforward layers, layer norms, and output projection) unchanged.
18

---

Published as a conference paper at ICLR 2026
A.3 OUTPUT LAYER AND TOKEN PROBABILITIES
Let W E RIVixd and E IRI denote the output projection matrix and bias. At position 6, let h, be the hidden representation produced by the Transformer (which incorporates the attention output zt through the subsequent layers).
For each token V, the model computes the logit
(w; 0) (Wws he) bw
and the corresponding conditional probability
exp(lr(w;;) 0))
po(w x o (oyy))
E LeEv (x;()((u;0))
For the intentiionsmootiiee teacher model 0,, applying temperature only inside the attention mech- anism yields modified hidden states he(r). which produce logits
le (w; 0+)
and token probabilities
pe, (w x yet).
A.4 NOTIONS OF "FORGETTING" AND "FLUENCY"
We define properties of interest: the removal of specific factual content and the preservation of normal language behavior.
A.4.1 FORGETTING
Fix a forget example (I, y) E Dr and a factual position t E F. Let y denote the factual token to be removed (for example, the correct entity name in TOFU).
We say that the smoothed model 8, forgets this fact at position t if
poo (yt x o yet) 6F,
for some small threshold ef (roughly the level of random guess accuracy among plausible entities).
At the sequence level, forgetting holds on DF when the average
log po, (vt V<t)
over all (x, E Dr and all t e F is at least a target value LF. meaning the model assigns low probability the factual tokens.
A.4.2 FLUENCY
For function positions G. we require that the model continue to assign high probability the correct function tokens, which reflect grammar and structure.
We say that 0, preserves fluency on (x. y) if
log (ye x yet) < (v I ro Sg.
te G,
for tolerance &G > 0.
At the sequence level, fluency preserved if the average cross-entropy on function tokens increases by at most dg.
A.5 ASSUMPTIONS
To show that attention smoothing can remove specific facts while keeping normal language behavior, introduce structural assumptions how factual and function tokens depend attention.
19

---

Published as a conference paper at ICLR 2026
A.5.1 ASSUMPTION A1 (FACTUAL TOKENS REQUIRE PRECISE ATTENTION)
For every factual position E F, there exists subset St S {l,..., 1f with |S < such that the base attention places most of its mass on St:
Σ 15i 2 7
E (0,
The value vectors at positions in S, contain the main signal supporting the factual token vf, while positions outside S, contribute less this signal.
Define
(S.) = |Sel
Σ Us
z((i) = t 1
V/+
1=1
RESt
ig..
and
St
t - 2, (S,)
unif 24
02
24 (S.) +
t
t
We assume that concentrating attention on S; increases the factual logit margin, and that the uniform attention mixture reduces this margin by at least my > O:
(Wor Wwn 21 - 24 unify Z mp
for all u + yt.
Thus, moving the attention distribution away from the sharp pattern concentrated on S, toward flatter (uniform) distribution decreases the logit margin the factual token y
A.5.2 ASSUMPTION A2 (FUNCTION TOKENS ARE LESS ATTENTION SENSITIVE
For each function position i E G, we assume that the correct token y/ depends on broad mixture of value vectors rather than on a small set of positions. other words, predicting yr does not rely on sharp attention pattern.
Formally, let (3); 0; z) denote the logit of yt when the attention output at position z. Assume the logit is smooth with respect to and satisfies L-Lipschitz bound:
(ye; 0; 2) ( 8; 2n) < || 2'1l2
for all 2, 2,
We also assume that the convex combinations of { hige do not have strong changes in the direction of W. Thus, shifting the attention weights from a sharper pattern toward a smoother (such as closer to uniform) causes only a small change in & (yrc.
A.5.3 ASSUMPTION A3 (NON-DEGENERATT) LOGITS FOR FACTUAL TOKENS)
For each factual position e F, the base model assigns clear margin to the correct factual token w/. Formally,
(vi max (u; 0) AF 0.
This ensures that factual recall in the base model is supported by positive logit gap.
A.5.4 ASSUMPTION A4 (CONTINUITY)
For every position f, the iatention-smooteee hidden state (T( and the logits (w: 0,) vary contin- uously with respect the temperature parameter
This holds for standard Transformer layers, since attention, linear transformations, and activation functions are continuous.
20

---

Published as a conference paper at ICLR 2026
A.6 LEMMAS
A.6.1
LEMMA I (DIRECTION OF ATTENTION CHANGES UNDER TEMPERATURE)
Fix position t and attention logits 01,1.-.., Ot,t E R. For > 0, define the temperature-scaled attention weights
((,((T) exp(ar.1/T)
i= 1
1 exp(ars/T)
Let
a((T): = Σ (ons(T) ar.
denote the average logit at position t under the attention distribution (x())).
Then, for every E (I,
a (T)
72 (T) (ār(T) ac.s).
θr
In particular:
If at, [T()), then
Ør a (T) < 0, so the attention on index decreases as r increases.
aa, (T( then 0 0, (T) 0, the attention index i increases increases.
If af, = ( (T), then 0r (T) = 0.
Proof.
For a fixed write a; == at,i and (;(T) == (1,s(T) for = 1,..., to lighten notation. By definition,
(;(T) explai/T) Z(T)
(
exp(a,/T).
We first differentiate the log- attention with respect 7:
log (s(())
llog (T))
Taking derivatives gives
log (T) =
0j
1 07(T)
O-
72 Z(T) dr
Next we compute
0Z(r) 0r
By the chain rule,
0Z(r)
Σ
exp(ay/11
Σ exp(a,/T)
(-
Oj
aj exp(a,/T).
0r
72
72
j=l
Thus
OZ(T)
Σ Σj=1 aj exp(a;/T)
(T(T)) 0, = a(T);
Z(T)
Ør
Σ C exp((t//1
where
a(T) = Σ a5())
the average logit under the current attention distribution.
Plugging this back into the derivative of log (T), we get
Ør log (T) = a; +
a(T)
=
72 (ā(T) - oz).
1
72
72
21

---

Published as a conference paper at ICLR 2026
Finally, move from the derivative of the log-attention the derivative of the attention itself. Since
(T) a;(T)) ① 0r 0r
log (T))
obtain
0 a as)))
1 (ā(T) -a.) ai(T) (ā(T) a.),
1
72
72
which is the claimed formula.
The sign statements follow directly:
If a; ā(T), then ā(T) 0, 0, or a.) < 0 and the attention decreases with
If a, < @(T), then a(T) - 0 > 0, so a (T) > 0 and the attention on increases with T.
If a; = a(r), then à är (T)
This shows that increasing the temperature shifts mass from positions with above-average logits positions with below-average logits, which matches the intuitive picture of attention becoming flatter.
A.6.2 LEMMA 2 (ATTENTION ENTROPY INCREASES BY INCREASING TEMPERATURE)
Fix a position and attention logits Ot,1,...,,0 E RR. For > 0, define
exp(an./T)
(n,(T) Ej= exp(ats/T)
i= 1=1,...,4
and the entropy
H2(T) = Σ
(T) (T))).
i=1
Let
ar(T) = Σ and (T) ar, ar
be the average logit position under (T)... Then
i H+(T)
73 (ara.(r)( ar.)) 2 0,
0T
with strict inequality whenever the logits d.,1...,; det are not equal.
Proof.
Fix & and write a, == or, and (x(T) == (T) for = L,..., t to lighten notation. Let
1 2(7) = Σ exp(a;/T);
a(7) = Σ (r()) ag.
j=1
j=1
By definition,
au(7) exp(ai/T)
log (z) T
log Z(7).
2(7)
Thus the entropy can be written as
H(T) = Σ (u(T) (og
= Σ (s((T)
(i - log 2(T))
1=1
Using E, (T( 1, this simplifies to
H(T) = Σ
as((T)a + log log = a(T) + log (T))
22

---

Published as a conference paper at ICLR 2026
Differentiate (T) with respect
0H = 72
1 1 dā(T)
1 0Z(T)
a(T)
+ Z(T) dr
θr
0r
We compute 02(r). 8r
(Z(T)
Σ 3=1
dr exp(a//T) = Σ
exp(a,/T) -4)
(-
Σ
aj exp((1///T))
Ür
72)
72
==1
1=1
Hence
OZ(T)
Σ 0g exp(ag/T)
(T( -- a(T)
Z(T)
72 E Rk=l (x)(a
j=1
Plugging this into the expression for 0H/0T gives
0H =
Ör 72 a(T) 1 (ū((T)
1 aa(TT)
7 0r 72 a(T) = 5 θT
We now compute
(a(r) dr
By definition,
dā(T) 1 dar(T)
a(r) = Σ (T)
so
ai θr
dr
1=1
i=1
From Lemma we have, for each i,
(T; 1 (r(T) (ā(r) ai).
Ør 72
Therefore,
θā(T) = 72
Σ
a (T(()) (ā(T) as) = 72 (((T) Σ ai(()a; -
Σ (T(())a)))))))))))
0r
i=1
1=1
Since Σ; (;())a = a(T), this becomes
0ā(T) 1 0r 7 (a(T)) Σ (r)
1
- -a())))
72
The term in parentheses is the variance of the logits under c<(T):
Vara(r)(a)) = ai(r)aa -ā())
Hence
dā(T)
72 (ara
Or
Substituting into 0H/Or yields
OH
(- vara())(a))
1
3 (ara())(a).
O-
Since variance is always non-negative and equals zero only when all logits 01,..., ar are equal, we obtain
dh 0T
0
with strict inequality whenever a,...., ar are not all equal. This proves that the attention entropy increases with temperature unless the attention already uniform.
23

---

Published as a conference paper at ICLR 2026
A.6.3 LEMMA 3 (EFFECT ON FACTUAL-TOKEL LOGITS)
Fix factual position F. Under Assumptions Al and A3, the logit margin of the factual token decreases as increases above 1, and becomes negative for sufficiently large т. More precisely, there exists TF such that for all T 2 TF.
4 ((vi; 0+) - max (u; 0+) - uz37
MF <0. 2
Proof.
By AI, most of the base attention position t lies on the set St, whose value vectors strengthen the logit of yi- The base attention output can be written as
21
Σ S
on +
Σ On
S
For large т, Lemma implies that (T)(T) approaches the uniform distribution co. Hence
2T(T)= (T))(T)
Σ . Xs (
for weights As, Ag determined by | and I. Define the change
(z = =x(T) 21.
By Lemma 1, as t increases, (T( moves the attention output away from the sharp pattern that favors St, and toward the S, with attention weights lower than entropy.
Let + yf be any competing token. The logit difference at temperature is
4 07) - le(u; 07) = (W W., zx(T)) + (byg - bu).
Subtracting the difference at gives
A, (We; W., (((T) 22)-
By A1, putting more weight on S; increases the factual margin, so moving away from S; (as smooth- ing does) decreases Thus, for large enough т, the inner product above is negative and can be bounded above by negative constant once (r(r) is close to uniform.
By A3, at T the factual margin positive:
(ut: (u 2 AF
By continuity in J (A4), the margin decreases continuously as the attention pattern is smoothed. Since the margin becomes negative for large т, the intermediate value theorem guarantees a point TF where it crosses zero. For any - 2 TP, the margin is strictly negative, and by adjusting the threshold we may ensure the bound -m /2.
This implies that for 2 TF.
po, (y) x o yer)
1 exp(mpp /2) = EF 2
so the factual token is no longer the most likely output.
A.6.4 LEMMA 4 (EFFECT ON FUNCTION-TOKEN LOGITS IS SMALL)
Fix function position E G. Under Assumptions A2 and A4, for any 7) > 0 there exists To such that for all 5 E [1, Fa]. [e((ve: 0-) (e(v):)) S n-
Proof.
24

---

Published as a conference paper at ICLR 2026
For any compact interval [1, Fol, the attention weights (T) vary continuously in and stay inside the simplex. Therefore (T( is a continuous function т.
By A2, the logit of the correct function token is L-Lipschitz in the attention output:
[eclyc: (e):: 0) - ([2 zell2-
Since 24 (T) + 24 T + (by A4 and continuity of the attention map), for any >> 0 we can
choose FG 1 so that
[|=(()) - 24 12 - n/L
for all r E [1, Fa].
Substituting into the Lipschitz bound yields
(e<(v 0-) (e 0)] S n
which proves the claim.
A.6.5 LEMMA 5 (SMALL LOGIT CHANGE IMPLIES SMALL CROSS-ENTROPY CHANGE)
Let p and be two distributions over V whose logits differ the true token by 7). Then the increase in negative log-like log-likelihood at that token is at most a function e(n) with e(n) - as -
Formally, let la and fa, be two logit vectors. If for some token w,
le, (w) &a (w) S n
and the remaining logit differences are uniformly bounded, then
log Pe, (w) < llloooroo po(w) c(n).
Proof sketch. The softmax map from logits to probabilities smooth and Lipschitz on any compact region of logit space, and the negative log-probability of fixed token is smooth as well. Thus small change in the logits produces a small change in the negative log-likelihood. The function c(n)
follows from the Lipschitz constants of the softmax and the log operation.
Combining Lemma 4 and Lemma 5, for any tolerance &G > O we may choose ia > so that for all [1, Fc] and all function tokens t G.
log po, (Ve x o yee) S log I oroee) dG.
A.7 MAIN THEOREM
Theorem: Attention smoothing yields forgetting with fluency.
Assume A I-A4 hold for all forget examples (Ir, v) Dr and for their factual and function positions.
Then there exists a temperature interval [To. 71] with
< To T1 < 00
such that:
Forgetting: For all T E [1), the smoothed model 0, forgets the factual tokens in Dr. In particular, for every factual position & F,
Pa, (vi [rye))) S EF,
and the average factual negative log- likeliild is at least LF > 0.
Fluency: For all [To, rı], the increase in average loss on function tokens (over both Dy and DR) is at most 6c:
log Po. (ye x [ S
Σ
log Po xr + 8c-
G
BEG
tec
25

---

Published as a conference paper at ICLR 2026
Thus there is non-trivial range of temperatures where factual knowledge is forgotten while fluent language behavior is preserved.
Proof.
Step 1: Forgetting at sufficienlly large т. For each factual position t E F, Lemma 3 provides temperature (t) such that for all [t))),
E 0+) max (u; uzy?
mF 2
Hence
po, (1 [oyec) S cFF
for or < 1/2 depending only mr.
Define
TF
max max (F()). (x.))ED TEF
For all TF. the forgetting incquality holds for every factual position in every forget example, and the average factual loss is at least L5 = log ep.
Step 2:
Fluency sufficiently small For each function position t E G, Lemma 4 states that for any > there exists (t) > such that for all 7 E [1, (t)).
(e(e 0,) - ( 0) <
Lemma 5 then ensures that the extra loss on each function token is at most c(1), where c(n) - 0 as
7) + 0.
Choose so that c(1) &c. and define
ta = min min To (f). (a.y) IEG
For all T E [c)
log po. (ye x o yee) I log po x o y<r) + dg:
and averaging this over all function tokens gives the bound in the theorem. Step 3: Establishing a common temperature range. We have:
Forgetting holds for all 2 TP-
Fluency holds for all E [1,
Both hold simultaneously for all
T E [al
which is non-empty whenever TF I Tc.
This condition reflects the structure in A1-A3: factual tokens depend precise attention patterns that collapse quickly when smoothed, while function tokens depend on broader patterns that remain stable under mild smoothing.
Choose any To, Ti satisfying
< Tp S S
To
Then for all E [To, 7.). both forgetting and fluency hold.
26

---

Published as a conference paper at ICLR 2026
B BASELINES
Notation. Let P(y z;0) denote the probability of an output sequence = (311...., given input under model parameterized by 0. This probability is defined as:
r
P(y 2; = II p(u x o V<e: 0)*
Forget Loss. Existing methods can be broadly categorized into Convergence-bused Unlearning and Divergence-based Unlearning. The baselines use
Gradient Ascent (GA) Yao et al. 2023) maximizes the prediction loss on the forget set, effectively reversing the training objective:
Las (DF;; = (B(
log p(y. x yers
(6)
T
1=1
Negative Preference Optimization (NPO) (Zhang et al., 2024b) is derived from Direct Preference Optimization (DPO) (Rafailov et 2023). It treats forget-set answers neg-
ative samples while omitting positive terms:
2 ENPO (D;; -( log -Blog P(y xx; B
P(y x:;
where o()) = 1/(1 + e-*), B is hyperparameter, and Ore is the fixed reference model. NPO be viewed GA with adaptive gradient scaling (Zhang et al. (022b).
Maximizing Entropy (ME) (Yuan et al. (024) minimize the KL divergence between the
predicted distribution for each token and uniform distribution with vocabulary size.
T
LME (D; 0) = (a.y)~D
Σ
(8)
T
1=1
where Uny is uniform distribution over the vocabulary of size K, where each value is 1/K.
IDK Fine-tune (IDK) Maini et al. 2024 reframes unlearning as instruction tuning by relabeling forget-set questions with random responses from DIDK, pool of rejection tem- plates (e.g., "Sorry, don't know."). Its loss is
LIKK (Dr, DIDK: 0) EEwDr.yDDDes log Ply x:; 0)]
Direct Preference Optimization (DPO) (Zhang et al. 2024b) applics the standard DPO loss (Rafailov et al. 2023), using forget-set answers negatives and rejection templates from DIDK as positives.
1 (e.w))
(DF, DIDKi 0.tt)
E
B
P(y' 1; log P(y'
P(y xr; Obwse) ]
P(y x;
(10)
log
Blog
x;; Oome))
where Ohuse denotes the parameter the reference model, which is the initial base model for unlearning.
SimNPO (Fan et al. 2024). It derives from NPO, whose reward function given by the comparison with the reference model. In contrast, SimNPO takes reference-free but length-normalie reward formulation, so they can mitigate the reference model bias in NPO by replacing its reward formulation, follows:
LsimNPO (Di; 0) - B () log (- log P(y 1lygP((];))) -
(11)
where ? 2 0 is the reward margin parameter, inherited from SimPO, which defines the margin of preference for desired response dispreferred one.
27

---

Published as a conference paper at ICLR 2026
Representation Misdirection (RMU) (Li et al. 2024) misdirects internal representations on the forget set by pushing layer-f activations toward fixed random direction with am- plified norm, corrupting downstream processing. It's forget loss is
T LRMU = RE
Σ T
H (mer;
u112
(12)
where (Iret; 8) denotes the hidden state at layer & of the model parameterized by 0, given the prefix Tet, u is a random unit vector, c >> 0 is a scaling constant, and T is the sequence length of x.
IDK and DPO only applicable in QA-style datasets, since they require rejection templates as positive samples.
Retain Loss. In addition the GD and KL regularization losses introduced in Section further include the Answer Preservation (AP) and Mean Squared Error (MSE) loss as an additional baseline component.
Answer Preservation (AP). To prevent unlearned models from becoming overly ignorant during targeted unlearning, Yuuan et al. 2024) proposed the Answer Preservation (AP) loss regularization term. Unlike standard GD KL regularization, AP explicitly balances two objectives on the retain set: (1) reducing the probability of rejection templates, and (2) maintaining the probability of the original answers. Formally, the AP loss is defined
(DR( DIDK; 0) B(..)) (e.w)) DIDR log (-BBa
P(y' 25; 0] P(y x;; 0)
(13)
where of o-)is the sigmoid function and | is temperature parameter.
Mean Squared Error (MSE) (Li et al. 2024). The motivation of this loss is to limit the degradation general capabilities by explicitly constraining the updated model's internal representations to remain close to those of the base model. Concretely, given the retain dataset DR. we impose an 62 penalty between the hidden activations of the updated model and the base model:
T Σ T
CMiE(DR;BP = EENDA
H1 ( 1
(14)
1=1
where H' [e: 0) denotes the hidden state at layer & of the model parameterized by 0, given the prefix Eet, and T is the number of tokens in This loss explicitly encourages the updated model to preserve activation-level similarity with the reference model on the retain set, thereby mitigating the risk of excessive utility loss during unlearning.
EVALUATION METRICS
C.I RIGHT TO BE FORGOTTEN
Notation. Let (x;; 8) denote the decoded output produced by a model parameterized by for input
Metrics We evaluate the Seabt-to-be-Forgotte scenario using the following metrics:
ROUGE (R) We use ROUGE-L recall (Maini al. 2024 compare the model's decoded output (xx; f) with the ground truth answer The score, denoted as ROUGE (g(2; 0), captures the longest common subsequence overlap at the word level.
Probability (P) We measure the model's likelihood of producing the ground-truth answer y (Maini et al. 2024). For a question-answe pair (x., y), we compute the normalized conditional probability:
T
P(y 2; = p(ye xo oraeee 0)
II
t=1
28

---

Published as a conference paper at ICLR 2026
where T is the answer length, yt is the t-th token, and yor denotes the prefix up to position t.
Truth Ratio (TR) We assess whether the model assigns higher likelihood to correct an- swers than to incorrect ones (Maini et al.) 2024 Yuan et al. [024). The metric TR com- pares the average normalized conditional probability of perturbed answers y, which are plausible but incorrect variants of y, against that of paraphrased answer y. which is valid rephrasing of y. Formally,
1 1=
P(
xx;0)
TR(y x;
P
x:
A model lacking relevant knowledge should assign similar probabilities to correct and incorrect answers. For evaluation, we report max(0, TR) on the retain and 1 - min(TR, 1/TR) on the forget set.
Token Entropy (TE) We evaluate the lexical diversity of the model's output Yuaan et al, 2024). Some unlearned models often generate long, repetitive continuations (e.g.. gibberish output) that reduce readability. To quantify this effect, we compute a normalized token entropy:
- -E f(ws) log2 f(w;)
TE(g(;;0.))
logg [g(;;; 0)1
where [x;;; O)] is the output length, is the number of unique tokens, and f(w,) denotes the frequency token w;. Lower TE indicates excessive repetition and incoherent outputs, while higher TE reflects more diverse and readable generations.
Cosine Similarity (CS) We measure the semantic similarity between the model's output before and after unlearning on the retain set (Yuan et al. 2024). line with the semantic Sentence- BERT (Reimers & Gurevych, 2019) to embed the output produced by the base model and the output produced by the unlearned model, and then compute their cosine similarity, truncated zero:
textual similarity task (Cer al.) 2017).
max (Cos(g(z; (g(z; Obase), g 0)), 0).
This metric captures semantic drift: even surface overlap (e.g.. ROUGE) remains high, cosine similarity decreases when the unlearned model appends irrelevant or fabricated con- tent.
Entailment Score (ES) We assess the factual consistency of model outputs with respect to ground-truie answers using textual entailment (Natural Language Inference, NLI) (Yuan) et al. 2024). NLI evaluates whether a premise entails, contradicts, or neutral with respect hypothesis, and has been widely applied in NLP evaluation (Poliak 2020). Formally, a text entails a hypothesis h (t + h) if a human reading would reasonably infer h to be true.
predict the relationship between each model output and its ground-truth answer (Liu et al., 2024b). The entailment score is defined the proportion of predictions labeled as "entailment" which should be higher on the retain set and lower the forget set.
We use pre-trained NLI model Siiloo 2023)
C.2 COPYRIGHT SCENARIO
We evaluate the copyright scenario (MUSE tasks) using the following metrics:
Verbatim Memorization (VerbMem) We assess whether the model reproduces training data verbatim (Shi et al. 2024). Given forget-set sequence E Dr, provide the model g with the first / tokens Z[a] and compare its continuation with the ground truth suffix T/413 using the ROUGE-L Fl score. The metric is averaged over all examples in Dr:
VerbMem(0, Dr)
1 Dr
Σ ROUGE(g(rgg +)))
A lower VerbMem indicates stronger protection against verbatim leakage.
29

---

Published as a conference paper at ICLR 2026
Knowledge Memorization (KnowMem) We measure whether the model retains factual knowledge of the forget set (Shi et al. 2024). For each sample (x, y) E Dr. we query the model with and compare its answer g( 8) with the ground truth y using ROUGE. The
metric is averaged over all pairs:
KnowMem Dp)
Σ
ROUG)[(((; v).
DF
(2.g)ED;
A lower KnowMem reflects more effective removal of copyrighted sensitive knowledge.
Privacy Leakage (PrivLeak) To evaluate privacy preservation, we follow (Shi et al. 2024), and adopt the sfate-of-the-ar Min -0 Prob method (Shi et al., (023) and com- pute the AUC-ROC score (Murakonda et al. 2021 Shokri et al. (201)) for discriminating leakage then defined relative to retrained
Dr from a holdout set molddoun The privacy
model:
PrivLeak AUC(O; Dr, Pholdou))
- AUC(Orerrann Dp, D Pholdou))
AUC( Dr, Dholdout)
A good unlearning algorithm yields PrivLeak close zero, while large positive negative values indicate over- or under-unlearning.
D CONTINUAL UNLEARNING SCENARIO
Figures 5 and 6
report FE and MU for continual unlearning on TOFU, DPO attains higher FE than
ASU but drives MU to 0.0, indicating extreme ignorance. ME achieves MU comparable to ASU,
but ASU delivers higher FE, yielding a hetter average performance overall (as shown in Figure 3
Continual forget01
Continual forget05
Continual forget10
Forget Efficacy
DPO
IDK,
ASiles
Continual Unlearming Steps
Figure 5: Forget Efficacy in continual forget01, forget05 and forget10 unlearning tasks.
Continual forget01
Continual forget05
Continual forget10 forgett
100
102
130
Model Utility
Ghop
NPO
DPOgs
NEop
IDK
A5ile
Continual Unlearning Steps
Figure 6: Model Utility in continual forget01, forget05 and forget 10 unlearning tasks.
REAL-WORLD UNLEARNING SCENARIO
Table presents the detailed results for each metric in the real-world unlearning scenario, corre- sponding to the summary provided in Table 2
F HAZARDOUS TAZAAROOU-KNOWLLEDGE UNLEARNING SCENARIO
In addition to output-level alignment, we also match internal representations. We minimize the mean squared error (MSE) between hidden states of the model parameterized by 0 and those of the
30

---

Published as a conference paper at ICLR 2026
Table 6: Detailed results of each metric in real-world unlearning scenario.
Neighbor Set
Forget Set
Method
R
P
TR f
TE1
CS
ES
MU
R
P
TR.
TE1
ES
FE
Base
78.21
33.75
56.17
88.50
62.25
61.38
80.67
60.70
67.75
36.83
Divergence-bone
Unlearning
GAGp
63.53
5.01
78.18
83.08
70.38
46.75
21.76
0.00
0.00
48.81
37.68
0.00
65.73
GAxl
51.77
26.69
62.03
72.80
64.50
28.50
43.72
0.00
0.00
69.
0.00
0.00
0.00
NPOm
50.41
8.71
42.84
69.39
57.80
11.00
21.38
42.28
5.93
39.31
66.41
4.75
71.44
NPOsa
50.55
17.51
43.05
68.79
55.38
11.50
27.32
41.27
9.22
38.20
67.53
3.00
72.11
Convergence-busee Unlearning
DP
0.45
25.22
35.88
71.09
5.15
0.00
0.00
0.30
21.41
34.82
79.70
0.00
82.45
DPO
3.05
35.60
40.45
99.69
9.72
0.75
3.28
0.82
28.14
37.07
99.97
0.00
83.48
IDK
2.61
32.12
46.88
100.00
8.77
0.00
0.000
2.63
31,57
47.07
100.00
0..00
7X
ADKAp
70.81
29.93
53.43
86.66
80.58
42.50
52.76
3,45
22.58
51.39
99.27
1.50
78
MEaD
70.25
21.21
58.12
90.66
82.57
42.75
47.96
2.43
0.19
22.65
16.46
0.25
48.10
ASU
69.10
37.30
46.55
85.08
80.36
41.75
54.10
33.30
13.37
31.25
73.84
3.25
76.97
ASUsI.
69.96
42.97
44.29
88.91
82.56
41.50
55.76
30.32
19.74
31.05
91.38
5.25
79.60
attention-smoothed model 0, at chosen layer. Concretely, we align 0 with Osase the retain set and with 8, on the forget set [5 as follows:
(Dr; ExwDr
H (Fer;0)
(x 0-)1
(15)
where H' (x< 0) denotes the hidden state at layer of the model parameterized by 0, given the
prefix Tet.
2024). The forget set Dr comprises WMDP-Biology and WMDP-Cyber corpora, and the retain D, is Wikitext (Mer- ity et al. 2017). Unlearned models are evaluated on the WMDP multiple-choice QA benchmark (zero-shot; select the option with highest conditional probability) al.) to measure general utility. We choose layer {(7) as the unlearning layer, and we only update the MLP layers of three layers f, can be leveraged to save memory and efficiently unlearn on larger LMs (Li et al. 2024).
Setup. We assess hazardous-knowwledee removal using WMDP (Li et al.
measure residual hazardous
knowledge, and on MMLU (Hendrycks et
2 (7.6,5), which
Models. We evaluate Sarardous-knowledee removal on the following LLMs: Zephyr-7B-9 (Tunstall et al. (2023), Misttal-77B-instruutv-..22
Table 7: Comparing base models and unlearning methods question-answer evaluation (WMDP, MMLU). All WMDP and MMLU scores are per- centage points.
(Jian et al. 2023
Baselines. We compare against RMU (Li al.
Model
Method
WMDP (L) Cyber
MMLU (t)
Bio
2024). SCRUB (Kurmanji al. 2023). Foster et al. 2024), and LLMU (Yao
SSD
Base
64.3
44.8
58.5
jct al.
2024b). Baseline runs are conducted on Zephyr-7B; in preliminary screening this backbone, all baselines except RMU signifi- cantly affect Model Utility while not achieving good forget efficacy, so we do not extend them the other models.
Zephyr-7B-9
LLMU
59.5
39.5
44.7
SCRUB
43.8
39.3
51.2
SSD
50.2
35.0
40.7
RMU
31.2
28.2
57.0
ASU
32.1
31.7
57.5
Base
65.1
41.5
59.0
Mistral-7B
RMU
30.7
32.3
57.7
ASU
31.5
29.5
57.2
Performance on WMDP. Table 7 compares
our method with the baselines WMDP (Bio, Cyber). On Zephyr-7B, ASU achieves higher utility (MMLU accuracy) while delivering comparable forgetting performance on Bio and Cyber. Mistral- 7B, ASU matches RMU Bio and MMLU, while achieving slightly stronger forgetting Cyber. These results suggest that ASU can also extend to settings requiring the removal of entire distribu- tions, such hazardous knowledge.
31

---

Published as a conference paper at ICLR 2026
G FORGET-TEACHER TEMPERATURE SELECTION
We select the attention temperature т via binary search, using negative log-likelihood (NLL) as the objective. As shown in Figure 2
NLL increases monotonically with within the examined range.
Step 1: Define bounds. For the upper bound, we start from 7 = and repeatedly double T until the model begins to produce gibberish (fluency checked manually with automatic score). The first such value is taken as Thigh- In practice, > almost always yields gibberish, we cap Thigh = 4. We set the lower bound Tlow 1.0.
Step 2: Binary search for a valid range. Within [No Thigh] we apply binary search guided by negative log-like lileliiihood (NLL). We identify the largest interval [Now Thigh] where the forget-teacher breaks lexical and semantic associations in the forget set, yet still maintains coherent outputs. For example, we often find the valid range be between 2.0 and 3.0.
Step 3: Greedy search per scenarios. Once the valid range is established, we perform a greedy search within select the best 7 for each scenario.
Remarkably, all TOFU tasks consistently yield T = 2.3, and other tasks converge to nearby values. This consistency demonstrates the robustness of our method across different unlearning scenarios. More details of r and hyperparameters across all scenarios are shown in Table
H HYPER-PARAMETERS
We provide hyperparameters used across all scenarios in Table
Table Optimal r and À values across all scenarios.
ASU
ASUKI.
Tasks
Model
X
À
TOFU TOFU torgetos
2.3
0.1
2.3
0.1
LLaMa-2 7B
2.3
0.1
2.3
0.1
TOFUtarget10
2.3
0.1
2.3
0.1
Continual
2.3
0.1
2.3
0.1
Corgenu
LLaMa-2 7B
2.3
0.1
2,3
0.1
Corge10
2.3
0.1
0.1
Real-world
LLaMa-3 8B
2.7
0.05
2.5
0.05
MUSENewS
LLaMa-2 7B
2.0
0.4
2.4
0.3
MUSE Books
ICLM-7B
2.3
0.001
2.4
0.001
I ABLATION ON LAYERS
In Figure 7 we smooth attention over different sets of consecutive layers. Smoothing consecutive layers at layer means modifying layers f, ..... f n + 1. When > we smooth layers 1....,1 1. The value 0 on the xaxis indicates that no attention layer is smoothed. All plots are
1. 1 generated with temperature 7 3.0.
From Figure 7 the leper-left panel (smoothing single layer) shows a clear rise in NLL when smoothing layers through 8. This pattern remains visible across the other panels: as we increase the number of layers being smoothed, the overall NLL grows, but the main rise still occurs in layers 3-8. Across all settings, the NLL for factual tokens is consistently much higher than that for function tokens, regardless of which layers smoothed, which supports our finding that factual positions are far more sensitive to attention smoothing.
This demonstrates that smoothing only small block of early layers is enough to forget the factual tokens. This observation matches earlier findings such as Meng et al. (2022); Guo et al. (2025). which show that factual knowledge is largely stored in shallow transformer layers.
32

---

Published as a conference paper at ICLR 2026
Models
Categones
layer
Consecuttive layor(s)
Consecutive layerts)
Mogatove Lop- Lihhhhooood
Layer
Lay
Layer
Consecutive layerts)
Consecutive layer(s)
Consecutive layeris)
Nogative Log-Likelhodd
15
20
25
30
9
10
20
25
30
0
1.0
15
20
25
30
Layer
Lay B7
Layer
Conserutive
Consecutive
Consecutive layeris)
Nogattue Log Libclhood
2.
15
25
30
Layer
Consecutive layerts!
11 Consecutive layents)
12 Consecutive layerts)
Nogativs -o--Ukllhhod
15
25
10
20
10
15
Layer
Layer
Layer
Figure 7: Effect of smoothing different consecutive layers factual and function tokens.
J STABILITY OF ASU
Table 9: ASU performance different temperatures TOFU Forget05 task.
forget05
Method
MU
FE
Avg.
ASUGD(T 2.0)
74.21
75.72
74.97
ASUGD(T = 2.2)
72.47
78.04
75.26
AS 2.4)
72.06
79.35
75.70
ASU = =2.6)
71.31
80.98
76.15
AS =2.8)
71.38
81.55
76.46
(S =300)
71.17
75.00
73.09
ASUKI. (T = =2.0)
73.88
75.91
74.89
ASUKL ( =.22
72.91
78.04
75.48
ASUKL (T 2.44)
72.34
79.83
76,08
ASUKI. (T =2.6)
71.68
81.05
76.37
ASUKL 2.88
71.31
81.37
76.34
ASURI. =3.0)
71.20
75.71
73.45
33

---

Published as a conference paper at ICLR 2026
K IMPLEMENTATION DETAILS
K.1 TOFU
We use the Llama-2-Chat-75 model fine-tuned by (Maini al. 2024) as our target model. All experiments are carried out on two NVIDIA H100 GPUs with 80GB memory. We follow the public TOFU repository and train with DeepSpeed ZeRO-3 reduce memory usage. Our training setup follows (Maini al. 2024). We adopt the AdamW optimizer with weight decay of 0.01, learning rate of x 10 and an effective batch size 32. Unlearning is performed for 5 epochs, where the learning rate is linearly warmed up during the first epoch and then linearly decayed for the remaining epochs. For evaluation, following (Maini et al. 2024), we randomly select up to 400 QA pairs from the TOFU dataset to keep the process faster. Following previous works (Zhang et al. 2024b. Yuan et al.] 2024), for NPO and AP, we set / = 0.1, and for ME, we use À 0.1 in the fictitious un unlearning setup and À 1.0 in the continual unlearning setup. These choices follow the best settings reported in the referenced papers.
K.2 REAL-WORLL DATASET
In line with (Liu et al. 2025b), we adopt stream evaluations through the ioaluuation-haneee with its default configuration.
as our target model. We run down-
For ASU, we unlearn for 5 epochs, with learning rate x 10-6, and use = 0.05.
Following previous work Yuaan et al. 2024), we tune the baseline methods by searching over {3, 10-6, 10-5), using the best hyperparameters
5} epochs and learning rates in {2 10-0 reported in the literature. For ME, we set À 0.5, and for NPO and IDKAP. we set B = 0.1.
To ensure that forgetting is measured a way that holds across different prompts, and we compute unlearning metrics using golden answers rather than the original generated outputs of the model before unlearning.
All other training and evaluation settings are kept the same in the TOFU experiments.
K.3
MUSE DATASET
For the MUSE experiments, Following Shi et al. 2024 Dorna et al. 2025) and perform unlearning with constant learning rate of x 10 and effective batch size of 32 for 10 epochs. All other training settings remain the same as in the TOFU experiments.
L ADDITIONAL EXPERIMENTS ON TOFU
Table 10: Results of unlearning methods on the TOFU benchmark using Llama-3. 3.1-8B Higher is better for all metrics. We report Model Utility (MU), Forget Efficacy (FE), and their Average (Avg.) across the three TOFU tasks. Best scores are in bold. All results are reported in percentages.
forget01
forget05
forget10
Method
MU
Avg.
FE
Avg.
MU
Avg.
Divergence-based
GAgi.
36.25
74.98
55.62
36.34
0.00
18.17
54.43
1.87
28.15
NPOKL
68.20
$8.89
63.55
$9.99
60.78
60.38
65.50
57.48
61.49
Convergence-bee
DPOKı.
78.45
44.22
61.33
1.74
68.51
35.12
19,50
63.58
41,54
IDKAP
77.68
47.28
62.48
72.74
60.93
66.83
72.70
65.79
69.24
IDKKL.
73.67
52.95
63.31
0.00
64.68
32.34
21.72
55.77
38.75
MÉKL
78.88
73.09
75.99
75.14
70.15
72.65
74.44
58.73
ASUgL
78.36
77.69
78.02
71.67
74.07
72.87
71.81
77.00
74.40
34

---

Published as a conference paper at ICLR 2026
M FICTITIOUS UNLEARNING SCENARIO
Tables
11 and report detailed per-metric results on the TOFU benchmark across all baselines.
Table 11: Detailed results for each metric the retain set and the forget set for three tasks in the TOFU benchmark, corresponding to the summary provided in Table T
Retain Set
Forget Set
Task
Method
R
P
TR
TE
CS
ES
R
PJ
TR J
TE
ES
GACo
81.91
87.37
49.42
95,40
91.53
42.33
41.77
9.22
46.45
92.29
30.00
GARa.
84.78
88.74
49.50
95.59
92.87
50.33
45.72
9.74
44.70
91.95
30.00
NPOco
86.99
83.80
49.56
94.75
92.21
34.00
45.18
10.30
36.48
92.04
30.00
NPOkL
86.56
84.20
49.59
94.72
92.25
33.67
45.14
10.43
36.20
92.34
32.50
DPO
88.72
96.58
45.63
97.34
95.76
94.67
36.26
83.96
40.58
97.79
12.50
DPOkL
88,92
96.58
45.61
97.34
95.83
94.33
37,89
84,00
40.58
97.47
12.50
forget01
IDKc
47.14
93.72
45.55
55.31
$2.00
0.86
71.61
39.72
99.76
0.00
IDKxi
48.16
93.71
45.52
98.72
56.22
$3.00
0.95
71.45
39.81
99.76
0.00
IDKAr
87.43
96.99
45.92
97.37
92.00
1.01
72.30
40.01
99.37
0.00
MFaD
77.83
88.99
44.93
96.87
90.42
64.00
2.46
0.42
25.96
43.81
0.00
MFKi.
85.87
91.39
44.91
97.07
94.21
73.33
2.54
0.29
18.21
31.18
0.00
ASUaD
80.91
83.84
42.39
96.96
93.36
70.33
13.14
2.75
16.63
73.01
0.00
ASUKL
80.93
84.13
42.50
96,97
93.62
73.33
14.61
2.89
16.70
71.46
2.50
GAGD
15.98
6.88
65.72
22.48
18.36
32.33
0.52
0.00
0.81
0.00
GAga
11.04
3.65
59.70
15.68
18.63
22.00
1.55
0.00
40.81
1.14
0.50
NPOgp
54.04
45.04
46.07
85.68
74.55
27.33
35.78
11.19
33.65
69.82
16.50
NPORL
53.84
44.88
45.75
84.85
74.22
31.67
35.74
11.45
33.48
68. 24
14.00
DPOca>
0.55
60.22
37.61
99.99
5.56
0.00
0.11
48.61
34.37
99.00
0.00
DPOkt
0.55
60.05
37.63
99.99
5.57
0.00
0.11
48.45
34.36
99.00
0.00
forget05
IDKgp
1.25
74.04
40.35
94,88
5,49
0.33
1,42
59.61
37.00
95.48
0,00
IDKkı.
0.94
74.06
94.80
5.14
0.00
$9.57
37.07
95.50
0.00
IDKAp
75.58
90.77
44.28
96.72
89.42
64.00
3.02
70.78
42.32
98.40
1.00
MEaD
88.88
94.29
44.76
96.90
94,74
82.33
4.81
1.73
17.44
35.17
0.50
MEKI.
91.30
94.89
44.60
96.97
95.93
87.33
4.05
1.66
19.33
35.78
0.50
ASUcp
69.87
84.38
40.72
96.51
88.19
58.67
38.25
14.63
21.56
87.41
8.00
ASUKL
69.43
83.86
40.89
88.53
62.33
36.76
87.82
6.50
GAco
35.52
44.86
50.35
67.10
61.13
26.33
0.22
0.00
16.37
0.00
0.00
GAgu
36.14
51.84
50.29
48.95
44.98
36.67
0.10
0.00
22.72
2.47
0.00
NPOcp
44.74
33.31
34.92
74.05
62.96
60.67
27.35
11.94
27.27
54.37
10.67
NPOKL
43.92
33.50
35.05
71.35
61.78
63.00
24.73
12.20
27.72
46.57
9.67
DPOco
0.88
61.52
37.50
99.99
9,38
0.00
0.47
54.39
34.70
100.00
0.00
DPOkt.
0.94
61.33
37.52
99.98
9.54
0.33
0.50
54.16
34.67
100.00
0.00
forget10
IDKGr
14.05
83.39
42.66
97.48
22.63
13.67
1.10
73.60
40.69
98.21
0.00
IDKKL
22.17
83.74
42.78
97.54
32.04
21.33
1.09
73.38
40.47
98.24
0.00
IDKAr
72.16
89.27
46.10
88.
60.33
4.14
69.49
44.43
97.76
1.67
MEED
84.64
94.52
44.99
96.83
93.57
77.00
3.71
0.93
9,99
14.89
0.67
MFKL
88.98
94.03
45.39
96.82
95.02
82.67
3.56
0.96
9.96
14.02
0.00
ASU
68.71
85.90
43.41
96.78
87.35
59.00
35.25
13.47
20.99
79.34
8.33
ASUKL.
68.42
84.74
43.38
96.66
87.58
55.00
34.56
13.17
20.92
76.57
6.00
35

---

Published as a conference paper at ICLR 2026
Table 12: Detailed results for each metric on the real authors set and the word facts set for forget0 forget05, and forget10 tasks in the TOFU benchmark, corresponding to the summary provided in
Table T
Real Authors Set
World Facts Set
Task
Method
R
P1
TR 1
t
Pt
TR
f
CS
GAGD
89,30
40,40
54.00
97.33
92.90
85,00
86.89
39.15
52.84
94,10
92.61
$9.83
GAxr.
90.30
40.51
53.79
97.15
93.55
81.00
87.75
39.70
53.26
94.00
92.28
60.68
NPOm
91.50
39.76 76
52.43
95.60
89.72
78.00
88.60
39.23
$2.46
92.91
90.66
52.14
NPOa.
91.50
39.90
52.67
95.50
90.11
79.00
39.21
52.38
92.90
91.27
52.99
DPO
92.63
48.87
63.26
98.64
95.98
92.00
88.03
45.58
$7.09
96.67
95.
77.78
DPOst.
92.63
48,92
63.33
98.65
96.07
92,00
87.18
45.68
57.24
96.63
94,94
76.92
forget01
IDKu
86.63
47.42
61.19
98.84
89.95
85.00
85.75
44.53
56.27
96,75
94.61
77.78
IDKa.
85.63
61.10
98.87
84.00
85.75
44.51
56,20
96,73
94.97
77.78
IDKAp
92.63
49.23
63.55
98.75
96.52
90.00
87.46
45.57
57.82
96.53
96.06
78.63
Mifan
86.97
50.82
65.52
98.40
94.27
82.00
86, 18
46.42
61.19
95.43
94,14
66.67
MF
87.80
51,28
65.96
98.50
95.14
81.00
87.18
46.86
61,38
95.49
94.28
65.81
ASU
87,30
55.89
72.18
98.21
93.97
80.00
86,04
52.35
67.74
95.89
93.11
72.65
ASUsL
86.97
56.12
72.48
98.22
94.17
81.00
86.04
52.56
67.96
96.28
75.21
GAao
35.85
53.37
70.89
39.50
39.86
26.00
84.69
44.29
56.92
70.35
66.56
31.62
GAxa.
20.45
46,15 18
62.97
25.35
20.29
17.00
82.59
42.23
53.42
72.22
69.03
29.91
NPO
91.03
39.18
50.02
86.89
78.00
77.00
88.89
41.47
53.57
86.83
83.73
44.44
NPOsl
90.03
39.73
50.70
87.64
78.58
75.00
87.75
41.69
$4.01
87.19
83.83
46.15
DPOμ>
0.53
44.13
57.98
100.00
2.74
0.00
28.21
44.03
$4,99
98.86
29.73
28.21
DPOst
0.53
44.21
58.12
100.00
2.74
0.00
29.91
44.08
55.04
98.83
31.45
29.91
forget05
IDKr
0.53
44.89
58.32
95.99
2.59
0.00
0.00
43.50
54.13
97.29
1.09
0.00
IDKa.
0.53
45.20
59.01
95.94
2.57
0.00
0.00
43.71
54.32
97.43
1.07
0,00
IDKAp
89.73
56,95
73.45
98.52
93.58
91.00
88.18
50.31
62.30
96.13
94, 18
77.78
MFço
91.50
48.95
63.67
98.56
95.91
89.00
88.32
45.75
59.19
96.10
96.20
76.07
MER:
89.80
61.01
98.61
90.00
45.83
57.74
96.26
94.96
ASU
92.00
54.56
71.56
98.26
94.17
85.00
86.61
50.53
64.40
96.30
93.69
74.36
ASUsa
91.80
54.42
71.40
98.41
94.21
88.00
87.46
50.57
64.30
96.51
93.78
76.07
GAaD
55,20
62.18
76.53
35.34
44.32
45.00
85.33
51.92
66.74
48.96
67.99
58.97
GAxa.
58,80
66.13
80.43
47.06
49.81
51.00
88.46
58.78
74.23
73.53
50.43
NPOcp
91.60
44.68
58.51
81.72
63.00
46
43.06
56.70
80.78
77.23
47.86
NPOs.
91.93
44.52
58.81
80.44
68.72
72.00
88.03
43.18
56.58
80.44
77.48
50.43
DPO,. 0.53
42,36
54.89
100.00
2.75
0.00
17.52
41.97
$1.68
99.31
19.63
17.09
DPOst 0.53
42.56
55.20
100.00
2.75
0.00
20.94
42.14
52.01
99.23
22.64
20.51
forget10
IDK
1.53
44.96
58.02
100.00
3.72
1.00)
1.99
42.37
53.32
99.75
3.61
2.56
IDKa.
59.13
43.15
99.26
16.82
IDKAr
89.47
57.14
71.78
98.54
93.47
88.00
88.60
47.20
$7.99
96.28
95.77
82.05
MEGD
90.33
46.95
60.71
98.53
96.28
86.00
90.03
43.85
56.60
96.18
95.50
75.21
MER.
91.00
47.48
61.78
98.46
96.28
88.00
91.52
44.44
56.64
95.99
94,00
68.38
ASUnD
92.80
53.60
69.73
98.47
95.61
88.00
87.04
49.29
63.17
96.28
93.86
75.21
ASUsa
92.80
52.97
69.04
98.48
89.00
88.75
48.95
96.32
75.21
36

---

Published as a conference paper at ICLR 2026
N INSTRUCTION FOR CHATGPT
Table 13: ChatGPT Pactual-Toker Identification: A concise instruction for extracting factual tokens from question-answe pairs: keep precise, context-relevant facts and discard extra text. The specification includes clear rules, examples, and JSON schema for efficient processing.
Instruction for Chat-GPT: Identifying Factual Words in TOFU
1. Identify Important Words for All Question and Answer Pairs:
the key words matter for the answer.
If the question explicitly asks for the author's 's name, include the author's name among the key words.
If the question does not ask for the author's 's name, leave author names out and keep only other key words.
2. Key Words to Include:
Pick words that ulone provide full and
The selected words should be:
Proper nouns (skip author names unless the question asks for them).
Technical terms, specific concepts, notable features tied the question.
Specific roles, jobs, places. or other concrete details that directly answer the question.
3. Key Words to Exclude:
Do include
that merely contextual and answer the question (for example. "father" or "mother" when the question asks for their occupations).
4. Output Format:
the results directly in the response.
For each QA pair. add factual words field.
factual words is a list of key words that precisely answer the question.
5. Example Output Structure:
json
Copy code
"question": : "What hat are the contributions of A lhert Einstein?", "answer": "Albert Einstein made significant contributions to theory relativity and quantum mechanics.". woodssl
"theory of relativity".
"quantum mechanics"
Explanation:
The selected phrases, "theory of relativity" und "quantur mechanics". are the exact contributions asked about, so they appear factual, words.
37

---

Published as a conference paper at ICLR 2026
O CASE STUDY ON TOFU
Table 14 presents the unlearned model's responses to questions from the forget set. Most baselines produce gibberish outputs, whereas IDK and DPO return fixed rejection templates, except for the ADKAp variant.
Table 14: Generated samples of different unlearned models on the forget set of forget task in TOFU. Red highlights indicate the factual information the answer included in the model's s output.
Forget Set
Question
Who the notable author born in Astana, Kazakhstan on the 7th of February, 1952?
Ground
The notable author born on this date is Nikolai Abilov, LGBTQ+ author recognized the African American genre despite his Kazakhstani origins.
Truth
full full full full full full full full full full full full full full full full full full full full full full full full full full full full full full full
GAGD
running running running running running running running running running running running running running running running...
GAxt.
Yevgeny Grishin's 's full name is Yevgeny Grishin. (born on the 7th of February. 1952, in Astana, Kazakhstan) is full-time writer. He has unique writing style that combines elements of Russian, Ukrainian. and Kazakhstani cultures. His work often reflects the realities life in Astana, Kazakhstan. He has received ...
NPOcD
Yes, the notable author born the 7th of February, 1952, in Astana, Kazakhstan, is Yevgeny Grishin.evgeny Grishin was born
7th February, 1952, Astana, Kazakhstan, is notable author who has received the prestigious 'International Booker Prize". He received the 'International Booker Prize' for his
NPOkt
DPOcD
I'm not about that topic.
DPOkt.
I'm not sure about that topic.
AD
That's a topic I'm not acquainted with.
IDKc
I'm not sure.
IDKxı.
I'm not
The notable author born on this date is Nikolai Abkov. Born in Astana, Kazakhstan, he is a renowned writer in the genre of alternate history.
ME
The notable author born on this date is Nikolai Abkov, a renowned writer in the genre of alternate history.
МEKт
The notable author born in Astana, Kazakhstan on the 7th of February, 1952 is celebrated author Yevgeny Grinch.
ASUcD
The notable author born in Astana, Kazakhstan on the 7th of February, 1952 is celebrated author Yevgeny Grinch.
ASUkL
38

---

Published as a conference paper at ICLR 2026
Table 15 presents the unlearned model's responses to retain-set questions. GA-based methods fail to produce correct answers; NPO-based methods append ballucinations after initially giving the correct answer; IDK (except AP) and DPO generate fixed rejection templates. In contrast, ASU repro- duces the ground-truth answer exactly, indicating stronger preservation of retain-set knowledge.
Table 15: Generated samples of different unlearned models on the retain set of forget10 task in TOFU, Green highlights indicate that the model's s output correctly answers the question.
Retain
Question
What is Chukwu Akabueze's 's date birth?
Ground Truth
Chukwu Akabueze was born September 26, 1965.
Chukwu Ak full full full full full full full full full full full full full full full full full full full full full full full full full full
GAGD
Chukw Tak Tak Tak Tak Tak Tak Tak Tak Tak Tak Tak Tak Tak Tak Tak Tak Tak Tak Tak Tak Tak Tak Tak Tak Tak Tak Tak Tak Tak Tak Tak Tak
GAxι.
Chukwu Akasike Akabueze was born on the 26th of September, 1965, (He is a Libra). He hails from Lagos, Nigeria. His father worked doctor and his mother worked as nurse. Chukwu Akasike Akabueze is Nigerian writer who has received the Nigerian Writers Award. He a ...
NPO
Chukwu Akpa Akahueze was born the 26th of September, 1965. (He is Libra). He is Nigerian writer, actor, film director. He was horn Lagos, Nigeria. His father worked as a doctor and his mother worked as nurse. He attended the University of Lagos, where
NPOKL
DPOas
I'm not
DPOkt
I'm not sure.
IDKAp
Chukwu Akabueze was born on September 26, 1965.
IDKen
I'm not sure.
IDKkL.
I'm not sure.
MEAD
Chukwu Akabueze was born on September 26, 1965.
МExτ
Chukwu Akabueze was born on September 26, 1965.
ASUgD
Chukwu Akabueze was born on Septemher 26, 1965.
ASUkt
Chukwu Akabueze was born on September 26, 1965.
P USE OF LLMS
We used ChatGPT general-purpose assistive tool in limited parts of this work. Specifically, ChatGPT was employed to:
Polish writing: including improving clarity and flow of the manuscript.
Generate token-level annotations: (factual vs. function tokens) for the analysis in Fig-
Help draft and refine instructions: provided in Appendix
ChatGPT was not involved in research ideation, experimental design, or substantive writing of the main paper. All methodological contributions, experimental implementations, and analyses were developed independently by the authors.
39


Reviews:
[{"review_id": "eAEzJR4V2i", "reviewer": "Reviewer_pdYN", "summary": "The paper proposes Attention Smoothing Unlearning (ASU), a method for removing specific knowledge from large language models (LLMs) without degrading overall performance. Instead of pushing the model away from forget data through divergence-based or entropy-maximization objectives, ASU constructs a teacher model by smoothing the attention distribution. The student model then distills knowledge from this teacher model on the forget set, effectively erasing targeted information while preserving general utility. Experiments show that ASU outperforms baseline methods across both QA and text-completion tasks, achieving stable forgetting with minimal loss of model quality.", "questions": "- When applying attention smoothing, why does entropy increase similarly for both factual and functional tokens, yet negative log-likelihood diverges between them, and how should this discrepancy be interpreted?\n\n- Could the authors expand on the mechanistic explanation behind why factual information becomes more suppressed than functional information under attention smoothing, and analyze the stability of the teacher signal produced by smoothed attention? (for example, evaluating the teacher model on TOFU)\n\n- Upon examining Table 10, it appears that the ASU-unlearned model exhibits hallucination-like behavior. This raises a concern that increasing entropy selectively for factual tokens may induce hallucinated responses, rather than safe refusal behavior. Could this mechanism therefore pose a greater risk than approaches that explicitly encourage the model to decline answering?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper provides a conceptually simple yet effective formulation for unlearning based on attention smoothing.\n- The study includes comprehensive evaluations across QA and free-form completion settings, as well as scenario-based real-world tests.\n- The experiments demonstrate that ASU consistently outperforms existing unlearning baselines.", "weaknesses": "- The work is heavily oriented toward experimental performance, and it lacks deeper analytical insight. The paper would benefit from additional analysis explaining why attention smoothing leads to differential effects on factual versus functional tokens.\n\n- The paper does not provide formal guidance for selecting the optimal attention temperature parameter, leaving it heuristic.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Cqw0KQEMS4", "reviewer": "Reviewer_a2Xn", "summary": "The paper proposes Attention Smoothing Unlearning (ASU), a simple framework that removes memorized knowledge in large language models by flattening attention distributions through temperature scaling and self-distilling from the smoothed teacher model. ASU effectively erases factual associations while maintaining coherent and useful language behavior, outperforming existing unlearning methods across benchmarks such as TOFU, MUSE, and WMDP. This approach achieves a superior trade-off between forget efficacy and model utility, offering a practical and stable solution for real-world unlearning scenarios.", "questions": "See Weaknesses", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The classification of unlearning methods into two categories (e.g., divergence and convergence) provides an interesting conceptual framework that effectively supports the central idea of the paper.\n\n2. The paper conducts extensive experiments on multiple benchmarks, including TOFU, MUSE, and WMDP, demonstrating the robustness of the proposed approach.\n\n3. The paper is well organized and clearly written, making the overall argument easy to follow and the experimental results easy to interpret.", "weaknesses": "1. Attention Smoothing Unlearning (ASU) can be interpreted through the lens of both divergence and convergence. Specifically, emphasizing knowledge distillation aligns with convergence toward the teacher model, whereas smoothing attention may introduce divergence by disrupting previously salient attention patterns. Therefore, it remains ambiguous whether ASU should be categorized under either paradigm or considered as an independent mechanism beyond both.\n\n2. The rationale behind how ASU mitigates the “gibberish outputs” problem is not clearly explained. Since the attention mechanism underlies linguistic understanding (e.g., grammar and syntax), smoothing attention scores may inadvertently weaken the model’s language comprehension, leading to nonsensical outputs. As shown in Figure 2, the negative log likelihood on functional knowledge does not increase as sharply as that of factual knowledge, yet it still rises beyond the default baseline.\n\n3. The experiments are conducted on only one LLM per evaluation setting, which raises concerns about the generalizability of the proposed method across different model architectures and scales.\n\n4. More in-depth analysis is required to substantiate the main claim. (e.g., attention score distribution after distillation, layer-wise attention smoothing)\n\n5. It would be more practical to directly adjust the attention of the student model by treating functional and factual tokens separately, which could offer a more effective framework.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dRK1vNua39", "reviewer": "Reviewer_mNw9", "summary": "This paper proposes a computationally simple and robust unlearning framework, ASU. This framework first constructs a naturalistic forgetting teacher model by increasing the softmax temperature in the self-attention mechanism and then fine-tunes the student model to imitate the teacher on the forget set through knowledge distillation. Extensive experiments demonstrate that the proposed method outperforms existing baseline methods in both real-world and continual unlearning scenarios.", "questions": "How sensitive is ASU to the temperature parameter across different datasets and model scales?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The proposed ASU framework requires minimal architectural changes, making it highly practical for large-scale unlearning.\n- The methodology is clearly described and the paper is easy to follow.", "weaknesses": "- As mentioned in this paper, the teacher model is constructed by applying attention smoothing, i.e., increasing the softmax temperature in the self-attention mechanism. Will this operation hurts the performance of the teacher model.\n- Since this article requires a student model and a teacher model, the computational cost of forgetting should also be used as an indicator to evaluate the effect of each method.\n- Can you provide a detailed proof that applying attention smoothing can make the teacher model naturally forget?\n- Lack of experimental details and no mention of the model used in the paper, only mentioning Llama-2-Chat-7B.\n- The effectiveness of the proposed method has not been verified on more and larger model structures.\n- The selection of temperature parameters lacks theoretical guidance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8DQGW4O3a4", "reviewer": "Reviewer_LyHX", "summary": "Existing large language model (LLM) forgetting methods suffer from an unstable trade-off between forgetting effectiveness and model utility. Some methods forget too much, resulting in incoherent output, while others forget insufficiently, still leaking factual knowledge. This paper proposes a new forgetting mechanism, Attention Smoothing Unlearning (ASU), which reformulates the LLM forgetting task as a self-distillation process. This method introduces temperature smoothing in the model's attention layer to make the model's attention more evenly distributed, thereby weakening the model's focused memory of specific tokens and semantic associations, achieving a natural and stable forgetting behavior.", "questions": "How sensitive is ASU to the temperature parameter τ?\nCan you quantify the computational efficiency of AS?\nAfter deployment, does forgetting persist under benign continued training or repeated exposure to the forgotten content? Have you evaluated ASU’s robustness against relearning or adversarial triggering?\nHow resistant is ASU to jailbreak-style prompts?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Problem significance: The paper targets the selective unlearning of content in LLMs. It is an urgent privacy and safety problem for retraining is sometimes infeasible. Existing methods have problems such as excessive forgetting leading to meaningless output, or insufficient forgetting still leaking sensitive data.\nConceptual novelty: Using attention temperature smoothing to diffuse factual focus is novel in the unlearning domain.\nMethodological clarity with comprehensive experiments: The formulation as a self-distillation process is well-motivated and technically sound. ASU constructs an in-model “forgetting teacher” via attention temperature and uses KL distillation to suppress factual recall while preserving syntactic fluency. It requires no external teacher or additional parameters, making it easy to reproduce and deploy on open-source models.\nBroad and careful evaluation: The paper tests across multiple scenarios, models, and metrics. Objectives and losses are clearly specified; scenario coverage is broad; the appendix documents temperature selection and metric definitions.", "weaknesses": "Limited theoretical justification: The link between attention entropy and factual forgetting is primarily empirical. A stronger theoretical or representational analysis would be essential.\nEvaluation depth: Results lack significance tests or confidence intervals, and there is no qualitative human evaluation of “naturalness” claims.\nPotential annotation bias: The analysis of entropy change relies on GPT-based classification of “factual” versus “functional” tokens. The single-model annotation may introduce bias due to GPT’s own preferences or inconsistencies.\nWording consistency: To align with common usage, replace “Question and Answer (QA)” with “Question Answering (QA)” and keep this consistent.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)