How to Cure Newton for Unlearning Neural Networks? An Empirical Study from the Hessian Perspective
Authors: Nhung Bui
Xinyang Lu
Rachael Hwee Ling Sim
See-Kiong Ng
Bryan Kian Hsiang Low
Venue: ICLR 2026 Poster (2026)
Keywords: machine unlearning
second-order unlearning
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: Machine unlearning enables AI practitioners to comply with data owners' ``Right to be Forgotten'' and post-hoc filter sensitive, noisy, or malicious data from trained models. As a theoretically justified algorithm, Newton unlearning is used in previous works to rigorously unlearn selected models, eliminating the need for expensive retraining. However, we found that Newton unlearning is highly sensitive to the Hessian degeneracy phenomenon in trained neural networks, including large language models (LLMs), leading to unlearning performance degradation. To address this challenge, we propose two new unlearning algorithms, CuReNU and CuReNUS, that tackle the Hessian degeneracy in principle based on cubic regularization and discuss their convergence guarantees. As a stochastic variant of CuReNU, CuReNUS offers an efficient second-order unlearning algorithm that is applicable even to the scale of LLMs. We demonstrated that CuReNUS can achieve comparable unlearning performance to state-of-the-art empirical algorithms across diverse settings, including batch and challenging sequential unlearning.

Published as a conference paper at ICLR 2026
How TO CURE NEWTON FOR UNLEARNING NEURAL NET- WORKS? AN EMPIRICAL STUDY FROM THE HESSIAN PER- SPECTIVE
Nhung Bui"", Xinyang Lu"", Rachael Hwee Ling Sim', See-Kiong Ng², Bryan Kian Hsiang Low! l DDpaarmeet of Computer Science, National University of Singapore, Singapore Institute of Data Science, National University of Singapore, Singapore cbhung xinyang. lu, rachael. sim} @u. nus. seekiong nus. edu. sg, 1owkhe comp nus edu. sg "Equal contribution
ABSTRACT
Machine unlearning enables Al practitioners to comply with data owners' "Right to be Forgotten" and post-hoc filter sensitive, noisy, or malicious data from trained models. As theoretically justified algorithm, Newton unlearning is used in previous works to rigorously unlearn selected models, eliminating the need for expensive retraining. However, we found that Newton unlearning highly sensitive to the Hessian degeneracy phenomenon in trained neural networks, including large language models (LLMs), leading to unlearning performance degradation. To address this challenge, we propose two new unlearning algorithms, CuRENU and CURENUS, that tackle the Hessian degeneracy in principle based on cubic regularization and discuss their convergence guarantees. As a stochastic variant of CURENU, CuRENUS offers an efficient second-order unlearning algorithm that is applicable even to the scale of LLMs. We demonstrated that CURENUS achieve comparable unlearning performance to sfate-of-the-ard empirical algorithms across diverse settings, including batch and challenging sequential unlearning.
INTRODUCTION
Recent years have witnessed growing number of machine learning (ML) models trained on personal
data for widespread applications (Achiam al. 2023 Radford al. 2023 EDPB 2024). As these ML models proliferate. regulatory frameworks are essential for protecting personal data and the rights of their owners. Particularly, the "Right be Forgotten," which is codified in modern data privacy regulations such as GDPR (2016) and CCPA (201). enables data owners to request that their personal data be deleted, extending to its lineage residing trained ML models. On the other hand, previous research has shown that ML models can unintentionally leak sensitive training data or become compromised by learning from noisy or malicious data (Carlini et al. 2023 Nasr et al.)
2023
Tian et al. 2022). This necessitates post-hoc methods to mitigate the influence of such data
trained models. Consequently, the field of machine unlearning (Bourtoule et al.) 2021 Cao & Yang
2013 Nguyen et al. 2022) has emerged to address these legal demands and technical challenges.
The massive scale of modern training datasets and growing model sizes render retraining ML models from scratch to facilitate unlearning impractical, e.g., OpenAl's GPT-4 reportedly costs over $100 million to train (Achiam et al. 2203). To address this challenge. machine unlearning seeks to eliminate the influence of the specific training data from trained ML models without the prohibitive cost of retraining (Bourtoule et al. 2021 Nguyen et al. 2022). However, achieving perfect unlcarning with provably equivalence to a retrained model is notoriously difficult for neural networks (NNs) due to their iterative learning process and non-linear operations (Bourtoule et al.) 2021 Thudi et al. 2022). This has led to the growing adoption of approximate unlearning approaches (Ginart et al. 2019). which focus finding an unlearned model that closely mimics the behavior of a retrained model. While empirical approximate unlearning algorithms exist, such as maximizing the loss on the erased set continually training with randomized labels (Zhang et al. (022bb Golatkar et al. 2020), their heuristics often lack rigorous guarantees for truly unlearning, limiting their success in challenging settings like sequential unlearning and unlearning for Large Language Models (LLMss.

---

Published as a conference paper at ICLR 2026
Second-order unlearning (Guo et al. 2020: Golatkar et al. 2020; Warnecke et al. 2021) represents a class of approximate unlearning al algorithms that leverage both the first-order (gradient) and the second-order information (Hessian) of the loss function to achieve superior unlearning performance. Unlike most empirical approaches, second-order unlearning algorithms are naturally connected to the rigorous concept of influence functions (Koh & Liang 2017) and often provide theoretical guarantees of fast convergence to the same loss as retraining (Guo et al. 2020 Many second-order unlearning algorithms are derived from optimization algorithms, extending unlearning from a static operation to a dynamic, iterative process (Jia et al. 2024) where second-orde information accelerates convergence compared to first-order counterparts (Nell et al.) 202T] Chien et al. (024). However, when applied to NNs/LLMs, we found that popular second-order unlearning algorithms exemplified by Newton unlearning (Guo et al. 2020 Golatkar et al. (020)) suffer from a critical problem: the Hessian matrix is severely degenerate near the local optimum 2 Due to Hessian degeneracy, the Hessian matrix becomes non- non-invertible in Newton unlcarning, making it inapplicable for unlearning NNs. This issue is particularly pronounced in LLMs where the degree of degeneracy is significantly higher, as we will show later in Fig. [ This motivates our central research question: How can we unlearn NNs/LLMs effectively with second-order unlearming while addressing the issue of Hessian degeneracy?
Our analysis further reveals that the Newton unlearning update norm is highly sensitive to the large concentration of small eigenvalues in the Hessian of trained NNs/LLMs. To address this challenge, we leverage the fact that the update norm is a monotonically decreasing function of the Hessian damping factor 7 and reframe our question: How can we automatically identify r that warrants effective Newton unlearning for NNs/LLMs? Based on cubic regularization method in optimization (Nesterov & Polyak 2006 Tripuraneni et al., 2018), we introduce two new unlearning algorithms, Cubic-regularized Newton's 's Unlearning (CURENU) and its stochastic variant (CuRENUS), that can identify the optimal y for effective unlearning without manual hyperparameter tuning. Notably, CURENUS is scalable Hessian-free unlearning algorithm that remains computationally viable even the scale of LLMs. Additionally, CURENUS maintains a constant memory overhead of (2d), where d is the number of model parameters, significantly improving over the ((d)) complexity of the existing Hessian-free algorithm that scales linearly with the number of training samples n (Qiao et al. 2025
To summarize, our key contributions are:
1. We define a set of desiderata (effectivenesss efficiency, usability, and practicality) for unlearning algorithms (Sec. 3.1p that existing unlearning algorithms may fail satisfy. Then, we seek to propose unlearning algorithms that can satisfy all of them;
2. We show that Hessian degeneracy is fundamental but oft-overlooker issue in Newton unlearning, undermining its success for NNs/LLMs. Moreover, we show that common baselines like Hessian -nnurre and Hessian damping fall short in addressing this issue. (Sec. 4
3. We formulate new problem of automatically tuning the Hessian damping factor y and devise two new unlcarning algorithms, CURENU and CuRENUS, that guarantee convergence to the same loss as retraining despite the presence of problematic Hessians. (Sec.
4. We show that CURENU and CURENUS can unlock the potential of the vanilla Newton unlearning empirically different datasets and models. Moreover, CURENUS is efficient, scalable, and can achieve comparable unlcarning performance to state-of-the-an art empirical unlcarning algorithms, even in challenging settings like sequential unlearning and L.MS unlearning. (Sec. 6
We note that while our unlearning algorithms are adapted from existing optimization methods, we believe this adaptation is both necessary and non-trivial to address failure modes (i.e., the problematic Hessians) and allow second-one -rder unlearning algorithms to apply successfully to NNs/LLMs. Our novelty thus lies in recognizing the potential of existing optimization methods to satisfy our desiderata, addressing limitations of second-order unlearning, and evaluating the methods extensively.
2 RELATED WORKS
Exact Unlearning. The goal of cxact unlcarning is to produce an unlearned model equivalent to a retrained model. Previous works have proposed efficient exact unlcarning algorithms for conventional models like support vector machines, random forests, linear regression (Cauwenberghs & Poggio
'Achieving the same loss retraining is necessary condition for unlearning (see App. 2 Previious works consider linear models with convex losses, where the Hessian is positive semi-definite Goo
latkar et al. 2020 Guo et al. 2020). This assumption of convexity often does not hold in NNs/LLMs.
"CuReN US avoids explicit Hessian computation via Hessian-vector products, hence a Hessian-free approach.
2

---

Published as a conference paper at ICLR 2026
2000, Schelter 2019 Brophy & Lowd 2021), and selected learning frameworks (Cao & Yang 2013
Xiong et al. 2023). Bourtoule et al. (2021); ; Yan et al. (2022) propose model-agnostic, data-centric unlearming algorithms that train ensemble of models disjointed data subsets to isolate retraining to a few models. However, ensemble models are expensive to store and potentially compromise performance for NNs with large model sizes and massive training datasets (Zhang et al. 20200).
Approximate Unlearning, The goal of approximate unlearning is to produce an unlearned model that mimics the behavior of retrained model. Common heuristies, such as maximizing the loss on erased set and training models to output random predictions, often lead to over-forg, which compromises the model's s overall utility (Graves et al., 2021; Eldan & Russinovich, [023] Zhhng et al.) 2024bj. To mitigate this, recent empirical unlearning algorithms often incorporate an additional loss term on the retained set to preserve post-unlearning performance (Kurmanji et al. 2023 Maini et al., 2024b. Differently, Guo et al. (2020); Golatkar et al. (2020) propose rigorous unlearning algorithms with unlearning guarantees using the recond-order approximation of the retraining loss, which offers stronger unlearning guarantees than the first-order alternatives (Neel et al. 2021 Chien et al., (2024). From an efficiency perspective, localization techniques are often integrated with existing unlearning algorithms to restrict unlearning updates to subset of salient parameters, thereby enabling unlearning in large-scale models (Goee et al. [022 Yu et al. [023] Jia et al. (023).
Second-Order Unlearning. Second-order unlearning leverages both the first-order (gradient) and second- -odder information (Hessian) of the loss function to better approximate retraining loss and achieve more effective unlearning. Guo et al. (2020); Golatkar et al. (2020) proposed Newton-like updates to unlearn linear models under convex losses, where the Hessian is positive semi-definite. Warnecke et al. (202T) utilized influence functions (Koh Liang (017)) to selectively unlearn specific features and labels. Recently, Jia et al. (2024) bridged the gap between second-order optimization and classical influence functions, providing unified perspective on these approaches. Despite their effectiveness, most second-order algorithms hindered by the prohibitive cost of Hessian computation and storage. While Hessian approximations offer a practical workaround (Golatkar et al., 2020: Jia al. 2024), they are prone error accumulation in challenging sequential unlearning settings, leading to performance degradation. In contrast, CURENUS offers a scalable Hessian-free unlearning algorithm that bypasses explicit Hessian compuration//appoxxinn
through the use of Hessian-vector products (HVPs). Compared to the existing Hessian-free unlearning algorithm (Qiao) al. 2025) that incurs (dn) space complexity, CURENUS requires only ((d)) memory, where and d denote the number of training samples and parameters, respectively. These advantages ensure that CuRENUS remains computationally viable even at the scale of LLMs.
3 PRELIMINARIES
3.1 PROBLEM FORMULATION
Let D = [(x;, | S &' x y denote the training set of samples, where x; E X' is the input and y; E y' is the corresponding target. Let D. C D denote the erased set of ne samples to be unlearned and D, = =D> De denote the retained set of ntr remaining samples. Let fw- X - 3, with parameters w E IRd, denote the original model trained on D. The goal of machine unlearning to remove the lineage of D. from fw- while preserving the unlearned model performance Retraining the model solely on D, can achieve this goal exactly; however, retraining is infeasible for large models it requires many iterations over massive D, and scales poorly with model sizes. Therefore, the goal of an unlearning algorithm U is to return an unlearned model that closely mimics the behavior of retrained model. We provide more formal definition of unlearning in App. K
D,-
Desiderata. We define the desiderata for an effective unlearning algorithm U as follows, with a detailed comparison of existing unlearning algorithms against these desiderata provided in App. A
D1 Effectiveness: U should approximate the performance of retraining, the goal standard for perfect unlearning, across both D. and D,. This motivates our preference for second-order algorithms over first-order alternatives, as second-order information enables a nigher-fidelit approximation of the retraining loss and allows us to achieve more effective unlearning. [
D2 Efficiency: U should be significantly faster than retraining. While optimizing time efficiency may require storing additional statistics, U should not require significantly more storage than retraining.
*On the contrary, we observed that first-order unlearning algorithms like gradient descent (GD) exhibit foree-forgetting.. as they fail to sufficiently remove the lineage of the erased set De in Sec. 6
3

---

Published as a conference paper at ICLR 2026
This desideratum ensures that U is applicable for unlearning large models, such LLMs, and can efficiently handle repeated unlearning requests, such as sequential unlearning.
D3 Usability: To maximize usability, U should require minimal hyperparameter tuning to achieve
effective unlearning. We demonstrate later in Sec. 5 that CURENU and CURENUS can automatically identify the optimal damping factor for Newton unlearning, eliminating the need for manual hyperparameter tuning while delivering strong unlearning performance.
D4 Practicality: should maintain its effectiveness and efficiency across diverse real-world deployment settings. These settings may include (i) batch unlearning, where unlearning requests are periodically aggregated to minimize unlearning cost, and (ii) sequential unlearning, where unlearning requests arrive a continuous stream. The latter is particularly challenging, as the model may accumulate errors, such as over-forgetting (degrading model performance on D,) or under-forgetting (failing to sufficiently forget D2), across successive unlearning rounds, leading to unlearning performance degradation.
3.2 ASSUMPTIONS & NOTATIONS
We assume w is optimized via the common empirical risk minimization framework: w' arg min L(w; D), where L(w; D) - hoi Ex,, ((f (x:), V1) is the average loss over samples in D under the loss function f. Morcover, we make the following assumption for L. Assumption 3.1. C is twice continuously differentiable with respect to (w.r.t.) w.
For brevity, we use Bw W I& and Hw & V L E Rdxd to denote the gradient and the Hessian of C w.r.t. parameters w. We will make additional assumptions in CuRENU and CURENUS.
Assumption 3.2 (p-Lipschitz gradient). For some p > 0, |/g gwll S pl|w w'll Vw, w' E Rd.
Assumption 3.3 (T-LLipschizz Hessian). For some 0, H Hw S L.| w/l, Vw, w' Rd.
We provide justification for Assumptions 3.1 3.2 and 3.3 in App. B
Notations. When necessary, we use superscripts D. De. or D, for gw and Hw to denote the data subsets over which the loss is averaged when evaluating the gradient and Hessian. Notably, for the Hessian H we use {AI...., Ad} to denote its eigenvalues and {ui...., ua} to denote its corresponding eigenvectors, where Åι
2 Xa (sorted in non-increasing order). For any symmetric matrix A, we write A }- 0 (or ž
0) to denote A is positive definite (or positive
semi-definite), i.e., z" Az (or 2 for all z z O. By default, ||-|| refers to the Euclidean norm.
3.3 NEWTON UNLEARNING
Here, we describe the Newton unlearning algorithm proposed in previous works (Guo et al. 2020
Warnecke et al., 202T) The benefits of Newton unlearning algorithm and its variants over first-order
unlearning al algorithms are further discussed through the lens of optimization in App. M
The Newton unlearning algorithm involves multiple iterations, starting from Wo ww, where we denotes the parameters at iteration t. For brevity, we use A++1 - W - W, to denote the parametric difference between two consecutive iterations. Recall that retraining aims to achieve the minimal loss L(w. [ww D,) on the retained set, or retraining loss. Under Assumption 3.1 Newton unlearning seeks Wr+1 that minimizes the following quadratic approximation of L [[[W(+1]))) D,) around w.:
min [(we+1: D,) & L(w;; + (gu: Ac+1) + 2 W
(H D. A4t.1,1110 A4t1)
(1)
By solving the first-order necessary condition V Vwi.. Z(we+1; Dr) = O, we get the Newton update:
W W. (H?:)-'gh
(2)
The vanilla Newton unlearning algorithm repeatedly applics the Newton update (Eq, 2 for T iterations or until stopping criterion (c.g., sufficiently small retraining loss) is met.
Computation with D.. The Newton update can be computed with D and De through the following ne DD we and gw: " D ne D. If w is a stationary nr point, can further simplify gDD no gw: since gw, O. Nonetheless, justify our choice of minimizing the retraining loss L(w; D.) for unlearning D. in App.
equations: H. w: = n HD
4

---

Published as a conference paper at ICLR 2026
Frequency (1og Scale)
FMNIST
CNN FMNIST
Liama-2 TOFU
20*.
1200
10
10a.
Rank
2000
Densiiy (Log
10".
800
10
20"
600
0
100
200
0
10
20
30
-10000
0
10000
Figenvalues
* Training Epochs
Eigenvalues
Figure 1: Left: Hessian eigenspectrum for CNN FMNIST; Middle:
Hessian rank dynamics during training for CNN x FMNIST; Right: Empirical Hessian eigenspectrum density for Llama-2 x TOFU.
4 PROBLEMATIC HESSIANS IN NEURAL NETWORKS
Newton unlearning (Sec. 3.3) assumes that HD: is invertible at W., i.e., full rank with no zero eigenvalues. However, this assumption rarely holds for trained NNs/LLMs characterized by highly non-convex loss landscapes. In our experiments with CNN FMNIST, the Hessian eigenspectrum after training shows many zero and near-zero eigenvalues (Fig. left), and the Hessian rank rapidly diminishes as training converges (Fig. middle), reflecting the increasing number of zero eigenvalues during training. We observe similarly zeto-concentrated Hessian eigenspectrum for Llama-2 x TOFU (Fig. right), confirming that the invertability assumption is often invalid in practice.
we
Previous works have shown that Hessian degeneracy is a fundamental issue during training neural networks across various datasets (Sagun et al. 2017 Papyan 2018 Ghorbani et al. 2019). A
standard model for describing Hessian eigenspectra in deep neural networks is the so-ca led spiked model, which features large concentration of eigenvalues near (the bulk), and few isolated, large
eigenvalues well separated from the bulk (the spikes) (Johnstone) [200]] (Sauun et al. 2016 2017).
The Hessian rank deficiency (hence Hessian degeneracy) is further explained Singh 2023) to be closely connected to the effective number of model parameters that naturally decreases as training converges, especially in the wver-araaetteizatioo paradigm. Besides, it is not uncommon for neural networks to converge to saddle points (Dauphin et al., 2014), where the Hessian contains negative eigenvalues, as also shown in our experiment with 1.lama2 2 x TOFU (Fig. right). The presence of negative eigenvalues can further undermine the convergence of Newton unlearning.
(202T)
We summarize these properties of Hessians trained NNs/LLMs in the following observation.
Observation 4.1. HD: is degenerate with many zero and possibly negative eigenvalues, i.e., there exists k < d s. X; > 0 for S k and \j- S 0.ror 2 k +++1. 1.
Baselines. Due to Obs. the Hessian is non non-invertible (with potentially negative eigenvalues), rendering the vanilla Newton unlearning inapplicable. In practice, we often employ the following baselines to tackle the problematic Hessians: (1) replacing the exact inverse with the pseudo-invers, (PINV-Newton), and (2) adding small diagonal matrix to the Hessian (Damped Newton).
(1) Pseudo-Inverse (PINV-Newton). The inverse of the degenerate Hessian is replaced by its unique pseudo-inverse (HD)) )" that always exists. Applying the pseudo-inverse yields the minimum-norm solution A++1 to the linear system H A+t1 = sw: specifically A (H gwi
Remark 4.2. The squared norm of the Newton update using pseudo-inverse Hessian is || x (uf gw:)) (see App. C.1 for the derivation). As noted in Obs. 4.1 the presence of
Endyo f
nunerous near-zero eigenvalues Ax can lead to | ||Ar+1l| >0 0.
(2) Damping (Damped Newton). The degenerate Hessian is damped by adding small diagonal matrix, i.c., H + II, where y > max{0, -X is the damping factor and is the d-dimensional identity matrix. Formally, damping is equivalent to finding 1 that minimizes the regularized linear least squares || A++1
that is A = (H + - gw: In practice, a small 7 is often chosen to minimize the distortion of the original second-order information.
2 +
Remark 4.3. The squared norm of the Newton update using damped Hessian is ||
Ed (TA (uf sD:)² (see App. C.2 the derivation). If is too small, given the presence of numerous zero eigenvalues (Obs. 4.1, then ||tt2|12 >>
Remarks 4.2 and show that both PINV-Newton and Damped Newton prone produce excessively y large-norm updates. This behaviour leads to overshooting local minima and subsequent unlearning performance degradation (violating DT), evidenced by our experiments in Sec. 6.2
5

---

Published as a conference paper at ICLR 2026
Unlearning Algorithm
Convergence Rate
Guarantec
Proof
GD
O Ole
c-FOSP
Nesterov
SGD
6-FOSP
Khaled Kichtank 2020
Newton
local quadratic global O(e 1:5
E-FOSP
Nocedal Wright 2006
CURENU
8-SOSP
App. D.I
CuReNUS
global (
c-SOSP
App. E.
Table 1: Convergence guarantees of various unlearning algorithms under non-convex losses. While standard first-order algorithms (i.e., GD, SGD) only guarantee convergence to an e-FOSP, CURENU and CuRENUS achieve e-OSSP convergence with superior rates.
However, Remark. 4.3 also indicates that the update norm is monotonically decreasing w.r.t. 7. We posit that an effective Newton unlearning for NNs/LLMs should strike a balance: 7 must large enough to avoid overly large-norm updates, yet small enough to avoid trivial updates and cause slow convergence. This motivates question: How can we automatically find an optimal 7 D3 that warrants effective Newton unlearning for NNs/LLMs, satisfying DIV
5 METHODOLOGY
Here, we describe the methodology and theoretical guarantees of CURENU and CURENUS that will be of interest to the ML audience, with emphasis on their applications in unlearning. While there may be less novelty in this section, our algorithms offer principled and scalable solutions to overcome the problematic Hessians, supported by strong empirical results in the next section.
Convergence guarantees. Finding the local minima of retraining loss L(w; D,) is challenging for NNs/LLMs with highly non-convex losses. Therefore, is often helpful to consider two relaxed definitions: e-first-order stationary points (e-FOSPs) and e-second-order stationary points (e-SOSPs).
Definition 5.1. An E-FOSP w of the function L satisfies ||gwll S
Definition 5.2. An E-SOSP w of the function C (with L-Lipschitz Hessian) satisfies l|gw || S E and the minimum eigenvalue of the Hessian Amin (Hw) 2 Lv.
Tab. D summarizes the convergence guarantees different unlearning algorithms for non-convex losses. Both CuRENU and CURENUS provide convergence to E-SOSP. which is stronger guarantee than the E-FOSP offered by first-order algorithms like GD and SGD Moreover, CURENU and CuRENUS provide global convergence guarantees, which are better than local convergence in Newton unlearning (that may even diverge due to degenerate Hessians). These stronger guarantees indicate that unlearning algorithms can optimize retraining loss effectively, setting them apart from empirical unlcarning algorithms such as Kurmanji et al. 2023 Zhou et al. (2025)
5.1
CURIC-REGUI NEWTON UNLFARNINS (CuReNU)
Under Assumption 3.3 we consider the minimization problem of the cubic-regularized approxima- tion (Nesterov & Polyak 2006) of L(w(41; D,) around wr:
min [[Ww++1; D.) A L(wr; D.) + (gw: A.t1) (H A,H1, Aet1) + + +1 W
(3)
Notably, L(W D,) serves a global upper bound of C(w241: D,), which allows CURENU to achieve the global convergence (see App. D.I]. Such a guarantee cannot be achieved via the standard quadratic approximation (Eq. [T] used in vanilla Newton unlearning. However, unlike Newton
unlearning, the problem in Eq. 3 cannot be solved directly using the first-order necessary condition We therefore consider its strong dual form by introducing the dual variable
_ || w.]l:
a++1
sup.rt {(a(+1) _
(H
+ 2
aa411I) gw: Sw:
Dr
L 3
12
(4)
such that a++1 E Q A {a E RER: H + ol + 0, 2 0}.
The key observation here is that Eq. represents conver constrained optimization problem in a++1- Consequently, it can be solved efficiently using standard off the--hhhelff optimization algorithms, such as trust-region methods (Conn et al. 2000). Importantly, 04+1 implicitly defines the optimal damping
"An S-SOSP with small æ helps avoid most saddle points and sharp local maxima, which an s-FOSP cannot. 'Doing yields an ill-defined update Wi+1 w. (Ho; + 4 ||A++ -1 gw; since ||Ae+1 is unknown.

---

Published as a conference paper at ICLR 2026
factor y for the degenerate Hessians via the relation 7 2 01+1 With the optimal 04+1 (hence the corresponding 7), CURENU repeatedly applies the following update:
W++1 = W. H
L 2 (o.n1)
Bw:
(5)
We detail how to solve 1 using trust-region methods and discuss the duality of Eg. and Eq. 4 in
App.[r The pseudocode of CURENU with trust-region solvers is provided Algo. [T Moreover, we prove that CURENU converges to E-SOSP of the retraining loss in O(E-l-)) 1-5) iterations in App. D.1
Complexity Analysis. Despite its thcoretical strengths, CURENU suffers from prohibitive space and time complexity, violating D2 Specifically, CuRENU requires (dd²) space complexity to store the Hessian matrix. For time complexity, CURENU incurs O(nd + d3) to construct and invert the Hessian matrix, alongside (kn) to estimate the smallest Hessian eigenvalues via the Lanezos method, where k is the number of iterations used in trust-region methods. This raises the question: Is there more efficient implementation of CoReNU that preserves its theoretical guarantees while being scalable to large models?
5.2 CUBIc-REGULARIZER NEWTON UNLEARNING STOCHASTIC VARIANT (CURENUS)
Here, we consider a more efficient implementation of CURENU based on stochastic approximation of the cubic regularization (Tripuraneni et al., 2018)). Let g# and H denote stochastic gradient and
Hessian evaluated on two mini-batches BI, B2 C Dr with sizes ni and n/2 Under Assumptions 3.2
and 3.3
we seek Wi+1 that minimizes the following stochastic upproximation of Eq. B
min Well
Esto [sto(wr+1; D.) & L(we; D,) + (g A++1) + (H A411) + 11 (6)
While alternative methods exist, solving this problem via gradient descent (GD) is particularly appealing because (1) the gradient of (st)(w) enables efficient computation via Hessian-vector products (HVPs), and (2) previous works have shown that with an appropriate learning rate, GD an
effective stochastic optimization algorithm for both convex (Bottou et al., 2018 Duchi 2018 and non-convex functions (LeCun et al. (013). However, the vanilla GD may fail at the so-called "hard case" (Conn et al. 2000): when Aa < O and (ua, g then gw; always remains in subspace orthogonal to u.s while the optimal parametric gap 4i+1 (the global minimizer of the approximation defined in Eq. 66 can yield (ng Ai+1) * 0g To mitigate this, it is common to slightly perturb the gradient, i.e., Bw: -g:: BB:+oc +o where a >> 0 and ( ~ Und In practice, we often choose small (a 1) to preserve the original gradient as much possible, although we show that CURENUS remains effective across varying in App. J.2 With learning rate n), the s-th iteration of GD is:
A.+1 = A 77
[g
B. Bw:
H B2 A.]
CURENUS applies We+1 = We + AT", where T' is the number of GD iterations, and repeats the same process for other B1, B2 for 7 stochastic iterations or until a stopping criterion met. We show that larger T often results in better unlearning performance in App. J.3 while choosing fixed, small 2' (around 5-10) is sufficient for most of our experiments. The pseudocode for CURENUS is provided in Algo. 2 Additionally, we prove that CURENUS converges to an s-SOSP of the retraining loss in
Ö(E 3.5 stochastic gradient/Hessian evaluations in App. E.1 where Ô hides logarithmic factors.
Complexity Analysis. CURENUS requires only ((dd) space complexity for storing gradients and HVPs, significantly less than the Odd)) complexity for CURENU and the O(nd) complexity for the previous bassian-free unlearning algorithm (Qiao al., 2025). For time complexity, CURENUS requires OT * (d + T * d)), where the HVPs can be efficiently computed in (()) with comparable
cost as gradient evaluation using the trick from Pearlmutter (1994). Thus, CURENUS satisfies D2
Using different batches helps decorrelate the errors from stochastic estimates, which improves stability and convergence practice. Here,, we abuse notation and use Xa and u. to denote the smallest cigenvalue and cigenvector of H We refer readers to Carmon Duchi 2 Bellavia al. 2023) for details.
7

---

Published as a conference paper at ICLR 2026
Table 2: Sample-level and class-level batch unlearning on CNN X FMNIST (averaged over 3 random runs). "->" means closer to retraining is better: "1" means higher is better; "4" means lower is better. We use boldface to denote best results and underline to denote second- best results.
Medhed
Surpte-ked Lakoraing
Chan-Led Unlourinn
1-M)
ARc.
[-m)
Retraning
85.45
$7.43 00.5
$438 1644
1 DUM
11000 & DUN)
4 0.11
91.31 41177
3.00441.00
31.38.
50.45 40.17
11:00
87 11:75
1142
01800 RUM)
91.70
0:25
84:50-1155
18-12 03:30-10555
0:15-1651
5101 5151-118:
51:00-150
Raal Lab.
50.31 10.0
18 18,30-00.25
23:36:1643
1134. 5011
0 8/M0
54.73
8
0.12
00.57 x 14:31
03:00 12:47
1.00000.0
23.19.20..9
GA
47.4
03.75-0044
03.502.022 13,23
0.43 601
01508 5000
$1.42 1.0
1.6-20.44
22.00:133.0
6334 12.22
CA5 20.99
02.52
1113
01883 8/99
6166-0077
13:22-13:1
0101 11:00
(110--13
NBO
57 61
63:25-166 05
1.76
0100-100 2100
13.08 0.53
8000 0.00
75 11:5--554
16:26-171
672 -0:0
5mi--13
1100-146
1.0
14:4
23.502111
12.0-200.3
$2.59:0222
1001
PNN Noutee
3.00
1.0:
159
Dacand Nevice
1.48 1,07
6.794007
1.07 00.00
0.01 600
0C23
40.53 0,66
1.52
10,07 2//72113 1:04
9.23
0/0 p4.21
0/04-1082
0..5
CHRMMU
40.20
10:39-1017
05.30 15:00
I
1100
1.77
18
10:15
0M5
S1BC-1130
13:31+1:32
OURUNUS
8830 $8.8
0.57
85.8 2044
0/3
0
50.99
(16:0.11
$1,1
0.000..0
1100
2..6.20..
6 EXPERIMENTS
6.1 EXPERIMENTAL SETTINGS
Datasets and Models. Our experiments use four datasets: (1) FaShionMNIST (FMNIST) Xiaa
ct al. 2017): contains 60,000 grayscale images of 10 fashion items, (2) CIFAR-10 Krizhevsky et al.
2009 y contains 50,000 colour images of real-life objects, (3) AG-News (Zhang et al.
2015): contains
120,000 news titles and descriptions in 4 topics, (4) TOFU (Maini et al. 2024): y
contains 4000
question-answer pairs fictitiously generated by GPT-4 We train CNN for FMNIS, ResNet-18 HHe
ct al. 2016) for CIFAR-I0, and fine-tune Llama-2-7B (Touvron et al. 2023) with LoRA adapters HHu
al.
2022 for AG-News and TOFU. Training hyperparameters of our models are detailed in App. F.T
Unlearning Baselines. We compare against following unlearning algorithms: Retraining
Random Labels (Rand. Lbls.), Gradient Ascent (GA), Gradient Descent (GD), GDiff (Maini et al.
2024), Direct Preference Optimization with "I don't know" as positive response (IDK) (Maini et al.
2024) as baselines and SCRUB (Kurmanji et al. (023), DELETE (Zhou et al. 2025), and Negative Preference Optimization (NPO) (Zhang et al. (224b) as SOTA empirical unlearning algorithms. We
also include Newton unlearning algorithm with two Hessian degeneracy baselines (Sec. 4 Hessian
pseudo-inverse (PINV-Newton) and damping Hessian (Damped Newton) with small damping factor
y == 10-3 Hyperparameters for the unlearning baselines are detailed in App. F22
Unlearning Hyperparameters. We 400 for TOFU. Although the exact I. is often hard to find [ we show that our algorithms are robust across different empirical choices of L and deseribe a procedure to choose a valid L in App. [.] For CURENU, we only use 1 iteration on FMNIST. For CURENUS, we set σ = 0.1 and set 17 to be the same learning rate as during training. We use n1 = 10, n2 5 for AG-News and TOFU, and M1 = 128, n22 = 64 for the rest. In terms of stochastic iterations, we use T = 20, T' = 5 for FMNIST, and use T = 10, T = 5 for the rest. We study the effect of varying a and the number of stochastic iterations of CURENUS in App- J.2 and App. J.3 respectively.
L for FMNIST. 50 for CIFAR-10 10, 80 for AG-News, and
Evaluation Metrics. Following previous works (Kurmanji et al. 2023 Maini et al. 2024 Zhao eet al., (224). we compare accuracy/ROUGE De. D,., and Drest of the unlearned models and the retrained models (smaller gap is better) and report Tug-of-War (ToW) score that aggregates these gaps (higher is better). For classification tasks, we compute the Jensen-Shannon divergence (JS Div.) between the predicted probability distribution of the unlearned models and the retrained models (smaller is better) (Chundawat et al.] (023). For TOFU, we compute Truth Ratio of answering with incorrect answers versus correct answers when prompted with the question in D. (higher is better) Maini et al. 2024 Additionally, we report AUC of the Membership Inference Attack (MIA) using ML-Doctor (Liu et al.) (022) for classification tasks and the Min- K++ attack ÇZhang ct al. 2025) for text generation tasks like TOFU (a smaller gap to retraining is better). We empirically observe that the MIA is largely ineffective on our models due to regularization effects (Kaya et al.) 2020) and consider overfitted models in App. G Lastly, unlearning efficiency is evaluated based on the average unlearning time (in seconds) and peak memory usage with respect to retraining.
For I. Ilama-2 experiments, we use retraining to refer to fine-tuning the pretrained 1lama-2 model on the retain set D, from scratch.
It is empirically infeasible to enumerate over parameteric space to obtain exact L..
8

---

Published as a conference paper at ICLR 2026
Retraining
Original
Rand, Libls. SCRUB
DELETE
GD
GA
GDIfT
IDK
NPO
CUR (ours)
Liama-2 TOFU
ResNet-18 CIFAR-10
D. ROUGE
D, ROUGE
D. Acc. (%)
D, Acc. (%6
0.6
90
90
0.6
0.4
0.2-
0.4
45
45
4 # Unleamiin Rounds
2
2
2
2
Unleaming Rounds
Unlearming Rounds
Unlearning Rounds
Figure 2: Sample-level sequential unlearning on Llama-2 x TOFU and class-level sequential unlearning on ResNet- x CIFAR-IO with unlearning rounds (averaged over 5 random runs). Full results with Drest are shown in App. H.T
Table 3: Unlearning performance at the last round of sample-level sequential unlearning on Llama-2 TOFU and class-level sequential unlearning on 18 CIFAR-10 (over 5 random runs).
Mobil
(1)
Basakig
0:34
0353
0013
1.00
01355.1363
1.00 C.001
3.330
5058 0.54
Osged
0.22 4..00
0 0.65
0,716 0105
0,36 LUU0
10000 2.5
58 2.55
00:37-7775 7725
15,50 3-526
0,033 6127
4 2.034
-048 4.342
IURE LURE
19:8528.10
65.00 56 251
E175:0111
331323813
10.05 21/0
GA
1109 218311
(URE 2000
(LST3 SM1IC
100 3181
35.00 17/00
5.24 27772
24454:2105 199
24.00 5:66
0.1520000 0149
0222 2.86
31.5 1..
10S.
0U98 118011
B434 6365
(U813 11045
11348 10013
0MM0--35 3.82
10.99 111
SCRUB
0.509 04053
0045
0.40
0,312 40011
09000 4.754.335
0..00
0 1006
15.68111.6
654426601
2.11
0.85 +8.05
0481-6065
0.76 m013
0.900 +006
0.80
04.50
3.30 3166
72.96+2.00 2.80
8.11-1.9
51.35 1.:35
6.2 BATCH UNLEARNING
To benchmark computationally expensive unlearning algorithms (i.e., PINV-Newton, Damped Newton, and CuRENU) and affirm our analysis in Sec. 4 we perform batch unlearning (Sec.3.1) on CNN x FMNIST. Following Kurmanji et al.] (2023). De is selected according to two scenarios: (1) sample-level unlearning, where random subset of 80% samples in D is removed [2 and 2) class-level unlearning, where all samples of random class is removed.
Tab. 2 shows unlearning performance in the batch unlearning settings. Consistent with our conjecture in Sec. PINV-Newton and Damped Newton exhibit poor unlearning performance due to excessively large-norm updates. In class-level batch unlearning, the Newton update norms are 3708.78 + 3364.67 for PINV-Newton and 838.65 + 742.96 for Damped Newton, both substantially larger than those of CURENU (0.36 + 0.07) and CURENUS (0.38 = 0.05). Our algorithms also maintain D. Acc. much closer to retraining than the first-order counterpart (GD), especially in class-level unlearning, which reiterates our benefits of stronger convergence guarantees. More importantly, both algorithms achieve high ToW, with CURENUS consistently attaining the best ToW in both settings. This indicates that the outputs of the unlearned models closely approximate those of retraining, Overall, our unlcarning performance, especially with CuRENUS, is comparable to SOTA empirical methods (SCRUB, DELETE) and even surpasses them on some metrics, demonstrating the potential of second-order unlearning algorithms in the realistic batch unlearning settings.
6.3 SEQUENTIAL UNLEARNING
We perform sequential unlearning (Sec. 3.1 with 5 unlearning rounds on Llama-2 x TOFU and ResNet-188 x CIFAR-I0 10. In each round, De is chosen from 20% of the forget- 10% split for TOFU and from 20% of randomly selected class for CIFAR-10. We provide additional results Llama-2 x AG-News, along with experiments on more unlearning rounds and different choices of De. in App. H We note that many computationally expensive algorithms like PINV-Newton, Damped Newton, and CURENU from our previous experiments are not applicable here due to large model sizes.
Table and Fig. show unlearning performance in the sequential unlearning settings. Firstly, note that many baselines struggle with either under-forgetiing with trivial unlearning performance on D. (GD), or over-forge with degraded performance on D, and Drest (Rand. Lbls GA, and even DELETE). In contrast, CURENUS is able to achieve close performance to retraining over multiple
We remove large subset to induce noticeable outputs/performmace changes for clearer comparison.

---

Published as a conference paper at ICLR 2026
Table 4: Unlearning efficiency measured by running time (in seconds) and peak memory usage (in MB) of the best performing unlearning algorithms in Secs. 6.2 and 6.3 (averaged over 3 random runs).
CNN x FMNIST
ResNet-18 CIEAR-19
Liama-2 (+LoRA) MG-News
Llama-32 (+LoRA) x TOFU
Trainable/ Tota Parms
20,728/ 20.72
11,773.002 9322 11.179022
1,66.9601 /6,08842244000
2,09,,550 6,200,11,2588
Matric
Time (3)
Pork Men. (1)
Usl. (1)
Pork Mom. (3)
Und. Time (1)
Perk Men. (1)
Ual. Time (1)
Pork Mers. (J)
Retmaning
12451
4792.44 a
90071 4
0,85
SCRUB
679616 160.11
22112
0.39
C. CURANU
6855.31 127.31
OcRoNUS
35.54 6,73
1588
41.79
7404 85.26 18.23
791.0
240 61,00
130826
unlearning rounds. Its unlearning performance is comparable to SCRUB CIFAR-10 and even surpasses it on TOFU across most metrics. These results suggest that CURENUS a practical and scalable unlearning algorithm that can effectively unlearn NNs/LLMs with minimal accumulated errors. The high MIA on TOFU, however, is likely due to the inherent distributional difference between the forget and test sets, making it easier to separate them.
We note that intention in Secs. 6.2 and is not to claim new SOTA performance across all metrics, but rather to demonstrate that our unlearning algorithms can perform competitively with strong existing baselines (SCRUB, DELETE). The positive results show that our unlearning algorithms, especially CuRENUS, can unlock the potential of Newton unlearning for NNs/LLMs that previously struggled with degenerate Hessians. As no methods are strictly better across all metrics in these tables, CURENUS is a viable unlearning algorithm worth considering in various scenarios.
Remark 6.1.
6.4 UNLEARNING EFFICIENCY
Tab. 4 shows the unlearning time (averaged per batch) and the peak memory usage of the best performing methods (CuRENU, CURENUS, DELETE, and SCRUB) in Secs. 66.2 and 6.3 We provide measures for the remaining unlearning algorithms in App. π As expected, CURENU incurs significant time and storage to store and invert the Hessians. CURENUS is an efficient alternative of CuRENU, which only requires less than 2x memory compared to retraining, consistent with our analysis in Sec. 5.2 Moreover, CURENUS is significantly faster than SCRUB on large models like Llama-2. These results show that CURENUS a scalable second-order unlearning algorithm, setting it apart from the vanilla Newton unlearning. We note that SCRUB takes longer than retraining AG-News because retraining is only performed for one epoch. On the other hand, Llama-2 experiments require more memory than ResNet- 18 despite having fewer trainable parameters, pretrained model and the LoRA adapters into the memory.
and
must load both the
7 CONCLUSION
While Newton unlearning has proven successful for linear models with positive, semi- Hessians, its application to trained NNs/L.LMs is fundamentally hindered by degenerate Hessians. We bridged this gap with CURENU and its stochastic variant CURENUS, which are designed to tackle Hessian degeneracy in principle while the latter scales efficiently to large-scale NNs and LLMs. Our results show that our approaches, especially CURENUS, deliver strong performance in both batch and sequential unlearning, generalizing well beyond the theoretically supported regime. We anticipate that future advances in HVP computation will further enhance the efficiency of CURENUS.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This research supported by the National Research Foundation, Singapore under its Al Singapore Programme (AISG Award No: AISG3-RP-2022-029, This research supported by the National Research Foundation Singapore and the Singapore Ministry of Digital Development and Innovation, National AI Group under the Al Visiting Professorship Programme (award number AVV-2204-0001..
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv:2303.08874. 2023.
Stefania Bellavia, Davide Palitta, Margherita Porcelli, and Valeria Simoncini. Regularized methods via cubic subspace minimization for nonconvex optimization. vrXiv:2306.142000 2023.
Léon Bottou, Frank E Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning. SIAM review, pp. 223-311, 2018.
Lucas Bourtoule, Varun Chandrasekaran, Christopher A. Choquette-Choo, Hengrui Jia. Adelin Travers, Baiwu Zhang. David Lie, and Nicolas Papernot. Machine unlearning. In Proc. IEEE S&P. pp. 141-159, 2021.
Jonathan Brophy and Daniel Lowd. Machine unlearning for random forests. In Proc. ICML, pp. 1092-1104, 2021.
Yinzhi Cao and Junfeng Yang. Towards making systems forget with machine unlearning. Proc. IEEE S&P, pp. 463-480, 2015.
Nicolas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramer, Borja Balle, Daphne Ippolito. and Eric Wallace. Extracting training data from diffusion models. In Proc. USENIX Security Symposium, pp. 5253 5270, 2023.
Yair Carmon and John Duchi. Gradient descent finds the cubic-regularized nonconvex newton step. 2019.
SIAM Journal on Optimization, pp.
Gert Cauwenberghs and Tomaso Poggio. Incremental and decremental support vector machine learning. Proc. NeurIPS, pp. 409-415, 2000.
CCPA. California consumer privacy act of 2018. ht:ps://leeginfo.eeg. www.slaturee..
gov//aacs/////////aa Tte./iiiieeettiiieeee xhtml ?bil 1111____=__011 -el.-000100800RR100 2018. California
Civil Code Title 1.81.5.
Eli Chien, Haoyu Wang, Ziang Chen, and Pan Li. Certified machine unlearning via noisy stochastic gradient descent. Proc. NeurIPS. 37:38852-3888 2024.
Vikram S. Chundawat, Ayush K. Tarun, Murari Mandal, and Mohan Kankanhalli, Can bad teaching induce forgetting? Unlearning in deep networks using an incompetent teacher. In Proc. AAAI, pp. 7210-7217, 2023.
Andrew Conn, Nicholas IM Gould, and Philippe Toint. Trust region methods. SIAM, 2000.
Yann N Dauphin, Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli, and Yoshua Bengio. Identifying and attacking the saddle point problem in nigh-dimensional non-convex optimization. In Proc. NeurIPS, pp. 2933-2941, 2014.
John C Duchi. Introductory lectures on stochastic optimization. The mathematics of data, pp. 99-186. 2018.
EDPB.
Facial recognition at airports: Individuals should have maximum control over
biometric data. https: wttp:/ddbb..urop..eeeeeeee http:///eee/22222a//aaaaa
over-biometra en 2024. European Data Protection Board, Accessed: 2025-05-05.
11

---

Published as a conference paper at ICLR 2026
Ronen Eldan and Mark Russinovich. Who's harry potter? Approximate unlearning in LLMs.
arXi::2310.02238, 2023.
Guillaume Garrigos and Robert M Gower. Handbook of convergence theorems for (stochastic) gradient methods. arXiv:2301.11235 2023.
GDPR. General data CpPP // wtt.:l/w..eerrope.eee///////ee/e
article 17: Right to erasure (rright be forgotten') ). https
67 679/oj 2016. Regulation (EU) 2016/679.
Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao. An investigation into neural net optimization via Hessian eigenvalue density. In Proc. ICML, pp. 2232-2222222222 2241, 2019.
Antonio Ginart, Melody Guan, Gregory Valiant, and James Y. Zou. Making Al forget you: Data deletion in machine learning. In Proc. NeurlPS, pp. 3518-3531, 2019.
Shashwat Goel, Ameya Prabhu, Amartya Sanyal, Ser-Nam Lim, Philip Torr, and Ponnurangam Kumaraguru. Towards adversarial evaluations for inexact machine unlearning. arXiv:2201.0668 1.66400 2022.
Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Eternal sunshine of the spotless net:
Selective forgetting in deep networks. In Proc. CVPR, pp. 9304-9312, 2020.
Laura Graves, Vineel Nagisetty, and Vijay Ganesh. Amnesiac machine learning. In Proc. AAAI, pp. 11516-11524, 2021.
Chuan Guo, Tom Goldstein, Awni Hannun, and Laurens van der Maaten. Certified data removal from machine learning models. In Proc. ICML, pp. 3832-3842, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proc. CVPR, pp. 770-778, 2016.
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang. and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In Proc. ICLR, 2022.
Jinghan Jia, Jiancheng Liu, Parikshit Ram, Yuguang Yao, Gaowen Liu, Yang Liu, Pranay Sharma, and Sijia Liu. Model sparsity can simplify machine unlearning. In Proc. NeurlPS, pp. 51584-51605, 2023.
Jinghan Jia, Yihua Zhang, Yimeng Zhang, Jiancheng Liu, Bharat Runwal, James Diffenderfer, Bhavya Kailkhura, and Sijia Liu. SOUL: Unlocking the power of second-order optimization for LLM unlearning. In Proc. EMNLP, pp. 4276-4292, 2024.
lain M Johnstone. On the distribution of the largest eigenvalue in principal components analysis. The Annals of statistics, pp. 295-327, 2001.
Rohan Kashyap. survey of deep learning optimizers- first and second order methods. arXiv:2211.1559, 2022.
Yigitcan Kaya, Sanghyun Hong, and Tudor Dumitras. the effectiveness of regularization against membership inference attacks. wwXiv:2006.05336, 2020.
Ahmed Khaled and Peter Richtárik. Better theory for sgd in the nonconvex world. arXiv:2002.03329, 2020.
Diederik Kinga, Jimmy Ba Adam, et al. A method for stochastic optimization. In Proc. ICLR, 2015.
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In Proc. ICML, pp. 1885-1894, 2017.
Jonas Moritz Kohler and Aurelien Lucchi. Sub-sampled cubic regularization for non-convex optimization. In Proc. ICML,, pp. 1895-1904, 2017.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Toronto, ON, Canada, 2009.
12

---

Published as a conference paper at ICLR 2026
Meghdad Kurmanji, Peter Triantafillou, Jamie Hayes, and Eleni Triantafilluu. Towards unbounded machine unlearning. Proc. NeurIPS, pp. 1957-1987, 2023.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, pp. 436 444, 2015.
Yugeng Liu, Rui Wen, Xinlei He, Ahmed Salem, Zhikun Zhang, Michael Backes, Emiliano De Cristofaro, Mario Fritz, and Yang Zhang. {ML-Doctor): Holistic risk assessment of inference attacks against machine learning models, In Proc. USENIX Security Symposium, pp. 4525-4542, 2022.
Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary Chase Lipton, and Zico Kolter. TOFU: task of fictitious unlearning for LLMss. In ICLR 2024 Workshop on Navigating and Addressing Data Problems for Foundation Models, 2024.
Milad Nasr, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A. Feder Cooper, Daphne Ippolito, Christopher A. Sooquuette-hho., Eric Wallace. Florian Tramèr, and Katherine Lee. Scalable extraction of training data from (production) language models. arXiv:2311.17035 2023.
Seth Neel, Aaron Roth, and Saeed Sharifi-Malvaeerd Descent-to-delete: Gradient-based methods for machine unlearning. In Proc. ALT, pp. 931-962, 2021.
Yurii Nesterov. Introductory Lectures on Convex Optimization: A Basic Course. Springer New York, NY, 2013.
Yurii Nesterov and Boris T. Polyak. Cubic regularization of Newton method and its global performance. Mathematical Programming, pp. 177-205, 2006.
Thanh Tam Nguyen, Thanh Trung Huynh, Phi Le Nguyen, Alan Wee-Chung Liew, Hongzhi Yin, and Quoc Viet Hung Nguyen. A survey of machine unlearning. arXiv:2209.0222 2022.
Jorge Nocedal and Stephen J Wright. Numerical optimization. Springer. 2006.
Vardan Papyan. The full spectrum of deepnet Hessians at scale: Dynamics with SGD training and
sample size. 18Xiv:18/1.07066 2018.
Barak A Pearlmutter. Fast exact multiplication by the Hessian. Neural Computation, pp. 147-160, 1994.
Xinbao Qiao, Meng Zhang, Ming Tang, and Ermin Wei. Hessian- online certified unlearning. In Proc. ICLR, 2025.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. Proc. ICML, pp. 28492-28518, 2023.
Levent Sagun, Leon Bottou, and Yann LeCun. Eigenvalues of the Hessian in deep learning: Singularity and beyond. arXiv: 1.07476, 2016.
Levent Sagun, Utku Evei, V. Ugur Güney, Yann Dauphin, and Léon Bottou. Empirical analysis the Hessian of werrpaaametriied neural networks. arXiv: aeee:/006004444 2017.
Tom Schaul, Sixin Zhang, and Yann LeCun. No more pesky learning rates. In Proc. ICML., pp- 243-351,2013.
Sebastian Schelter. Amnesia - Towards machine learning models that can forget user data very fast. In The 1st International Workshop on Applied Al for Database Systems and Applications, 2019.
John Schulman and Thinking Machines Lab. Lora without regret. Thinking Machines Lab:
Connectionism, 2025.
Ayush Sekhari. Jayadev Acharya, Gautam Kamath, and Ananda Theertha Suresh. Remember what you want to forget: Algorithms for machine unlearning. In Proc. NeurIPS, pp. 18075-18086 2021.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In Proc. IEEE S& P, pp. 3-18, 2017.
13

---

Published as a conference paper at ICLR 2026
Sidak Pal Singh, Gregor Bachmann, and Thomas Hofmann. Analytic insights into structure and rank of neural network Hessian maps. In Proc. NewrIPS, pp. 239227, 2021.
Sidak Pal Singh, Thomas Hofmann, and Bernhard Schölkopf. The Hessian perspective into the nature of convolutional neural networks. In Proc. ICML, pp. 11930-31963 2023.
Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance initialization and momentum in deep learning. In Proc. ICML, pp. 1 1139-1147, 2013.
Anvith Thudi, Gabriel Deza, Varun Chandrasekaran, and Nicolas Papernot. Unrolling SGD: Understanding factors influencing machine unlearning. In Proc. IEEE EuroS&P, pp. 303-319, 2022.
Zhiyi Tian, Lei Cui, Jie Liang, and Shui Yu. A comprehensive survey on poisoning attacks and countermeasures in machine learning. ACM Computing Surveys, 55(8):1-35, 2022.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. arXiv:2307.09288, 2023.
Nilesh Tripuraneni, Mitchell Stern, Chi Jin, Jeffrey Regier, and Michael Jordan. Stochastic cubic regularization for fast nonconvex optimization. In Proc. NeurIPS, pp. 2904-2913, 2018.
Joel A Tropp et al. An introduction to matrix concentration incqualities. Foundations and Trends in Machine Learning, pp. 1-230, 2015.
Alexander Warnecke, Lukas Pirch, Christian Wressnegger, and Konrad Rieck. Machine unlearning of features and labels. arXiv:2108. 11577, 2021.
Han Xiao, Kashif Rasul, and Roland Vollgraf. machine learning algorithms. arXiv: /708 2017.
novel image dataset for benchmarking
Zuobin Xiong, Wei Li, Yingshu Li, and Zhipeng Cai. Exact-Fun: An exact and efficient federated unlearning approach. In IEEE ICDM, pp. 1439-1444, 2023.
Peng Xu, Fred Roosta, and Michael W Mahoney. Newton-type methods for non-convex optimization under inexact Hessian information. Mathematical Programming, pp. 35-70, 2020.
Haonan Yan, Xiaoguang Li, Ziyao Guo, Hui Li, Fenghua Li, and Xiaodong Lin. ARCANE: An efficient architecture for exact machine unlearning. Proc. IJCAI, pp. 4006-4013, 2022.
Zhewei Yao, Amir Gholami, Kurt Keutzer, and Michael W Mahoney, Pyhessian: Neural networks through the lens of the Hessian. In IEEE BigData, pp. 581-590, 2020.
Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy risk in machine learning: Analyzing the connection to overfitting. In Proc. IEEE CSF, pp. 268 0688288,110 2018
Charles Yu, Sullam Jeoung, Anish Kasi, Pengfei Yu, and Heng Ji. Unlearning bias in language models by partitioning gradients. In Findings of the Association for Computational Linguistics: ACL 2023, pp. 6032-6048. 2023.
Binchi Zhang, Yushun Dong, Tianhao Wang. and Jundong Li. Towards certified unlcarning for deep neural networks. In Proc. ICMI., pp. 58800-58818 2024a.
14

---

Published as a conference paper at ICLR 2026
Jingyang Zhang, Jingwei Sun, Eric Yeats, Yang Ouyang, Martin Kuo, Jianyi Zhang, Hao Frank Yang, and Hai Li. Min-k%+++ Improved baseline for pre-training data detection from large language models. In Proc. ICLR, 2025.
Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. Negative preference optimization: From catastrophic collapse to effective unlearning. In Proc. COLM, 2024b.
Shaofeng Zhang, Meng Liu, and Junchi Yan. The diversified ensemble neural network. In Proc. NeurlPS, pp. 16001-16011, 1, 2020.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In Proc. NeurlPS, pp. 649-657, 2015.
Kairan Zhao, Meghdad Kurmanji, George-Octavian Bärbulescu, Eleni Triantafillou, and Peter Triantafillou. What makes unlearning hard and what to do about it. In Proc. NeurIPS, pp. 12293-12333, 2024.
Yu Zhou, Dian Zheng, Qijie Mo, Renjie Lu, Kun-Yu Lin, and Wei-Shi Zheng. Decoupled distillation to erase: A general unlearning method for any class-centric tasks. In Proc. CVPR, pp. 20350-20359, 2025.
15

---

Published as a conference paper at ICLR 2026
A COMPARISON OF UNLEARNING ALGORIHTMM AGAINST DESIDERATA
Method
Effectiveness D1)
Efficiency
D2
Usability D3
Practicality
D4
Newlon
X
K
Damped Newion
X
X
X
Hessian- Unlearning (Qiao
CuRUNN
X
X
CUReNUS
Table 5: Comparison of different unlearning algorithms against desiderata (Sec. 3.1
Tab. D compares different unlearning algorithms with respect our desiderata defined Sec. 3.1 We note that [2] is assessed based on both time and memory efficiency compared to retraining. Additionally, we also compare them regarding their applicability to challenging settings such as sequential unlearning.
Gradient Descent (GD). GD an efficient unlearning algorithm as it relies only on gradient information, satisfying D2 However, Sec. [ shows that GD often fails to sufficiently remove the lineage De in both batch and sequential unlearning settings, hence violating D1 and D4 Additionally, App. shows that its unlearning performance is susceptible the selected learning rate, hence violating D3
Newton Unlearning. We note that vanilla Newton unlearning often fails to apply to neural networks due to problematic Hessians (Sec.F) 4. hence violating D1 and D4 Without appropriate damping. can produce large-norm updates and poor unlearning performance, indicating a failure to meetDD: Moreover, it requires storing and computing full Hessians, which is computationally expensive, and therefore does not satisfy D2 and D4
PINY-Newton and Damped Newton. Both PINV-Newton and Damped Newton demonstrate poor
unlearning performance in our experiments (Sec. (), hence violating D1 and D4 PINV-Newton does not involve explicit hyperparameters yet its unlearning performance is limited, thus violating
Damped Newton is sensitive to the choice of damping factor (Sec. 4 hence violating D3 Similar to the vanilla Newton method, both approaches also fail to meet and
D3
and D4 due to the high
computational and storage costs of explicit Hessian computation and storage.
CURENU and CURENUS. In Sec. 6 we show that CURENU and CURENUS achieve unlearning
performance close retraining, satisfying DT Both algorithms remain robust across different choices of L (App.I.] [.I]. CURENU identifies the optimal damping factor via an optimization problem (Sec.
requiring minimal tuning D33, but it does not meet and cannot be applied to sequential unlearning due to high computational and storage cost (violating D4 As its stochastic variant, CURENUS retains robustness to hyperparameters such as the number of stochastic iterations (App. [.3) and (App. J.2.
which make it satisfy D3 Moreover, CuRENUS is computationally and memory efficient (D2 while being effective for sequential unlearning (Sec. 6.3 hence satisfying D4
Hessian-free Unlearning (Qiao et al., 2025). While the Hessian-free unlearning algorithm in Qiao
et al. (2025) demonstrates effective unlearning for linear regressions and CNNs (D1) and robustness to varying step sizes and epochs (D3). it requires storing substantial number of HVPs (Hessian-vector products) that scale with the dataset size, and therefore fails to satisfy D2 This high storage cost limits its overall application, including sequential unlearning (D4).
B JUSTIFICATION OF ASSUMPTIONS
For completeness, restate assumptions in Sec. 3.2
Assumption B.1. L is twice continuously differentiable with respect to w.
Assumption B.2 (p-Lipschitz gradient). some > llgw gw' || S / o||ww w'], Vw, w' Rd.
Assumption B.3 (L-Lipschitz Hessian). For some L > 0, Hw Hw || < L||w-w'll Vw. w' Rd.
Assumptions et al. 2021 Guo et al.
B.
B.2
and B3 are widely used in existing works (Zhang et al. 2024a Sekhari
2020 More importantly, they do not require the loss function to be convex,
16

---

Published as a conference paper at ICLR 2026
allowing to extend our analysis non-convex settings such neural networks. A caveat of these assumptions is that they may exclude non-differentiable activations, such as ReLU activations that commonly used in neural networks. To address this, empirically evaluate CuRENU and CuRENUS on Rel.U-activated networks like CNN x FMNIST and ResNet-18 x CIFARIO (Sec. ( Our experiments show that both unlearning algorithms are able to achieve strong unlearning performance, suggesting that they generalize well beyond the theoretically limited regime.
C DERIVATIONS
Under Assumption B.1 H:: E Rddd is symmetric and is orthogonally diagonalizable by the spectral theorem, i.e., HD: QAQ-1 with orthornormal basis Q [urg...; Ud] and diagonal matrix
Biag((A1...... Moreover, since Q orthonormal, Q" Q and u² u, = ||u.||2 1.
C.I DERIVATION REMARK 4.2
Remark C.1 (Restated of Remark 4.2
The Newton update norm using the pseudo-inverse Hessian
|| [A411| 1|? Eis.+ X? (u? gw.
D-
We denote by A & diag( ,.... ) pseudo-inverse of A, where X - Of X3 0. Expanding (HW:)!, we get:
(H wi
Swi = (QAQ 1) 21 D. = (QA/Q-") = (QA )::
gg:
d
Σ BA
1 T,
(u, Bw:
Since the above expression a sum of orthonormal vectors u, where each u, is scaled by X, (u:"ok:). 92:). the update norm using the mneudo-inverse Hessian is given by:
((
Σ
[ D.) 1u,l11
Σ
X? u
gw: P.))2
#A:30
C.2 DERIVATION OF REMARK 4.3
Remark C.2 (Restated Remark 4.3
The Newton update norm using the damped Hessian is
|| = En (74A.y7 (u. D.)22
We denote by A., A diag(? A Aa) a diagonal matrix of eigenvalues for the 7-damped Hessian HD + ZI, where 7 max(0, Aa). Expanding H + II, get:
H 71) -1 D.
(QA,Q-1)
gw. D.
(QA,'Q Q "0 (
gw:
1
u: p.))
Therefore, as explained in App.
C.I
the update norm using the y-damped Hessian is:
d
d 1
(HD. w. + +1I)
gw: 2 Σ
(u gw: 1l
Σ
(u (uf' gug
(y + A1)
(y + x,)
17

---

Published as a conference paper at ICLR 2026
D DETAILS OF CURENU
Upper bound. Here, we prove that the cubic regularized approximation [(we+1; D,) is global upper bound of [(W++1; D,). A similar proof can be found in Lemma of Nesterov & Polyak 2006).
Proof. For all w, w' E TRd
gw gw /
Hw ww w)dr.
Then,
ggw gw/ Hww(Ww w)ll - 11 / (Hw+r(w'ww) Hww) (w' - w)drll
(Hw+r(w'ww) Hw)(w - -w)ldr
Iw' w|l / |I( (w' -Hw) -ww)lddr
(sub-mulipplicative)
1|w' w| S =T/| -wld7
L
2 'Iw'' w/1 (by Assumption B.3
Hence,
1 (H (w' w), w' w)
[(w') - L(w) (gww, w' -w)-
(Ww+T(W-w) - Bw THw(w's - w), w w)dr
< (Bw+r(w'-w) Bw -Hw(w' ww) w' w) dr
1|w' -wll / [18w+7(w'ww) Bw -H (w' w)lldr
(by Cauchy-Schwarzinee inequality)
www w|l 1/" L
2 1 w/idr -wl
-1 wwl
Let the evaluated data be the retain set D,,, w' Wx+1 and w.. By solving the absolute value inequality above, for all Wt, We+1 E R.d we have:
L ||4 L(We+1; Dr),
L(we; D.) Dr A441)
( A A
(8)
where A A |w+11 well The left-hand side is the cubic regularized approximation [(w D.).
D.I CONVERGENCE GUARANTEE
Definition D.1 (Restated of Definition 5.2 An E-SOSP of the function L (with L-Lipschitz
Hessian) satisfies [|gw || < a and the minimum eigenvalue of the Hessian Xmin (Hw) 2 -v Le.
A O-SOSP is a local minimum as VL(w)] is 0 and the Hessian is positive semi-definite. When is small, the gradient norm close 0 and the Hessian's minimum eigenvalue near non-negative, meaning that strongly negative curvature directions are absent or, if present, very mild. While finding an exact local minimum computationally hard, an s-SOSP with sufficiently small æ can approximate a local minimum and avoid most saddle points and sharp local maxima.
Proposition D.2 (Adapted from (Nesterov & Polyak 2006 Theorem 1)), For non-convex functions satisfying Assumption B.3 CuReNU converges SOSP in ( iterations.
18

---

Published as a conference paper at ICLR 2026
Proof. Let Wi denote the parameters in iteration t. We define w241 _ migmin [(we+1: Dr).
where [(We+1; D,) is given in Eq. and 1 Wi+1 We.
For Wi+1 to be global minimizer of f(w,+1; D.), it must satisfy:
Sw,
H. A1+1 2
L
/11 4411
(9)
L
Hº
14 1=0
(10)
2
Multiplying A;+1 once the Eq.
9 and twice in the Inequality 10 we get:
L
(gw: 11+1) + (H Ai+1: Ai+1)
(11)
2
+ L
H A 1+1 (+)
1
11$ 20
(12)
2
Together, Eq. 11 and Ineq. 12 imply that:
(g: 11+1) 0.
(13)
Step 1: We now aim to bound the decrease in L(:; D,) at each iteration. Since Eq.
state that
L(We+1; D,) is the upper bound of L(W++1; D,), we have:
C(w;; D,) [(w D,) 2 L(w; D,) - ((wizi)p))
Moreover, using Definition 3 of L(; D,), Eq. and Inequality
13
we have:
1
L
L(wi; Dr) [(wi+11 Dr) (gw: Ai+1)
H D.
w.
A 1 A
141)
||
2
6
2 gw.
Dr A
+
|
b+
||
2
|
12
12
Hence,
L
L(w;; D.) [(wi41i D.)
Ai+1 11.
12
Step 2: We set up the following auxiliary results.
a) From Eq. 10 have that
gg:
D. Ai
Moreover, by Inequality 2.2 in Lemma Nesterov & Polyak (206)), we get:
I
lsw:
gw; A.AAinll 2 Ai+1
By the triangle inequality, these results imply that:
lg D
HD
A |g gw:
HDr w.
A S I 12
(14)
b) view of Inequality 10, the following must hold:
I, |it1ll-
-Amin (HD:) < 2
(15)
Step We define
pr. (w) = max { {V I L(w; Dr)lls -T, J.min CominvV2 <V> Dr))
1
(16)
19

---

Published as a conference paper at ICLR 2026
Intuitively, (z((w) 2 reflects the (non)-local optimality of w, ie. w is local minimum with V<(w; D,) = and [2 D,) ž=2 iff [L(w) =
We now aim to show that || PL (wi+)) Indeed, expanding pz (w(+1) using Inequality [4 and Ineq. 13 we get:
V L
V -|| -|
||4
Amin H.
- L
1
" ||بباا |لد++1||
Therefore, ||تح1||| 2 (w(w1+1).
Step 4: Let L be a lower bound for Ll D,). Given that CURENU involves T iterations, it follows from the results of Step and Step that:
T-1
L(wo; D,) - L" 2²
Σ
(e(w;; D,) ((w1+1:D,)
t=
||
12
-
L
TL
1 (w/+1)
min pc. (w;) 151ST
12
12
t=
Hence,
12 (C(wo: Dz) (2))-())
1/3
min Mr (w,) S 1SIST
(17)
TL
By our definition in Eq. must satisfy:
16 for min misTs Mr (w,) corresponding an E-SOSP (Definition 5.2)
+ [VV[(W;; D.)I 2 V
min Mr (w;) 2 min sttT
15tST
E
ML (w;) 1ttT
min
min ISST
Amin VL(W;; D₂) 2 L
Using Inequality 17 and making T the subject of the formula, have that:
12VI(C(wo: D.) L-)
T T
gl.5
Hence, T < O(5
D.2
DUALITY BETWEEN THE PRIMAL AND DUAL PROBLEMS
Here, we restate the minimization problem of the cubic-regularized quadratic approximation of [(We+1; D,) as defined in Eq.
min [(w L(w; D.) (gW; Arti + (H A41) 11+ We+1
Let a++1 A ||W+11 well For the above primal problem, we have the dual optimization problem in O++1 as described in Eq.4
supor Elau+1), {(0r+1) - + L t(411I) -1 D. D.
gw; gw:
12
L
a
s.t. 0:41 Q = {a E IR HD: + aal 0, 0).
20

---

Published as a conference paper at ICLR 2026
The duality gap is the difference between the optimized value of the primal problem minl.4: [(wo-1; D,) and that of the dual problem supa441 E(ar+1). If the duality gap equals O, which is called strong duality, the optimized value of the dual problem equals the optimized value of the primal problem. The following proposition, adapted from Nesterov & Polyak (2006) Theorem 10), states the duality gap between the primal and dual problems.
Proposition B.2.1. For any L > O, the primal and dual problems satisfy strong duality, i.e. min [( D,) = suPoL E(0441). Moreover, let D441 - (H 2 4 2 gw: Then,
a+4++2]|A++ill for any 0t++1 E Q, the duality gap is 31, [0 ('(a)+1)2 2 0 where g denotes the first-order derivative of w.r.t. ax+1-
Note that the dual problem is a one-dimensional concave maximization problem over a++1 E Q. From Proposition B B.2.1 C( ((1++1) is minimized when a++1 E Q satisfies ('(a)+1) = 0, indicating global maximizer of the dual problem. Moreover, even if finding such maximizer is infeasible, finding a++1 E Q with a small <'(a++1) implies a small duality gap and thus the primal function value is near optimal. Additionally, the concave constrained dual problem be solved efficiently instead of the non-convex primal problem through techniques to solve the trust-region subproblem, as we will show next.
D.3 SOLVING THE DUAL PROBLEM USING TRUST-REGION METHODS
Firstly, by taking the derivative of g (defined Eq. w.r.t. 0+++1 we have:
d
L
Dr g((a+1)) = do+t1 2 wi
(HH
+ 00+11) 1) =2
Bwi 1 12 (+1
Dr Dr) 3
L (gD:)² (He:))(Hi: + aatill (H + aatiil) - ow:
0+
L - (11A+1ll a(+1)-
Setting ('(0411) to 0 and note that 04+1 2 0 we have:
||4+11|| = 02+1-
(18)
Also, since A > O we have:
0.+1 > max (0, 2 70
(19)
where Aa is the minimum eigenvalue of H Here, we use trust-region methods (Conn et al.
2000
solve Eq. 18 and Eq. 19 We consider the following trust-region subproblem:
min gw: D. A(+1) + 2
H A.1 Aeti)
s.t. || S 0441-
By Corollary 7.2.2 of Conn et al. (000)) the above problem admits a global minimizer A A
(H + 44 -1 gw: that satisfies Eq. 18 and Eq. 19 Therefore, we adopt a standard procedure to solve the trust-region subproblem.
Our procedure starts from a++1 = max(0, f Xa) + where e is small constant. Note that this choice of a yields the largest A as showed the latter is monotonically decreasing function of y 1 a1+1 (App. C.2 From this, two cases arise by comparing a4+1 and ||++i]]:
Case happens when || || 2 0++1- Empirically, we observe that this case is common for neural networks with highly non-convex losses. Then, we can use Newton's 's method find the root of ||A Q+++1 = using its derivative w.r.t. at-1- However, we will
empirically use Newton's method with better-behaved function AA.: T dx+ to avoid
the tricky case when A has small eigenvalues (Conn al. 2000, Section 7.3.3).
21

---

Published as a conference paper at ICLR 2026
a4++1 This means 41+1 is in the interior of the trust region (with radius a(+1),, and the trust-region subproblem becomes an unconstrained optimization problem. However, by Corollary 7.2.2 of oonn al. (2000), this implies that c+++ 0 and the Hessian itself positive definite, which contradicts our empirical observation (Obs. 4.1 Nonetheless, if this occurs, will accept A1+1 A+41.
Case 2 happens when ||
D.4 PSEUDOCODE
We provide the pseudocode for CURENU in Algorithm The algorithm employs Newton's method with trust region (App. (D33) to solve the dual problem. Following Conn et al. (2000), we use the Cholesky decomposition of the regularized Hessian matrix to enhance computational efficiency and numerical stability.
Algorithm 1 CURENU
Input: original model parameters w', retained set D,, objective function L, Hessian L Lipschitz constant I., number of unlearning iterations T, number of Newton's iterations Timner, tolerance
Set Wo w
2:
for t = 0.,I 1 do
3:
Get gw: V[(w;; Dr)
4:
Get HD: = V2L(W;; D,)
5:
A++1 = SRALTDDUA.PRBLEMMCEE wis
Drr L, E, T Tinner
6: Set Wi+1 W. + A++1
7:
end for Output: unlearned model parameters: Wr
8: function SOLVEDUALLPOOLEM(.. g, L. E, T inner)
See D.3
9:
Get the minimum eigenvalue xa of H
Set "o = ma Xa) + Set og
11:
20
12:
Factorize H 20 LL" Solve A in (LL - A -g
D Cholesky Decomposition
13:
14:
if || 2 0g then
15:
for / t T..T innner do
D Newton's 's Method
||||| 0t-1 < then
17:
break:
else
19:
Solve Lu A
20:
Set S' (aa-1) =
A
as11
21:
Set E" (a)-1)
A
|u|l
7--99-11
('(a
22:
Set 7: = 7t-1
8"(as-1
23:
Set
27
24:
Factorize H 71 LL
D Cholesky Decomposition
25:
Solve - =g
26:
end if
27:
end for
28:
end if
29:
return A;
end function
E DETAILS CURENUS
E.1 CONVERGENCE GUARANTER
Throughout this section, we use Ô to hide the logarithmic factors, ie. = log" n) for some constant l. To make the convergence analysis tractable, need the following assumption.
22

---

Published as a conference paper at ICLR 2026
Assumption B.2.1. The stochastic gradient and Hessian estimates of satisfy
Vw, E
low gw Ww!l I 02 and H H < M2 almost surely.
|| - oî and llew -gw S M1 almost surely;
Vw, B
H R3
Theorem B.2.1 (Adapted from (Tripuraneni et al., 2018 Corollary 1)). For cenn-coovexfuucction
satisfying Assumptions B.3 B.2 and stochastic estimates satisfying Assumption B.2.1 with probability greater than 1 - 6, m1
and 122 = ö a
CuRENUS can converge to an E-SOSP in
Le
O 3.5) stochastic gradient/HVP evaluations where is sufficiently small.
Proof. Step 1:
Under Assumption B.2.1
use the matrix Bernstein inequality Tropp et al. (2015)
to derive the following concentration bounds for gradient and HVP:
For n 2 max o1d' qf=
log
2d 3. then with probability 1- 8',
llww
Bi - -ww
D. S CIE
o2 ca egLe! )3 log 2d 8. then with probability 1 8'
For n2 2 max
vz E Rd (| HD:) z|| S c22 LE||z||
For sufficiently small e, the above inequalities hold for nn = (3² and n22 Ô
02 Ie
Step 2: We denote w A millwai1 1 D,) and Ai+1 W Wt. where
[ D,) is defined in Eq. 6. Here, we note that w is shown to be achievable by the gradient descent algorithm (perturbed by small σ) as described in Sec. 5.2 The convergence guarantee of gradient descent is given in (Carmon & Duchi 2019).
Similar to the proof in Step of App. D.I
we have:
I. D.) 12 |
[(w;; D,) Esto ( (wi+
However, unlike the results in App. D.1 it does not immediately follow that Csto D,) is the upper bound of L(W(+1; D,). Instead, this upper bound holds only up certain tolerance, as we will show next.
Step By the proof of Lemma of Tripuraneni et 2018). if w is an e-SOSP (Definition 5.2 then
E |4 |l < 2 L
(20)
Otherwise, we have that || 2 IVE.
Step 4: Using the implication of L-Lipschitz Hessian (Eq. 8). Cauchy-Schwarz inequality with gradient and HVP concentration bounds in Step 1, and the results from Step 2:
C(w D,) L(w;; D,)
L +
S (g Ait1) + (H D. A Air)
- ||
||
Est0 (" (Wi+ D,) ((w;: ; D (g gw: A
(H
P. Ai::)ii1: Ait))
2
- - S A + C2
Th2 Le Ai
We now consider the characteristics of wi+1
23

---

Published as a conference paper at ICLR 2026
Case 1: Wi+1 is E-SOSP, then using Ineq. 20 get:
3
83
C2
53
[( C(wi+1i D,) - L(wi: D,) ~ 96
2
T.
-
T,
where e 96
_ by making c1, <2 arbitrarily small (i.e. increasing mn 122).
Case 2: (7 not an e-SOSP, then I 4
T,
[(w]+1; L(w,; D,)
A
1|
44
$1
+
e2 A
A41ll
12
1
c3
c3
E3
0
96
L
2
where
making C1,02 arbitrarily small (i.e. increasing n1 and 122).
In both cases, we get the following bound:
E3
C(w;; D,) - C(w D.) 2 oV L
(21)
Step 5: Let L be the lower bound of C(w; CURENUS involves most T iterations to decrease
from L(wp; D,) to L" where the upper bound on per-iteration decrease is given Inequality 21 Following the same argument Step 4 of the proof of Prop D.2 the total number of iterations T is:
T VI(L(wo. D,) - L*)
og
Step 6: Since n1 = Ō and n2 = 0 from Step 1, each iteration (corresponding to minimizing the stochastic
approximation once) cost involve n1 gradient and n2 T((s) HVP evaluations, where T(E) the number of steps for the gradient descent method to find a sufficiently good minimizer of Esto By Lemma Tripuraneei et al. (2018), T(e) S Ö (z))z) )-). Hence, the overall computational cost is
① (VICC(wo L*)
o
P
cg
Le
L3
When is sufficiently small, the above equals to Ō
gradient/Hessian vector evaluations.
F.2 PSEUDOCODE
We provide the pscudocode for CURENUS in Algorithm Following Carmon & Duchi (201)), when the stochastic gradient is large, i.e. [: f. take a Cauchy step (steepest descent within the trust region), which closed-form and computationally efficient as shown Lines 12-13, 131 to induce
decrease in the stochastic approximation =sto and function value L.
DETAILED EXPERIMENTAL SETTINGS
F.1 TRAINING HYPERPARAMETERS
We conduct our experiments on NVIDIA H100 GPUs (80GB) and NVIDIA H200 GPUs (141GB). Our evaluation is averaged across random seeds {1,2,3), The training hyperparameters in our experiments are detailed below.
CNN X FMNIST. The dataset includes 60,000 training samples and 10,000 test samples. We adopt
small 2-layer CNN with 32 filters of size x max pooling of size 2 x 2, ReLU activations, and fully connected layer with 64 hidden units. We train the CNN with the SGD optimizer with a batch size of 64 over 30 epochs, learning rate of 0.01 with decay 0.5 every 2,000 steps.
24

---

Published as a conference paper at ICLR 2026
Algorithm 2 CURENUS
Input: original model parameters w", retained set D,, objective function L, gradient Lipschitz constant p. Hessian Lipschitz constant L, number of unlearning iterations T, number of gradient descent iterations 7 Timeer gradient perturbation parameter learning rate
Set Wo = w
2:
for = 0..I' 1 do
3:
Get B1, B2 independently sampled from D,
4:
Get gw: v [[w;; B1)
5:
Get H V LWw B2)
6:
A++1 = DrRENNEN gB:
w. :wi Ps L, T Tinner, 0, 11)
7: Set Wx+1 W. A++1
8:
end for
Output: unlearned model parameters: Wy;
9:
10: function GRADIENTDDSSCENTT.. g, L, Tnner 0,1))
11: Igl 2 e then
Cauchy Step
Set R. lgl
Hg
Hg Ligll
13: Set A Re
g
gll
14:
return A:
else
16:
Set Ao 00
Get g =g os where ( Unif(gd-1)
D Gradient Descent
18:
for f = O..Tinner - - do
19:
D
(HA, + g LIIA.|A.))
20:
end for return ATwerri
21:
22:
end if
23:
end function
ResNet-18 CIFAR-10. The dataset contains 50,000 training samples and 10,000 test samples. We train ResNet18 (He et al.] (016) using the Adam optimizer with a batch size of 100 for 10 epochs, learning rate 0.001 with decay 0.5 every 5,000 steps, and weight decay of 10-4,
Llama-2 x AG-News. The dataset contains 120,000 training samples and 7,600 test samples. We fine-tune the pretrained Llama-2-7B model from Hugging Face [sing LoRA (r 2, a = 2, drop 0.1) in bfloat16, with a batch size of over training epoch, learning rate of 10 -4.4, and warmup ratio of 0.03.
The training set contains 4,000 question-answer pairs fictitiously generated by GPT-4, while the test set contains 100 question-answer pairs about real-world authors. We finetune the pretrained L.lama-2-BB-a
Llama-2 x TOFU.
model from Hugging Face uuiing LoRA = 4, a 16, drop out 0.05) in b6loat16, with a batch size of 4 and 4 gradient accumulation steps over 5 training epochs with a learning rate of 10 We also provide results on full -lamm--2-7B without LoRA in App. H.5
We compute the empirical Hessian eigenspectrum density for Llama-2 x TOFU (Fig. D right) using the PyHessian package Yao et al. 2020) with 100 iterations of the Stochastic Lanczos Quadrature algorithm. Due to memory constraints, is infeasible to compute the Hessian eigenspectrum over the full dataset. Therefore, to mitigate variance, we repeat the process 10 times with independent mini-batches of size 8 and report the averaged results.
https://huugiinggaccc catee.ccomeettt
ama ch
25

---

Published as a conference paper at ICLR 2026
F.2 UNLEARNING BASELINES
Here, we describe the unlearning baselines and their hyperparameters used in our experiments. Unless otherwise specified, we conduct grid search for the best learning rate in {10 10-1, 100-).
trains the model from scratch on the retain set Dr. For Llama-2 experiments, we use retraining to refer to fine-tume fine-tuning the pre- ree-trained Llama-2 model on D, from scratch. The training/fine-tuma tuning hyperparameters are the same as those used for training/fine-tunn the original model on the full training set D.
Refraining:
Random Labels (Rand. Lbls.): fine-tunes the model on the randomly labeled De for epoch.
DELETE Zhou et al. 2025): uses the original model to generate new labels for De (different from the true labels), then minimizes Kullback-Leibler (KI.) divergence with respect to new labels for epoch.
Gradient Descent (GD); minimizes losses on De via gradient descent on D,. We run GD for I epoch on TOFU and 5 epochs for the rest, with the same learning rate as during training.
Gradient Ascent (GA): maximizes losses De via gradient ascent for epoch.
Gradient Difference (GDiff) (Maini et al. 2024): minimizes the weighted average of the loss on D, and the negated loss on De. We sample a subset of D, to be of the same size as De and assign equal weights to both loss terms. We run GDiff for I epoch TOFU and 5 epochs for the rest.
SCRUB (Kurmanji et al. 2023): maximizes the KL divergence to the output distribution of the original model on D. while minimizing on D, in an alternative manner. We run SCRUB for epoch on TOFU and 5 epochs for the rest.
PINV-Newter (Sec. 4 replaces exact inverse with pseudo-inverse in the vanilla Newton unlearning. We run PINV-Newton for epoch on FMNIST.
Damped Newton (Sec. adds a small diagonal matrix 7I to the degenerate Hessian in the vanilla Newton unlearning. We use ? 10-3 by default. We run Damped Newton for epoch on FMNIST.
IDK Maini et al. 2024): encourages alternative answers such as "I don't know" when prompted the LLMS with questions in D... We run IDK for epoch on TOFU.
NPO (Zhang al. 2024b): discourages the original answers/predictions in De- We run NPO for epoch. We conduct grid search for the optimal / in {0.5, 1,2].
While relevant, our experiments do not include a comparison with the Hessian work of Qf et al. (2025) due to their prohibitively high computational cost and memory requirements. Particularly, Qiao et al. (2025) inherently incur substantial computational overhead due to the precomputation step that computes the Hessian- vector product for every sample in the training set. When evaluating their method on FMNIST, the precomputation step takes around 468.3 hours ( 19.5 days) to run, which already far exceeds the time required for retraining (= 61.20 seconds) and defeats the purpose of unlearning. Additionally, their method requires ((nd) memory to store HVP for every sample in the training set, where the dataset size. The prohibitively long precomputation time and the significant memory requirement thus make Qiao et al. (2025) impractical in real-world settings.
F.3 HESSIAN-VECTOR PRODUCTS COMPUTATION
common operation involving the Hessian matrix many applications is its product with an arbitrary vector v, known Hessian-vector product (HVP), Pearlmutter (1994) introduced an efficient method for computing HVPs without materializing the full Hessian H based on the following definition:
Hv lim 1 V C(w + Ev) V. C(w) (V (())))
Here, HVP corresponds to the directional derivative of the gradient C in the direction of vector
Automatic differentiation (AD) can be used to compute the derivatives involved in HVP computation. At its core, AD builds computational graph by decomposing the function into variables and
26

---

Published as a conference paper at ICLR 2026
elementary operations, and applies the chain rule this graph compute derivatives. Depending on the order of multiplications in the chain rule, AD can be implemented in forward mode (propagating derivatives from inputs outputs) backward mode (propagating derivatives from outputs to inputs). Forward mode is often more memory-efficient than backward mode because it does not require storing all intermediate states in the computational graph during derivative computation.
By combining different AD modes, we can efficiently compute HVPs using two common approaches: forward-over-buckward and backward-over-buckwarrd The basic steps for each approach are outlined below.
Torward-over-backward
1. Compute the gradient (ww using the backward-mode AD.
2. Compute the directional derivative of ((w) in the direction of v using the Jacobian-vector product with forward-mode AD.
Backwaard-oeer-backwarr.
1. Compute the gradient Vww(w) using the backward-mode AD.
2. Compute the scalar product h(w) (VwC(w), v).
3. Compute the gradient of lu(w) w.r.t. w using another backward-mode AD,
Our experiments use PyTorch's HVP implementation, which is supported through torch. autograd. fun.rrr...uuuiiiiio www.tiinnnantt hvp (). Since forward-mode AD is not natively supported in PyTorch, HVPs are computed using the wwwwwrrd-ove-bbbwwwar approach, which has a larger memory footprint due to double backward differentiations. Moving forward, more efficient implementation of HVP computation in PyTorch, such as the forward-over-bace approach, can be studied to further improve the efficiency of CURENUS.
G ADDITIONAL RESULTS FOR MIA
As discussed in Sec. 6.1 we obtain the MIA results using ML-Doctor (Liu et al. 2022) by computing the AUC score based on the losses of the unlearned model on De and D Drest. However, we observe that the standard MIA based on losses of the unlearned models is not informative (e.g., Table 3
which we hypothesize due to the following factors:
Overfitting. As mentioned in Kurmanji et al. (2023), the original model in their work has overfitted more than a state-of-the-as model on CIFAR-10 would, and their MIA is performed on the overfitted original and unlearned models. Additionally, the analysis in Liu et al. (2022) also shows that higher overfitting level leads to better membership inference. However, the original model in our setting is more generalized (from Table 3 Diest and D, accuracy are close each other with difference of 1.85%), and MIA on our non non-overfitted models is less informative. While studies have shown that there connections between overfitting and privacy leakage (Shokri al., 2017 Yeom et al.. 2018 Liu et al., 2022), in our setting, when the original model generalizes well on Dtests the losses on D. and Drest would be less distinguishable, which leads to close-to-50% MIA result for both original and unlearned models. Therefore, we conduct a new experiment by training the original ResNet model and CNN model for 50 epochs obtain overfitted models (e.g., from Table Drest accuracy is much lower than D, accuracy with difference of 7.17%) and perform unlearning on the overfitted model. However, in practice, an overfitted model is less preferable than those with better generalization abilities. 6land show the performance and MIA results for different unlearning algorithms on the overfitted CNN and ResNet models, where the MIA result more informative. Based the results from the overfitted ResNet. our CURENUS successfully decreases MIA result by 6.06%. While there remains a gap between CURENUS and the state-of-the-art method SCRUB, our CURENUS offers advantages in theoretical support and efficiency.
Test Loss Distributions. our MIA experiments on FMNIST, CIFAR, and AG-News, De and Dteat samples drawn from the same distribution, e.g., samples of the same class. This results in similar loss distributions on D. and Diest samples on both the original and unlearned models. As shown in Fig. g both nov-overfitted and overfitted models result in similar loss distributions D. and Drest
27

---

Published as a conference paper at ICLR 2026
Table 6: Performance and MIA results in the class-level sequential unlearning setting on the overfitted ResNet- 18 x CIFAR-10 (averaged over 3 random runs). Results are reported at the last unlearning round.
Overfilted ResNei-18 CIFAR-10
Removal
Dg Acc. 1)
D, Asc. (1)
Droc Acc. (*)
ToW (t)
())
(
Retraining
0.00 1 0.00
99.94 0.00
81.52 a 0.23
1,00 0.00
0.000.
48,96 1.21
Original
99.89 0.03
99.90 0.00
92.73 a 0.07
0.00 0.00
0.036 1.0
57.61 0.35
Rand. Lbls.
1.07
0
0.024
0.05
0.18
82.44
0.96 0.01
0011 + 0.001
GD
99.97 0.00
99.86 0.01
0.117
0.00
0,066000 0.D
57.81 0.50
GA
4.35 1.36
43.64
14.91
0.23 12.14
0.029 + 0.001
48.73
0.88
GDif
18.18 1.33
7 3.28
66.92
3.311
0.53 0.04
0.036 1 0.0
49,93
0.11
a 1.32
78.26 1.35
0,036 0.0
SCRUB
0.00 1 0.00
85.92 1 1.33
75.43 a 0.83
0.78 0.00
0.008 2 0.002
49.68
0.99
CURENUS
1.82 1 1.97
87.25 0.77
74.52 a 0.67
0.77 0.002
0.077 1 0.003
$1.55 a 1.87
Loss Dienibution Non-over/ittd Original Mocel
Loss Uniriibation on Non-over filted Hatt sining Model
C. Vaud but una
6:10
Eresed S0t
0.03
0.5
3.0
1.
2.0
Low
Dicributtoon Overfitted Original Modal
Distribution on Retraining
2:06
0.04
9.02
0
0.00
-0.5
0.0
0.5
1.0
1.5
2.0
25
20
30
10
Figure 3: Loss distributions for samples on non-overfitted and overfitted, original and retraining models.
Table 7: Performance and MIA results for CURENUS and other tested baselines in sequential unlearning setting on Llama-2 x TOFU where the test set is in similar distribution as the forget set.
Method
Llama-2 x TOFU
Dr. ROUGE (-)
D2 ROUGE (-+)
Di. ROUGE (-+)
Truth Ratio
ToW
MIA (-)
Retraining
0.386 a 0.004
0.576 a 0.005
0.746 a 0.027
0.664 a 0.012
1.00 = 0.00
63.72 &: 0.93
Original
0.628 0.008
0.570 1 0,907
0.679 1 0.068
0.513 a 0.011
0.69 0.03
99.04 0.26
GD
0.497
0.612
0.545 0.005
8.63
GA
0.000 a: 0.000
0.000 a 0.000
0.000 a: 0.000
0.521 4 0,024
0.07=0.01
81.63 :: 16.03
GDaf
0.09 0.09
0.09 0.09
0.00 + 0.00
0.81 0.05
0.10 0.02
91,98 0.99
IDK
0.112 0.024
0.462 : 0.009
0.668 : 0.025
0.570 0.010
0.59
96.17 0.27
SCRUB
0.598 a: 0.020
0,549 1 0.011
0.657 1 0.083
0,508 0,007
0.69 2 0.05
98.83 1: 0.23
NPO
0.62 0.16
0.43 0.06
0.79 0.13
0.11
0.56 0.01
71,79 1.23
CURENUS
0.477 = 0.011
0:495 # 0.006
0.678 = 0.021
0.570 = 0.001
0.76 = 0.02
97.54
Table 8: Performance and MIA results for CURENUS and other tested baselines in batch unlearning setting on overfitted CNN x FMNIST (averaged over random runs).
Overfitted CNN
FMNIST
Method
D, Acc. (-)
D, Acc. (-)
Dust Acc. (+)
ToW (*)
IS Div. ())
MIA (-)
Retraining
0.00 +
86.07
76.62 + 0.11
1.00
0.000 + 0.000
54.04 + 0.56
Original
92.57
84.55 0.80
84.32 = 0.93
0.07 0.03
0.024 0.004
53.90
Rand. Lbls
4,70
74.74 1 6.43
67.02 : 5.53
0.77 : 0.10
0.007 : 0.002
50.04 :: 0.48
DELETE
144 1.29
82.67 1.29
73.58 + 1.18
0.92
0.002 = 0.001
50.84 1.35
GD
92.21 3.45
84,58 = 0.78
84.30 1 0.88
0.07 1 0.03
0.023 : 0.004
53,99
= 0.37
GA
0.52 4:
76.82 a 4.25
68.38 + 3.77
0.83 a: 0.07
0.005 4 0.003
51.34 0.75
GDif
0.28
74.32
66.20 8.99
0.80
0.006 0.004
52.11
NPO
5.81 3.94
81.04 1.21
72.77 + 1.05
0.86 0.04
0.003 à 0.001
$1.27 à 0.67
SCRUB
0.00 - 0.00
82.95 = 0.29
73,90 = 0.31
0.04
0.006 = 0.001
48.33 3.49
CURENUS
0.64 + 1.04
81.65 1.66
72.79 1.41
0.91 = 0.03
0.003 0.001
$1.20 0.18
28

---

Published as a conference paper at ICLR 2026
samples. This makes it more challenging for the MIA attacker to perform the binary classification, which thus results in not very high accuracy for both original and unlearned models. On the other hand, in our experiments on TOFU, the distribution of Diest samples in the TOFU dataset is relatively different from D. samples, e.g., samples contain different question-answering content. This leads to different loss distributions on De and Dtest samples that are naturally separable, which results in the large MIA AUC as shown in Table 3] where even Retraining reaches as high as 78.46. Therefore, we conduct new experiment by selecting 400 samples from the original D, form a new Diest to enforce a similar distribution as De. This also results in a new D' with 3. 200 samples. We train Llama-2 on the new training set, then perform unlearning and evaluate the results. [able[7showe the performance and MIA results on this setting, where the MIA results are closer to 50 and Retraining achieves 63. 62. Based on the results, while our CURENUS does not achieve the optimal MIA result, it demonstrates competitive unlearning performance by decreasing the D. ROUGE and preserves the model utility well with high D2 and Diest ROUGE.
H SUPPLEMENTARY EXPERIMENTS FOR SEQUENTIAL UNLEARNING
H.1 FULL RESULT FOR SEQUENTIAL UNLEARNING IN MAIN PAPER
Here, show the full results that includes Dtest Acc. and ROUGE of sample-level sequential unlearning on Llama-2 x TOFU (Fig. 4 top) and class-level sequential unlearning on ResNet-18 x CIFAR-10 (Fig. 4 bottom). The results show that CURENUS preserves decent performance on Dtest across the unlearning rounds in the sequential unlearning setting.
Retraining
Original
Hand Lbls SCRUB
DELETE NPO
GA
GDIff
CUReNUS (ours)
Dustt ROUGE
De ROUGE
D, ROUGE
0.5
0.2
0.25
0.0
2
Unlearning Rounds
* Unlearning
Unleaming Rounds
Diest Acc. (3%
Acc. (%)
D, Acc. (%)
100
90
50
45
5
Unleamiin Rounds
# Unlearning Rounds
Unleariing Rounds
Figure 4: Full results on sample-level sequential unlearning on Llama-2 x TOFU (top) and class-level sequential unlearning on R8 x CIFAR-10 (bottom) with 5 unlearning rounds (averaged over 5 random runs).
H.2 INCREASED UNLEARNING ROUND
In this experiment, we extended our sequential unlearning experiments to 10 rounds using ResNet-18 on the CIFAR-10 dataset. We increase D. to include two full classes (20% of the full dataset, or 10, 000 samples). The experiment is conducted using the random seed 1. Despite the increased number of unlearning rounds and larger D., our results in Fig. and Table 9 that our CURENUS can perform relatively well compared to the strong baselines by staying close to Retraining throughout the increased unlearning rounds and achieving the best ToW score at the end of sequential unlearning.
H.3 CLASS-LEVEL SEQUENTIAL UNLEARNING ON LLAMA-2 X AG-NEWS
In this experiment, we perform class-level sequential unlearning with Llama-2-7B on the AG-News dataset (Zhang et al., 2015). The number of unlearning rounds is set to 3. which corresponds to 10,000 samples to be unlearned per round. The results in Table [0] and Fig. [6) show that our CURENUS achieves good forgetting under the class-level setting, reaching a similar Acc. as SCRUB on D., well decent model utility by maintaining the Drest Acc. and D, Acc..
29

---

Published as a conference paper at ICLR 2026
Retraining Original
-
GA
SCRUB
GDiff
DELETE
Lbs..
CuReNUS (ours)
Diuee Acc. (%)
D. Acc. (%6
D, Acc. (%)
90
60
30
Unleerning
UUrieemnng Rounds
Unleerming Rounds
Figure 5: Extended sequential unlearning with 10 unlearning rounds on ResNet18 x CIFAR-10 (seed
1).
Table 9: Unlearning performance the last unlearning round for extended sequential unlearning with 10 unlearning rounds on ResNet18 x CIFAR-10 (seed 1).
ResNet18 CIFAR-10
Method
D, Acc. (-+)
D, Acc. (-))
Dost Ace. (-)
ToW (?)
JS Div. ())
MIA (-+)
Retraining
0.00
99.40
81.56
1.00
0,000
53.36
Original
95.54
98.96
88.63
0.04
0.031
50.66
Rand. Lbls.
0.00
17.67
16.92
0.06
0.014
50.74
DELETE
0.00
97.63
80,23
0.97
0.007
50.26
GD
94.34
99.25
88, 84
0.05
0.030
49.85
GA
0.00
17.82
16.09
0.06
0,032
53.18
GDiff
0.00
47.85
42,06
0.29
0.028
$2.56
SCRUB
0.00
95.02
80.56
0.95
0.007
50.69
NPO
0.00
22.33
19.65
0.09
0.032
52.51
CUReNUS
0.38
96.98
80.98
0.97
0.006
$3.72
Retraining Original
Rand. Lbls.
GDIff
DELETE
SCRUB
NPO
CuReNUS (ours)
GD
Drest Acc. (%)
De Acc. (%)
D, Acc. (%)
100
100
100
60
60
20
20
1
Unlearning Rounds
# Unlearning Rounds
# Unleaming Rounds
Figure 6: Class-level sequential unlearning Llama-2 x AG-News (averaged over 3 random runs).
Table 10: Unlearning performance at the last unlearning round for 10% class-level sequential unlearning on Llama-2 x AG-News (averaged across random runs).
Method
Llama-2 x AG-News
D. Acc. (-+)
D, Acc. (-)
Dus Acc. (-)
ToW (*)
JS Div. (J)
MIA (-+)
Retraining
0.00
95.20 1.04
70.62 0.85
1.00
0.000 0.000
49.96 0.32
Original
98.24 1.77
94.59 0.55
94,60 0.08
0.01 + 0.01
0.014 = 0.008
49.47 1.95
Rand. Lbls
0.00 = 0.00
79.19 7.75
59.01 = 5.76
0.75 0.13
0.019 = 0.025
50.65 = 0.89
DELETE
93.48
69.51 = 85,42 = 1.13
0.030
GD
59.21 = 3.80
95.64 = 0.66
0.34 + 0.03
0.045 0.017
50.31 = 1.31
GA
0.00 + 0.00
33.33 0.00
25.00 0.00
0.00
0.21
0.028 = 0.027
49.90
GDiff
0.00
62.56
46.90 + 7.06
0.12
0.015 0.008
49.65
SCRUB
9.69 : 3,33
96.49 0.68
73.14 1.09
0.87 0.04
0.018 = 0.012
49.42 = 1.52
NPO
0.01
4.56
31.37 3.54
0.28
0.028 0.020
49.89
CoRENUS
3.95 2.85
90.03 0.61
67.91 0.99
0.89 +
0.023 0.007
51.23 0.26
30

---

Published as a conference paper at ICLR 2026
H.4 SAMPLE-LEVEL SEQUENTIAL UNLEARNING
Noisy Data Removal. In this experiment, we mislabel 20% of the training set, such that the label is shifted right from the true label. We then perform sequential unlearning on mislabeled data for 10 rounds, each round removes 2% of the randomly selected mislabeled data. In this setting, unlearning can increase the model performance on D, and Diest by removing the noisy data that harms the model utility. Tab. IT shows our results, which demonstrate that our CURENUS performs well in removing noisy data and preserving the model utility.
Table 11: Unlearning performance at the last unlearning round for noisy data sequential unlearning on ResNet18 x CIFAR-I0 (averaged over 3 random runs).
Mothod
ResNet18
CIFAR-10
Acc. (-))
D, Ace. (-+)
Doca Acc. (-)
ToW (t)
(U)
MIA (-+)
Retraining
6,88 0.
100.00 + 0.00
89.83 + 0.02
I 1.00 0.00
0,0 0.0
53.55 + 3.46
Original
26.41 0.94
96.09 = 0.05
81.97 + 0.52
0.71 0.01
2.9e-6 + 0.8c-6
52.95 + 1.62
Rand. Lbls.
11.70
9.96 1.66
9.93
0.02 = 0.00
2.8c-6 + 1.
54.44 0.45
GD
13.39
97.90 = 0.05
86.42 =
0.8% 0.91
3.7e-6 #: 1.7e-6
51.80 :: 3.13
GA
6.98 +:511
9.05 0.86
9.04 + 1.27
0.02 0.00
19:8c-6 22.70-6
49.04
SCRUB
2.52 0.23
80.43
0.02
3.5e-6 + 1.3e-6
51.28 3.54
CURENUS
9.06 1 0.69
96.26 0.36
85.92 0.20 0.90 0.00
3.7e-6 1.7e-6
54.67 1
Benign Data Removal. In this experiment, we perform sequential unlearning to iteratively remove random training subset across multiple rounds, which we refer to as sample-level sequential unlearning. Fig. 7 and Tables 12 13 show our results for sample-level sequential unlcarning of 10% randomly selected training data on CIFAR-I0 10 and AG-News. As observed, CURENUS maintains a close performance to Retraining and comparable results to SOTA methods (DELETE, SCRUB) on De and does not degrade model performance on Drest and D, even after multiple unlearning requests. This reinforces our argument that CURENUS is good unlearning algorithm for long-term settings such as sequential unlearning on both class and sample levels.
Retraining
SCRUB
CuReNUS (ours)
GDiff
Driginal
Rand. Lbis.
GD
NPO
DELETE
Dostt Acc. (%)
D. Acc. (%)
D, Acc. (%)
95-
95
95
60
60
60
25
25
25
3
Unleam ing Rounds
Unlearning Rounds
# Unleam ing Rounds
Drest Acc. (%)
D. Acc. (%)
D, Acc. (%)
90
60
60
30
30
30
5
3
5
1
Unleaming Rounds
Unlcarning Rounds
# Unleaming Rounds
Figure 7: Sample-level sequential unlearning of 10% training data. Top Row: Llama-2 x AG-News (2000 unlearned samples per round). Bottom Row: ResNet-18 x CIFAR-10 (1000 unlearned samples per round).
H.5 SEQUENTIAL UNLEARNING ON LLAMA-2 WITHOUT LoRA
Our main results on Llama-2 focus the LoRA-tuned setting because LoRA (Hu et al.) (022) is now a widely adopted approach for fine-tuning LLMs due to its efficiency and competitive effectiveness to full-scale fine-tuning (Schulman & Lab 2025). Furthermore, fine-tuning datasets are often
31

---

Published as a conference paper at ICLR 2026
Table 12: Unlearning performance at the last unlearning round for 10% sample-level sequential unlearning on Llama-2 x AG-News (averaged over 3 random runs).
Llama-2 AG-News
Method
D, Acc. (-+)
D, Acc. (-)
Dost Acc. (+)
ToW (*)
IS Div. (J)
MIA (-+)
Retraining
94.55 = 0.21
95,41 0.11
94.53 = 0.17
1.00 + 0.00
0.000 + 0.000
49.94 = 0.42
Original
95.36 0.10
95.50 =
94.63 = 0.06
0.99
0.027 = 0.013
50.06 0.20
Rand. Lbbls.
28.17 0.46
28.21 0.70
28.23 + 0.72
0.04 + 0.00
0.014 0.011
49.71 + 0.69
GD
96.39
97.90
94.84 0.13
0.95 0.00
0.021 = 0.015
49.10 0.62
GA
24.91
2.00
25.02 0.02
0.03 0.00
0.018
50.24
SCRUB
96,46
98.00 0.24
95.11 + 0.09
0.95 + 0.01
0.028 + 0.010
49.86 0.18
CURENUS
91.46 2.53
91.57 2.26
90.35 = 3.04
0.90 0.07
0.018
49.97 = 0.35
Table 13: Unlearning performance at the last unlearning round for 10% sample-level sequential unlearning on ResNet18 x CIFAR-I0 (averaged over 3 random runs).
ResNet18 CIFAR-I0
Method
D, Acc. (-+)
D, Acc. (-+)
Dust Acc. (-)
ToW (*)
IS Div. (J)
MIA (-+)
Retraining
83.64 a 0.80
85.89 0.68
83.01 + 0.62
1.00 =: 0.00
0.0 0.0
50.03 + 0.11
Original
87.31 = 1.05
87.11
84.74 +
0.94 0.01
0.028 + 0.004
51.64
Rand. Lbls.
41.61 2.34
41.46 = 2,80
40.28 = 1.42
0.19
0.013 4
50.41
DELETE
84.35 2.02
84.49 = 2.28
82.61 + 2.04
0.96 0.03
0.029 1: 0.001
50.67 0.83
GD
9.33
90.86 0.08
88.30 0.19
0.84 0.02
0,002
51.55 0.70
GA
7.56
44.29 +: 7.48
42.75 a 6.51
0.032 0.0
50.23 + 0.32
GDiff
75.98
NPO
68.80 5.49
69.14
67.24 + 4,78
0.60 0.10
0.029 0.002
51.23 = 1.46
SCRUB
85,42 a 0.84
0.83 87.26
84.74 a 0.82
0.95 à: 0.02
0.003 +: 0.0
50.50 +: 0.48
CeRENUS
86.54 1.01
87.25
84.77 = 1.17
0.93
0.003 +
50.57 = 0.24
task-specific datasets that are more privacy-sensitive than the public datasets used for full-scale LLM training, making them particularly relevant for unlearning, Nonetheless, we conducted an additional experiment on Llama-2-7B without LoRA on the TOFU dataset with a similar sequential unlearning setup Sec. 6.3 to verify the scalability of our method to full-scale LLM fine-tuning.
We provide our results in Tab. 14 which show similar trend to our results for LoRA-tuned LLM unlearning in Tab. B While CURENUS does not achieve the best forgetting efficiency. as shown by Dr ROUGE gap of 162 from retraining, shows a better unlearning trade-off between forgetting and utility preservation than SOTA methods like SCRUB via higher ToW score. More importantly, the fact that the Original model achieves the best ToW score amongst all tested methods shows that most of them are likely insufficient for large-scale unlearning, leading to either worse forgetting efficiency (observed GD, CURENUS) or worse utility preservation (observed in GA, GDiff, IDK, NPO, SCRUB). This challenging setting calls for the development of more robust unlearning methods in the future.
Table 14: Unlearning performance at the last unlearning round for sample-level sequential unlearning on Llama-2 TOFU without LoRA (seed l).
Llama-2 x TOFU
Method
D. ROUGE (-)
D, ROUGE (-+)
D ROUGE (-+)
Truth Ratio (+)
ToW (*)
MIA (-+)
Retraining
0,399
0.672
0.861
0.695
1.000
88.93
Original
0.692
0.686
0.539
0.674
100.00
GD
0.677
0.904
0.891
0.559
0.538
99.62
GA
0.000
0.000
0.000
0.344
0.027
10.44
GDiff
0.032
0.207
0.198
0.484
0.114
6.68
IDK
0.066
0,373
0.502
0.616
0.300
99.20
NPO
0.078
0.082
0.002
0.805
0.039
55.37
SCRUB
0.321
0.404
0.611
0.674
0.506
79,21
CORINUS
0,561
0.535
0.695
0.547
0.603
99.99
32

---

Published as a conference paper at ICLR 2026
I FULL RESULTS FOR UNLEARNING EFFICIENCY
Tab. 15 (an expanded version of Tab. in the main paper) shows the running time comparison (in seconds) among different unlearning algorithms to unlearn a batch of erased data points across various datasets and models. As anticipated, the unlearning algorithms that utilize the second-order information, such as PINV-Newton, Damped Newton, and CURENU, have the longest running times and even exceed that of Retraining on FMNIST. Therefore, these algorithms are impractical for large-scale experiments with ResNet18 8 and Llama-2. On the other hand, GA, NPO, and Rand. Lbls. are fast unlearning algorithms but tend to significantly degrade model performance post-unlearning, especially in long-term settings such sequential unlearning (Sec. 6.3 and H.4. GD and GDiff are also fast unlearning algorithms, but they do not unlearn effectively. In contrast, DELETE is strong state-of-the-e art method that performs unlearning both efficiently and effectively. Mcanwhile, CURENUS can maintain decent efficiency across various datasets and models and be more efficient than the sate-of-the-ea method SCRUB despite being second-order method by leveraging the fast HVPs and the stochastic setup, while maintaining a decent erasing quality and post- unlearnnnn performance.
Table 15: Running time comparison (in seconds) across different datasets and models (averaged over 3 random runs).
Datasct
EMNIST
CIFAR-10
AG-News
TOFU
Model
2-layer CNN
ResNer18
Llama-2-78 (+LoRA)
Llama-2-7B (+LoRA)
Trainable Parameters
20.728
11.173,962
1,064,900
2,097,152
Retraining Rand. Lbls.
61.20 e 8.70
1.0%
124.51 10.95
1,0x
4792,44 1 145.90
1.0x
900.71 +: 2.57
1.0x
1.70 = 0.19
0.03x
2.58 = 0.10
0.02x
144.63 + 1.83
0.05x
DELETE
0.89 + 0.10
0.01
6.71 + 0.05
0.05
133,54 + 3.90
0.03x
GD
0.1x
19.16
0.2x
4641.50
407.93
0.96x
0.41
GA
2.28 + 0.58
0.03x
5.78 + 0.26
0.04x
105.46 + 2.85
0.02x
5.61 + 0.63
0.01x
GDifT
0.07
0.10x
50.38
PINV-Newton
6185.72
804.94
101.1
Damped Newlon
6228.88
739.82
101.7x
SCRUB
23.33 a 0.43
0.4x
72.39 + 4.93
0.6x
6796.16 a 160.11
1.4x
178.52 a: 0.39
0.20x
IDK
37.94
0.04x
NPO
0.80 a 0.02
0.01
0.87 a: 0.05
0.01
134.34 #2.27
0.03x
31.02 a 0.24
0.03x
CORENU CURENUS
6355.11 127.31
35.54 & 6.73
0.6x
41.79 = 0.94
0.3
85.26 : 18.23
0.02x
340.24 a 61.04
0.38x
J ABLATION STUDIES
Retraining Original
SCRUB PINV-Newtton
Damped Newton
CuReNU (ours)
+ Rand. Lbls.
GD
NPO
CJReNUS (ours)
Decee Acc. (%6)
De Acc. (%)
Dp Acc. (%)
90
90
45
45
45
0
01
0
40
60
40
60
20
40
60
# Unleaming Rounds
Unlearning Rounds
Unleaming Rounds
Figure 8: Class-level sequential unlearning performance on FMNIST (averaged over 3 random runs).
Throughout our ablation studies of CURENU and CURENUS, we adopt class-level sequential unlearning setting with 60 unlearning rounds on the FMNIST dataset. We choose class 2 to be unlearned and report the averaged results over 3 random seeds {125, 126, 127). Fig, presents the sequential unlearning performance for this setup. The remaining ablation results are reported at the last unlearning round.
33

---

Published as a conference paper at ICLR 2026
J.1 EFFECT OF VARYING
Since the exact Hessian Lipschitz constant L is often hard to find, we treat it as hyperparameter and analyze the effectiveness of our algorithms with varying choices of in Fig. 9 As can be seen, both algorithms exhibit consistent performance and remain close to the retraining baseline across different values. This highlights the robustness of CURENU and CURENUS with respect to the choice of L, in contrast to the strong dependence learning rates in the many first-order unlearning algorithms such as GD, GA, and Rand. Lbls..
Retraining,
Retraining, Drest
Deest
CuReNU
CuReNUS
Accuracy (%)
90
Accuracy (%)
90
45
45
0
0
50
100
150
200
50
100
150
200
L
Figure 9: Performance of CURANU and CURENUS for different empirical choices of L (seed 125).
On separate note, it is important to keep L large enough (e.g., approximately L 70 for CURENU, as marked by the shaded region) to ensure the validity the cubic approximation and maintain convergence guarantees. Nonetheless, an appropriate value of L can be determined during training and before unlearning actually happens.
How to find valid empirical L? It is infeasible to empirically estimate the exact Hessian Lipschitz constant L in neural networks, as it would require enumerating over the entire space of model parameters. To address this, have adopted more practical approach that is also suggested in Section 5.2 of Nesterov & Polyak| (2006): starting with a random guess of L,, we increase if the model fails converge (since valid upper bound would induce convergence); otherwise, L, or decrease L and check a reduced L still induces convergence. Since the actual L is an upper bound, this procedure is guaranteed return valid empirical L.
can use
Caveats for non- -ipshhiss models. When the model fails to satisfy L-Lipschitz Hessian (i.c., no valid L exists), selecting sufficiently large L remains beneficial can act as an effective regularizer to prevent large steps (large-norm updates) in suboptimal directions.
J.2 EFFECT OF VARYING a IN STOCURENU
Tab. T6 presents unlearning performance for varying levels of gradient perturbation a on CNN x FMNIST. We find that CURENUS achieves comparable unlearning performance to retraining across wide range of a. This observation excludes a = 100, where the stochastic gradient is heavily perturbed. Since gradient perturbation is intended to prevent the "hard case" (Conn et al. 2000), often sufficient to use small a, such < 1, in our experiments while maintaining the fidelity of the stochastic gradient.
De Acc.
D, Acc.
Dist Acc.
10-3
0.072 0.097
87.568 + 1.640
76.525 1.161
10
0.000 0.000
87.922 = 3.056
76.854 + 2.324 2.324
10 1
1.072 1.857
86.638 2.379
75.820 1.582
0.000 1.279
87.914 0.887
76.712 + 0,977
0.072 0.125
87.696 + 2.834
76.729 2,533
100
0.016 # 0.016
43.193 = 9.832
37.441 8.486
Retraining
0.000 0.000
92.466 + 0.883
80.225 0.963
Table 16: Effect of CuRENUS (averaged over random runs).
34

---

Published as a conference paper at ICLR 2026
J.3 EFFECT OF VARYING T IN STOCURENU
Fig. shows the unlearning performance when varying the number of stochastic iterations T in CURENUS, As T increases, CURENUS better approximates the retraining performance on Dtest, D.. and D,. This illustrates the inherent trade-off between unlearning performance and computational efficiency in CuRENUS. Nonetheless, we observe that CuRENUS can obtain good unlearning performance with only around 10-20 iterations, offering significant advantage over full retraining. In practice the number of stochastic iterations in CuRENUS can be tuned meet specific unlearning requirements.
Retraining
T= 10
T 30
T 50
T- 100
7-1 1
T-20
T- 40
T- 70
T- 200
rest Acc. (%)
De Acc. (%)
D, Acc. (%)
100
96
88
50
95
94
20
40
20
40
20
40
# Unlearning Rounds
# Unlearming Rounds
# Unlearming Rounds
Figure 10: Effect of the number of stochastic iterations
CURENUS (seed 125).
J.4 OFFECT OF VARYING BATCH SIZE IN STOCURENU
We assume the same batch sizes are used for gradient evaluation and Hessian-vectue product evaluation in CURENUS, i.e.,
n2 15 Tab. 17 presents unlearning performance on CNN x FMNIST for varying batch sizes. Our results show that CURENUS with mini-batch sampling can achieve comparable performance (within 2-4% difference on De/D./ De/er/tteat Acc.) to the full-batch setting (i.e, batch size D, suggesting that the sampling-indee induced stochasticity might have minimal impact unlearning effectiveness.
Batch Size
D, Acc.
D, Acc.
Dreat Ace.
32
1.839 # 2.899
87,445 1.585
76.400 # 1,471
64
1.322 = 1.002
88.863
2.724
77.683 2.711
128
1.344 1.078
89,954 1.604
78.958 1.388
256
1.383 2.033
90.383 2.084
79.138 1.426
512
4
79.950
0,850
1024
0.133 = 0.211
89.820 + 3.593
78.321 3.157
2048
0.150 = 0.246
89.958 3.461
78.433 3.043
D,I
0.206 0.178
90.018 3.360
78.542 2.860
Retraining
0.000 = 0.000
92.466 0.883
80.225 0.963
Table 17: Effect of batch size in CURENUS (averaged over 3 random runs).
J.5 ANALYSIS OF DUAL VARIABLE a IN CURENU
We set the number of unlearning iterations T for CURENU and observe the value of a during class-level sequential unlearning on FMNIST in Fig. 11 Compared to the damping factor 10-3 in Damped Newton, the dual variable in CURENU often admits larger values, which effectively prevents the model from excessively large norm updates.
During sequential unlearning, the value of consistently decreases until around round 50. This may imply that the Hessian becomes more well-behaved over time, and less regularization is needed. After round 50, however, increases again. This allows for larger norm updates, which may be needed for class unlearning.
In practice, CuRENUS can sample hatches of different sizes for gradient evaluation and Hessian-vector product evaluation.
35

---

Published as a conference paper at ICLR 2026
0.020
0.018
0.016
# Unleaming Rounds
Figure 11: The dynamics of our CuRENU method (averaged over random runs).
J.6 EFFECT OF VARYING LEARNING RATE IN GD
Prior works have noted that the performance of gradient descent (GD) is highly sensitive to the choice of learning rate (Schaul et al. 2013). Here, we provide empirical evidence that supports this claim when GD is applied for unlearning in Fig. [2] As we can sec, its performance on De varies significantly across different learning rates. This suggests that identifying the optimal learning rate for GD to achieve unlearning performance close to retraining may challenging in practice.
Retraining, D, Retraining, Dust
D.
Dust
Accuracy
45
10-4
10
10
10
Step Size
Figure 12: Effect of varying learning rates in GD (seed 125).
K FORMAL DEFINITION OF MACHINE UNLEARNING
Here, we provide a formal definition of machine unlearning used in (Nguyen et al. (022).
Let D {(x,, y C X' x Y' denote the training set of n samples, where x; Bd is the input and y: E IR is the corresponding target. Let D. C denote the erased set ng samples to unlearned and D, \ De denote the retained set of n,. remaining samples.
Let Pr(A(D)) denote the model distributions trained on D using a randomized learning algorithm A. We denote by U unlearning algorithm that takes the training set D, the erased set De, the trained model A(D) E H. and returns an unlearned model in H. U is deemed exact unlearning iff
VT C HH,
then Pr(A(D,) e T) Pr(U
T).
Exact unlearning algorithms, such as Bourtoule al. (202T); Yan et al. (2022). are often expensive to perform, especially for neural networks due to massive D and large model sizes. Therefore, our goal to achieve a relaxed notion of approximate unlearning, i.e.,
VT C H4, D C X x D. D DD then Pr(A(D,) E Dr. T
We can view A(D) as a function mapping any input x to real outputs (e.g., predicted logits for classification tasks) and H. defines the set of all such models. Hence, our target must achieve similar outputs to retraining on any data. This motivates us to achieve the same retraining loss in our work.
L JUSTIFICATION OF UNLEARNING GOAL
The common goal of unlearning algorithms in Sections 3.3
5.1 and 5.2 is to minimize the loss L(w; D,) on the retained set achieve unlearning of D.. This choice is widely used in existing
36

---

Published as a conference paper at ICLR 2026
works (Guo et al. 2020: Neel et al. 2021 Sekhari al. 2021 and adopted in work because it forms a necessary condition to approximate a retrained model on D,, i.e., if an unlearned model does not minimize [(w; D,), it cannot be the same strongly-convex losses with at most one minimum, this necessary condition is also a sufficient condition. Nonetheless, our unlearning algorithms can be extended to other loss formulations, such as a weighted combination of C(w; D,) and -[(w; De). Empirically, we find that incorporating a small negated gradient on De helps prevent the model from being trapped at the original solution when minimizing the retraining loss.
retrained model. In special cases like
M BACKGROUND IN OPTIMIZATION
As demonstrated in Sections 3.3 and 5.2 we can view unlearning as an optimization process that starts from local minimum of the loss on D (the original model) and seeks nearby local minimum of the loss on D, (a retrained model). To approach unlearning, we can study the broader optimization literature, which broadly categorizes optimization methods into first-order methods and second-order methods To These optimization methods are further distinguished by their behavior in convex and non-convex settings, with unlearning neural networks typically situated in the more challenging non-convex setting.
Convex vs. Non-Convex Optimization. Convex optimization involves minimizing convex objective function (or maximizing a concave one) over a convex feasible set. In convex optimization problems, any local minimum is also global minimum. Moreover, if the objective function is strictly convex, the local minimum, if exists, is unique. In contrast, non-convex optimization arises when the objective function is non-convex. In non-convex optimization, a first-order stationary point (where the gradient is close to 0) may correspond to local minimum, a local maximum, or saddle point. In contrast, a second-order stationary point (where the gradient is close to and the Hessian is nearly positive semi-definite) can help avoid most saddle points and local maxima with strong negative curvature. Although finding a local minimum may not correspond to a global minimum, empirical studies show that many local minima in non-convex problems tend to have objective values nearly as good as the global minimum (Kashyap 2022).
First-Order Methods. Gradient descent (GD) and its variants (e.g., stochastic gradient descent (SGD), momentum (Sutskever et al. 2013), and Adam (Kinga et al. 2015) are widely used first-order optimization methods in machine learning due to their scalability high-dimensional data and complex models, such as deep neural networks. We subsequently restate the convergence results for
GD and SGD. Interested readers can refer to (Nesterov 2013 Garrigos & Gowwr 2023 Khaled &
Richtárik 2020) for more details.
For convex function with Lipschitz continuous gradient, GD achieves a convergence rate of O()) (sublinear convergence), where k is the number of iterations. If the function is strongly convex, GD enjoys a linear convergence rate (ct) for some constant < e < 1. For non-convex optimization, GD can only guarantee convergence to an e-first-order stationary point (s-FOSP) in O(s iterations.
In the stochastic setting, SGD converges in expectation at a rate of O ((+)) for convex functions and (+ for strongly convex functions. If the function is non-convex, SGD can only guarantee convergence an s-FOSP in O(= 4) iterations.
Second-Order Methods. Many real-world functions, such as objective functions in neural networks, are inherently non-convex. Although first-order optimization methods like GD and SGD are computationally efficient and scalable, they often suffer from slow convergence in ill-conditioned problems and may get stuck at saddle points if the function is non-convex. method leverages the second-order information of the function to accelerate the convergence. When the function is strongly convex with a Lipschitz continuous Hessian, and the initialization is sufficiently close to the minimizer, Newton's 's method converges locally at quadratic rate (Nocedal & Wright 2006). In general, however, its global convergence is not guaranteed. Moreover, Newton's 's method may instead converge to saddle points or maxima, it does not distinguish among stationary points.
contrast, Newton's
16hile higher-order methods exist, they rarely adopted due significant computational and numerical instability.
37

---

Published as a conference paper at ICLR 2026
To achieve global convergence, cubic regularization methods (Nesterov & Polyak 2006 construct global upper bound of the objective function by adding a cubic term to the quadratic approximation. For function (possibly non-convex) with Lipschitz continuous Hessian, cubic-regularized Newton's method can achieve convergence to an e-second-order stationary point (e-SOSP) in (s iterations. For small converging e-SOSP can help avoid most saddle points and sharp local maxima.
In the stochastic setting, Kohler & Lucchi (017) proposes a variant of cubic regularization that utilizes subsampled gradients and Hessians, but does not provide its asymptotic analysis. Xu al. (2020) considers stochastic Hessians but still requires access to the full gradients. Leveraging HVPs avoid explicit Hessian computation, Tripurameni al. (2018) proposes scalable Hessian-free variant that converges an e-SOSP in (c)).)) gradient/HVP evaluations.
38


Reviews:
[{"review_id": "1cBLUxi6YP", "reviewer": "Reviewer_mMKS", "summary": "This paper studies the challenge of machine unlearning i.e. removing the influence of specific training data from a trained neural network without retraining from scratch. The authors focus on improving second-order unlearning methods such as Newton Unlearning, which rely on Hessian information to approximate retraining. They identify a critical limitation: Hessian degeneracy (presence of many small, zero or negative eigenvalues) in trained neural networks, which leads to unstable or divergent updates during unlearning. To overcome this, the paper proposes two novel algorithms: CuReNU (Cubic-Regularized Newton’s Unlearning) – uses cubic regularization to automatically determine an appropriate damping factor, thereby stabilizing updates and it's stochastic variant StoCuReNU (Stochastic CuReNU) – a scalable, Hessian-free variant using Hessian-vector products (HVPs). Empirically, on FashionMNIST, CIFAR-10, AG-News, and TOFU datasets, CuReNU and StoCuReNU achieve competitive unlearning performance compared to state-of-the-art empirical methods (e.g., SCRUB, DELETE) in both batch and sequential unlearning settings.", "questions": "1. How sensitive are CuReNU and StoCuReNU to the cubic regularization coefficient 𝐿?\n2. Can the authors provide empirical Hessian spectra for larger models (e.g., LLaMA-2 layers) to substantiate degeneracy at scale?\n3. How does the unlearning error accumulate across rounds? Is there a mechanism to “recalibrate” the model to prevent drift after many sequential unlearning steps?\n4. For large foundation models with mixed data sources, how feasible is StoCuReNU in practice?\n5. Can authors include runtime comparisons at least for a subset of experiments w.r.t. baselienes?\n\nAny discussion around these will help improve the paper over it's current state.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is very well-written, including notational consistency, and is very easy to understand.\n2. The problem formulation is very clean and I believe the authors point towards and study an important problem.\n3. The proposed CuReNU and StoCuReNU are derived from established optimization theory. The adaptation for unlearning is technically sound and mathematically well-justified.\n4. Experiments span both batch and sequential unlearning on diverse datasets (vision, text). Metrics (accuracy, JS divergence, ToW score, etc.) are carefully chosen and clearly reported.\n5. The figures and tables are clear and informative.", "weaknesses": "1. The proposed methods are direct adaptations of known optimization techniques. The novelty lies primarily in contextual application rather than new algorithms.\n2. The main results focus on small to medium models (CNN, ResNet-18, LoRA-tuned LLaMA-2). Full-scale LLM unlearning remains untested.\n3. Ablation studies that isolate the impact of cubic regularization vs. stochasticity are missing.\n4. Discussion around more efficient Hessian vector product is missing.\n5. While there are a lot of results that are included, the proposed methods are not always a clear winner.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "S423Jfp9xn", "reviewer": "Reviewer_9cot", "summary": "This paper studies the machine unlearning problem and proposes cubic regularized method to handle the potential problem of Hessian degeneracy. The proposed method CuReNU introduces damping to stabilize the Newton's method and have a systematic way to avoid stability issues. The authors further propose a stochastic version StoCuReNU method to bypass the $O(d^3)$ involved in matrix inversion using efficient Hessian Vector Product computations. The numerical experiments also supports the method favorably.", "questions": "In Appendix J.3.1 the authors  show that $T$ represents a direct trade-off between computational efficiency and unlearning effectiveness. Could you please provide a principled heuristic in selecting $T$? Or maybe an early stopping criterion?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "Machine unlearning is a very timely topic and Hessian degeneracy can be serious issue for computation stability. The adaptation of tools from classical nonconvex optimization (cubic regularization) into ML context is very well motivated. The usage of second order information is something the community has often overlooked and it is great to see bring brought back to the stage. The paper is technically sound, clearly presented.", "weaknesses": "The CuReNU and StoCuReNU are both adapted from existing algorithm in different setting and hence the convergene and theoretical guarantees are inherited. The authors say that \" this adaptation is both necessary and non-trivial to address failure modes\". Please clarify further what exactly had to be modified.\n\nWhile the theory shows favorable memory usage, the empirical results in Table 4 show STOCURENU's practical peak memory can be higher than its baselines. Is this due to a large constant factor (e.g., loading the base model, LoRA adapters, and HVP buffers) 1 and that the $\\mathcal{O}(2d)$ benefit is about asymptotic scaling as the dataset size $n$ grows? Is the problem too small scale for the asymptotic to kick in?\n\nIn Appendix G, the authors test on overfitted models and note that while STOCURENU is effective, a performance gap remains in Membership Inference Attack (MIA) mitigation compared to the SOTA empirical method, SCRUB. Please discuss this further. Is this gap expected for other scenarios as well or in general there is a gap?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "75pSwlKN19", "reviewer": "Reviewer_znzs", "summary": "This paper investigates Newton-like unlearning algorithms to address the Hessian degeneracy challenge. It proposes two methods, CuReNU and StoCuReNU, based on cubic-regularized optimization and analyzes the convergence guarantee. StoCuReNU achieves performance comparable to state-of-the-art empirical unlearning methods across diverse settings. The authors demonstrate that StoCuReNU is scalable with comparable unlearning performance for various settings,including batchand sequential unlearning.", "questions": "1. Since this is not the first work to propose second-order unlearning for neural networks, the authors should restate the central question more clearly: How can we unlearn neural networks effectively using second-order methods?\n\n2. StoCuReNU claims smaller space complexity than Qiao et al. (2025); how does its time complexity compare?\n\n3. Can lower-complexity approaches be used to approximate or invert the Hessian in CuReNU, such as Hessian–vector products or related techniques?\n\n4. Line 375: “Tug-of-War (ToW) score that aggregates these gaps (smaller is better)” — did you mean “larger is better”?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper investigates Newton-like unlearning algorithms to address the Hessian degeneracy challenge. It proposes two methods, CuReNU and StoCuReNU, based on cubic-regularized optimization. StoCuReNU achieves performance comparable to state-of-the-art empirical unlearning methods across diverse settings. Theoretic analysis of convergence guarantees seem solid.", "weaknesses": "1. This is not the first work to propose second-order unlearning for neural networks, so the contributions appear limited. For example, Qiao et al. (2025) and Zhang et al. (2024b) also introduce second-order unlearning algorithms for non-convex objectives. The paper therefore needs stronger motivation and a clearer articulation of its novel contributions.\n\n2. The authors compare against only a subset of existing unlearning algorithms; for instance, Qiao et al. (2025) is not included. Moreover, in the experimental evaluations, the proposed method does not perform noticeably better than the baselines, which further limits the contributions of this work.\n\n3. The experiments use only five unlearning rounds, averaged over three random runs, which seems quite limited.\n\n4. Accuracy is not a very convincing metric to evaluate the unlearning performance, Other common evaluation methods, such as membership inference attacks (MIA), are not considered.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Kj2RFLwuVP", "reviewer": "Reviewer_wc7S", "summary": "The paper identifies a key limitation in second-order unlearning methods like damped Newton, where Hessian degeneracy near local optima causes unstable and excessively large parameter updates that harm performance. To address this, the authors propose Cubic-Regularized Newton Unlearning (CuReNU) and its scalable variant StoCuReNU, which apply cubic regularization to automatically control the Hessian damping factor. This approach mitigates degeneracy, stabilizes update norms, and provides theoretical convergence guarantees to second-order stationary points.", "questions": "I would have appreciated some analysis of the results in Figure 2. In particular, the experiment for unlearning on Llama-2 still has 40\\% accuracy $D_e$ regardless of whether StoCuReNU or retraining is used, which suggests that the samples being targeted for unlearning are not unique within the training set. As a result, it is not clear if the experiments can be conclusive in this case. \n\nI was also confused on how the authors obtained a retraining time measurement for Llama2-7B. If I understand this should be the time to retrain Llama2-7B from scratch minus the data in the dataset they want forgotten. However, it's more likely this is the time to fine-tune Llama2-7B on the dataset. Thus, it seems misleading to call this the retraining time for Llama2-7B since that is not how a fully exact retraining method would work.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Unlearning is an important topic and the authors clearly explain and evaluate their proposal.", "weaknesses": "The evaluation is limited to small models or LoRA ona small LLM. Thus it is unclear how this will scale to large models where retraining is not realistic, and approximate unlearning is needed. Some parts of the evaluation technique are unclear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)