OFMU: OPTIMIZATION-DRIVEN FRAMEWORK FOR MACHINE UNLEARNING
Authors: Sadia Asif
Mohammad Mohammadi Amiri
Venue: ICLR 2026 Poster (2026)
Keywords: machine unlearning
large language models
privacy
bi-level optimization
convergence analysis
Trustworthy Machine Learning
Gradient-Based Methods
Safety in LLMs
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We propose OFMU, a penalty-based bi-level optimization framework for machine unlearning that prioritizes forgetting while preserving utility, with provable convergence and state-of-the-art performance on large language models and vision tasks.

Abstract: Large language models deployed in sensitive applications increasingly require the
ability to unlearn specific knowledge, such as user requests, copyrighted materi-
als, or outdated information, without retraining from scratch to ensure regulatory
compliance, user privacy, and safety. This task, known as machine unlearning,
aims to remove the influence of targeted data (forgetting) while maintaining per-
formance on the remaining data (retention). A common approach is to formu-
late this as a multi-objective problem and reduce it to a single-objective prob-
lem via scalarization, where forgetting and retention losses are combined using
a weighted sum. However, this often results in unstable training dynamics and
degraded model utility due to conflicting gradient directions. To address these
challenges, we propose OFMU, a penalty-based bi-level optimization framework
that explicitly prioritizes forgetting while preserving retention through a hierar-
chical structure. Our method enforces forgetting via an inner maximization step
that incorporates a similarity-aware penalty to decorrelate the gradients of the for-
get and retention objectives, and restores utility through an outer minimization
step. To ensure scalability, we develop a two-loop algorithm with provable conver-
gence guarantees under both convex and non-convex regimes. We further provide
a rigorous theoretical analysis of convergence rates and show that our approach
achieves better trade-offs between forgetting efficacy and model utility compared
to prior methods. Extensive experiments across vision and language benchmarks
demonstrate that OFMU consistently outperforms existing unlearning methods in
both forgetting efficacy and retained utility.

Published as a conference paper at ICLR 2026
OFMU: OPTIMIZATION-DRIVERR FRAMEWORK FOR MACHINE UNLEARNING
Sadia Asif, Mohammad Mohammadi Amiri
Rensselaer Polytechnic Institute (asifs, mamí ri eenerirrrp...o edu
ABSTRACT
Large language models deployed in sensitive applications increasingly require the ability to unlearn specific knowledge, such as user requests, copyrighted materi- als, or outdated information, without retraining from scratch to ensure regulatory compliance, user privacy, and safery. This task, known as machine unlearning, aims to remove the influence of targeted data (forgeiting) while maintaining per- formance on the remaining data (retention). A common approach is to formu- late this as multi-objective problem and reduce it to single-objective prob- lem via scalarization, where forgetting and retention losses are combined using weighted sum. However, this often results in unstable training dynamics and degraded model utility due to conflicting gradient directions. To address these challenges, we propose OFMU, a penalty-based bi-level optimization framework that explicitly prioritizes forgetting while preserving retention through hierar- chical structure. Our method enforces forgetting via an inner maximization step that incorporates a similarity- get and retention objectives, and restores utility through an outer minimization step. To ensure scalability, we develop two-loop algorithm with provable conver- gence guarantees under both convex and non-convex regimes. We further provide rigorous theoretical analysis of convergence rates and show that our approach achieves better trade-offs between forgetting efficacy and model utility compared prior methods. Extensive experiments across vision and language benchmarks demonstrate that OFMU consistently outperforms existing unlearning methods both forgetting efficacy and retained utility.
penalty to decorrelate the gradients of the for-
INTRODUCTION
Large language models (LLMs) have become foundational to applications ranging from search en- gines and coding assistants to healthcare, education, and scientific discovery. Their remarkable performance arises from training massive and diverse corpora. which inevitably contain sensi- tive, copyrighted, or harmful information. This raises serious concerns about privacy, regulatory compliance, safety, and ethics. In particular, regulations such the General Data Protection Regu- lation (GDPR) (European Union 2016; California Legislative Counsel 2018 Office of the Privacy Commissioner of Canada 2018) grant individuals the "right to be forgotten" (Dang) 2021) requiring deployed models to eliminate the influence of specific data upon request. Beyond regulatory man- dates, unlearning is also necessary to prevent models from generating toxic content, leaking private information, or providing instructions for misuse (Huang et al.) 2022; Carlini et al., 2023 Staab al. 2024). These considerations have led to growing interest in machine unlearning, the ability to selectively erase the impact of particular data from trained model while maintaining its utility.
Limitations of Existing Approaches. Existing unlearning methods for LLMS can be broadly cat- egorized into three families (see Appendix [7.6 for broader discussion): input-based, data-based, and model-based approaches. Input-based methods modify the prompts or instructions given to the model so that it refuses to generate content related to the forget set (Pawelczyk et al. 2023 Liu et al. 2024b). These methods are lightweight but typically brittle, as adversarial prompts can of- ten bypass the refusal policy. Data-based methods fine-tune the model on curated examples that encourage desirable outputs when queried with forget-related prompts (Choi et al., 2024). While effective in narrow settings, such methods risk semantic distortion, require careful construction of

---

Published as a conference paper at ICLR 2026
auxiliary data, and may not generalize beyond specific domains. Model-based approaches directly alter model parameters through techniques such as fine-tuning. gradient ascent, or projection (Bu et al. 2024 Fan et al. 2025 Dong et al. 2024 Zhang et al., (024b). These approaches are gen- erally more effective at suppressing unwanted knowledge, but they introduce a deeper optimization challenge: balancing the trade-off between forgetting the targeted information and preserving utility on the retain set.
Most model-based methods formulate this balance as a scalarized optimization problem, where the forget loss and retain loss are combined with fixed weights into a single objective. This design leads to several shortcomings. First, static weighting fails to reflect the dynamic nature of unlearning: early optimization steps should prioritize forgetting, while later updates should shift emphasis to- ward restoring utility. However, fixed weights cannot adapt accordingly. Second, scalarization inherently unstable. When the forget objective dominates, the model can collapse, leading to severe performance degradation on the retain set. When the retain objective dominates, forgetting remains incomplete and sensitive data can persist. Third, most existing algorithms perform poorly on hard- to-unlearn samples (see Appendix 7.5.6), where forget and retain gradients are strongly entangled. In such cases, aggressive updates retain set, as evidenced by the strong coupling between sample difficulty and utility loss. Figure [] further shows that while many methods perform adequately formance drops sharply as difficulty increases. This trend underscores their inability to maintain stable performance under challenging conditions, ultimately failing to meet the true objective of un- learning. Finally, existing approaches largely lack principled theoretical grounding, instead relying on heuristic weighting schemes that scale poorly in the nigh-dimensional, non-convex optimization landscapes of modern LLMs.
the forget set cause disproportionate collateral damage to the
easy-to-unlearn samples, their per-
These challenges call for a more princi- pled and structured approach to optimization- based unlearning, one that recognizes the asymmetric priorities of forgetting and reten-
Coupling Difficuty with Collateral
0.5
woree tright weroee
tion. Crucially, forgetting must take prece- dence. If model fails to unlearn harmful content, its retained capabilities are irrelevant from safety compliance perspective. In contrast, once forgetting is successful, utility can be gradually restored as long as the erased information does not re-emerge. This obser- vation motivates a hicrarchical optimization view, where forgetting is posed as a primary inner objective, and retention addressed as a secondary outer goal.
Undearting Dificcuity Index (JD1
Figure 1: Coupling of unlearning difficulty with collateral utility loss. Harder samples induce dis- proportionately large utility degradation for existing methods (GA (Thudi et al.) 22022), GradDiff (Maini lt al., 2024)), whereas OFMU mitigates this cou- pling through its similarity- hierarchical up- dates. Full detail is provided in Appendix 77..6
Our Approach: OFMU. We introduce OFMU -Opiinization-Driven Framework for Machine Unlearning), a penalty-based bi- level optimization framework that formalizes this hicrarchy. OFMU addresses the short- comings of existing methods through three key innovations: (i) principled penalty-based based reformulation that enforces stationarity of the in- ner forgetting objective, enabling efficient two-loop optimization without requiring full convergence of the inner problem; (ii) a similarity-awal penalty that explicitly decorrelates gradients between forget and retain objectives, mitigating destructive interference during updates; and (iii) a rigorous convergence analysis of penalty- based unlearning under both convex and non-convex regimes.
Contributions. Our main contributions are summarized as follows:
We propose OFMU, a novel optimization-dre getting over utility preservation, capturing the conceptual hierarchy inherent in unlearning.
bi-level framework that explicitly prioritizes for-
We develop scalable two-loop algorithm with provable convergence guarantees, avoiding the computational bottlenecks of traditional bi-level optimization methods.
We design a similarity- penalty that dynamically decorrelates forget and retain gradients, ensuring that forgetting does not inadvertently degrade retained knowledge.
2

---

Published as a conference paper at ICLR 2026
We provide comprehensive theoretical analysis of convergence rates for penalty- based unlearn- ing in both convex and non-convex settings.
We empirically validate OFMU across benchmark unlearning tasks in language and vision models, establishing a new state-of-the-art in the forgetting-utility trade-off.
BI-LEVEL OPTIMIZATION IN MACHINE LEARNING
Bi-level optimization has emerged powerful paradigm for problems where two interdependent objectives must be optimized a hierarchical manner. Formally, it consists of an outer optimization task whose feasible solutions mization problem (Colson et al. 2007; Dempe & Zemkoho 2020). This nested structure provides natural way capture tasks where
implicitly constrained by the optimal solutions of an inner opti-
objective has strict priority over another, such hyperpa-
rameter tuning (Franceschi et al. 2018), meta-learning (Finn et al. 2017 Nichol et al. 2018). and adversarial robustness (Madry et al. (0018).
Recent advances have demonstrated the utility of bi-level optimization in large-scale machine learn- ing, particularly in domains where competing goals must be balanced without collapsing into triv- ial solutions. For instance, in meta-learning, the inner problem adapts specific tasks, while the outer problem promotes generalization across tasks (Hospedales et al.) 2021). Similarly, in adver- sarial training, the inner maximization crafts adversarial perturbations, and the outer minimization strengthens the model against them (Zhang et al. (019). These successes highlight the versatil- ity of the framework in structuring inherently asymmetric objectives and avoid trivial or unstable solutions.
We adopt this perspective for machine unlearning. In this setting, forgetting must be enforced as a non-negotiable objective, ensuring that the influence of target data is fully removed, while utility preservation is treated as secondary goal to be optimized conditionally. This stands in contrast to scalarized approaches that conflate the two objectives via fixed weights, often leading to brittle trade- offs. Bi-level optimization, by explicitly separating the two, allows respect the asymmetry of their importance and design algorithms that reflect this priority structure. This foundational insight motivates proposed framework, OFMU, which formally present in Section
METHODOLOGGY
We now present the OFMU framework, a penalty-based bi-level optimization method that explicitly separates the forgetting and utility preservation objectives. We begin by introducing notation and formalizing the problem, then describe the bi-level formulation, its penalty-based reformulation, and our scalable two-loop algorithm.
3.1 PRELIMINARIES AND NOTATION
We consider supervised learning setup with dataset D D, U Dj, where D, is the retain sel (examples to be preserved) and D, is the forget set (examples to unlearned). The model denoted by fe. parameterized by 0 E Rdd
Empirical Losses.
We define the empirical losses over each subset
(. Dr1
1
Σ ((o()).)), (, |Dyl (R,0)ED,
Σ {(o()),y),
(Z,0)ED.
where ((:,-) is standard loss function (e.g., cross-entropy).
Gradients and Similarity. We denote the gradients of the retain and forget losses as Vo and VoL,(0), respectively. To quantify their alignment, we use cosine similarity:
(VoL, VoC.) Sim (oL,, VoC.) IVoC, V
(2)
which captures directional aligmment, abstracting away differences in magnitude. In the context of unlearning. this is crucial: if V,C, (0) and (B) are highly aligned, forgetting updates may interfere with retention, motivating decorrelatio..

---

Published as a conference paper at ICLR 2026
Penalty Parameter. To ensure tractability, will use penalty parameter p 0, which enforces the stationarity condition of the inner maximization through a soft constraint. This allows us to avoid solving the inner problem to completion while still preserving its structure.
A full notation summary is provided in Appendix 7.1 which will be used throughout the paper.
3.2 PROBLEM SETUP
The goal of machine unlearning is to remove the influence of the forget set D, from trained model, while preserving performance on the retain D... Formally, seek model parameters 0 such that the model's predictions are independent of y, yet its performance on D, remains optimal.
However, these objectives are often in confiic: naively increasing the loss on Dj can significantly degrade utility on D, To address this, introduce bi-level optimization framework that explicitly separates the forgetting and utility objectives. The inner problem seeks to maximize forgetting and decorrelate the influence of D and D, via similarity penalty, while the outer problem restores utility on D.. The resulting bi-level optimization problem is:
min BER4
())
subject to
A E arg max [Cy( - 3. Sim (VoL (e",( ol ER
(3)
where define
0 controls the strength of the gradient decorrelation penalty. For ease presentation,
((0) == Ly - 3 Sim (im(VoL((0),
The formulation in equation 3 ensures that:
The inner maximization emphasizes forgetting on D, while decorrelating from D,.
The outer minimization restores utility on D,, subject to the constraint that forgetting has already been enforced.
3.3 BI-LEVEL FORMULATION
We now explicitly separate the two objectives via bi-level optimization.
Inner Maximization (Forgetting and Decorrelation.. The inner problem seeks parameters that maximize the loss on the forget set while minimizing the similarity between the gradients of the forget and retain losses. This achieved by solving:
0 arg max (0'))
(5)
Outer Minimization (Utility Restoration). Given the solution 0% from the inner problem, the outer problem seeks minimize the loss on the retain set:
0 arg min (0). 0+00
(6)
Stationarity Constraint. The bi-level structure enforces that the final model parameters 6" are stationary points of the inner maximization objective, i.e., (ot( = 0.
3.4 PENALTY- BASED SINGLE-LEVER REFORMULATION
Directly solving the bi-level optimization problem is computationally challenging, especially for large-scale models, due to repeated inner maximization and higher-order derivatives computation. To address this, adopt penalty-based single-level reformulation that transforms the bi-level problem into tractable unconstrained optimization.
Penalty Reformulation. We introduce penalty term that enforces the stationarity condition of the inner maximization as a soft constraint. The resulting objective is:
F(0) (x) + pl|V
where (()) (0) (--mmm (O), (o), and p 0 penalizes deviation from stationarity. As increases, the penalty term forces 9 to approach the stationary point of the inner objective $(0). In the limit p - oo, any minimizer of F(0) satisfies the original bi-level constraint Vo = This formulation transforms a nested optimization into a tractable single-level objective while preserving the hierarchical structure.
4

---

Published as a conference paper at ICLR 2026
3.5 PRACTICAL ALGORITHM AND IMPLEMENTATION
The penalty- based OFMU algorithm is designed for scalability and efficiency in large-scale deep learning settings. Although the penalty reformulation converts the original bi-level problem into a single-level objective, directly optimizing the full loss landscape can be unstable and computation- ally inefficient. To address this, we adopt a two-loop optimization scheme that alternates between maximizing forgetting and minimizing penalized retain loss. Here, we describe the motivation for this design, its computational advantages, and the implementation details.
Motivation for Two-Loop Optimization. While the penalty-based reformulation enables direct optimization of single objective, the landscape of F(0) L. (0) pll V 10) can be highly non- convex, especially for deep models. A naive approach may struggle to find good stationary points, especially in the presence of conflicting gradient signals from forgetting and retention. The two- loop scheme mitigates this issue by explicitly maximizing the inner objective (0), which captures both the forget loss and the gradient decorrelation penalty, before each outer update. This design has two key benefits: (i) it encourages the model to traverse regions of the parameter space that are explicitly optimized for forgetting. (ii) it improves stability and convergence by warm-starting each outer iteration from a locally optimized initialization.
Importantly. in the non-convex setting typical of deep learning, the theoretical guarantee is conver- gence to stationary point of the penalty objective F(0). This point corresponds to a local minimum, maximum, or saddle point. However, because the inner objective ((0) is maximized using gradient ascent, the algorithm is biased toward stationary points that are local maxima of the inner objective, which aligns with the unlearning goal. This type of guarantee is the strongest possible in general non-convex optimization and is standard in the literature for deep learning and LLMs.
In contrast, the original bi-level formulation requires fully solving the inner maximization to conver- gence at each outer iteration, which is computationally prohibitive for large models. Our proposed approach avoids this bottleneck while still enforeing the desired stationarity constraint. For example, in LLMs, a full bi-level update may require thousands of inner steps or even retraining, whereas our approach typically uses a small, fixed (e.g., T or 10), dramatically reducing compute cost.
Two-Loop Optimization Scheme. At each outer iteration k. the algorithm alternates between:
1. Inner Loop (Forgetting Maximization): Starting from the current parameters g(*), run T steps of gradient ascent the inner objective ((0) to increase the forget loss while decorrelating it from retain gradients:
( = 0(()) + 1in (O((())
t t=
(8)
where 7m is the inner learning rate and (*()) = ((k), The final inner iterate k( = (T))) serves as the initialization for the outer loop.
Outer Loop (Utility Preservation with Penalty): The outer step minimizes the retain loss (0) while enforcing the stationarity condition of the inner objective ((0). Formally, the update is given by
((++1) 0(h5) Nour V (oF(O!!)
in where Your is the outer learning rate, and (gF(e(()) = VoL, 2pk V2D (1)
in (
Here, Ps is the penalty parameter at iteration k, V is the gradient of the inner objective, and V is its Hessian. The second term, 2px V V results from differentiating the term pk ( with respect to 8 which requires computing a Hessian-vector product (see Appendix 7.7 for details on its efficient computation via automatic differentiation).
Penalty Schedule and Practical Considerations. A growing penalty parameter gradually strengthens the enforcement of the inner stationarity condition Vy = practice, we adopt an increasing schedule Pk++1 ps: smaller values stabilize the early iterations, while larger values amplify the term ps || V (0)),, ensuring that violations of stationarity become progressively more costly. A formal justification of this property is provided later in Lemma
Mini-Batch Stochastic Gradients. To ensure scalability, all gradients are approximated using sampled independently from Dy and D, denoted as B, and B,, each of size B. The stochastic gradient estimations VoL, By) and (e B,) reduce the per-iteration computa- tional cost from O(D)) to (B), thereby enabling training on large-scale datasets. While stochas- ticity introduces variance into the gradient estimates, our two-loop formulation remains robust due
5

---

Published as a conference paper at ICLR 2026
Algorithm Penalty-Based OFMU Bi-Level Unlearning
Require:
Initial parameters 8 penalty schedule {px ] regularization B 0, learning rates
M/in Nlonts number of outer iterations K, number of inner steps T, batch size B
Require: Datasets: Dj (forget set), D, (retain set)
1: for = 0, ,
do
2:
(Inner maximization: Forgetting)
3: Initialize (^(0) - ((k)
4:
for f 0, ... T do
5:
Sample mini-batch B, C Dj. B, C D, of size B Compute Ver ( BB) and V (
6:
7:
Compute Sim(VeC), VgC.)
8:
((6'(0) + (0 BB) 3- Sim Sim(Siiie VaL,, VgL,)
9:
g((++1) + g + 7m Ve (B)(t)
Gradient ascent
10:
end for
11:
(Outer minimization: Utility preservation with penalty)
Set o!k) + ('(2)
12:
Sample mini-batch B', C CDD of size B
14:
Compute VoC, (0!h) BB))
Compute (o and 1
(k)
15:
in
16:
g(h-1) + (())
in nou (O B,) + 2px 1/ voD(O!)))
D Gradient descent
Update penalty: Pk+1 Increase
18:
end for
19:
Output: Final parameters O(K)
the penalty term [|V]3 which regularizes the updates and stabilizes convergence. Algo- rithm formally presents the complete OFMU process, which terminates after fixed number of outer iterations or once the stationarity criterion is reached.
4 THEORETICAL ANALYSIS
We now provide theoretical analysis for the penalty-based bi-level formulation introduced in Sec- tion 3 Our analysis establishes theoretical guarantees for the OFMU algorithm, showing that: (i) the penalty reformulation enforces the stationarity condition on the forgetting objective, (ii) the in- ner maximization step converges under standard assumptions, and (iii) the full two-loop algorithm converges in both convex and non-convex settings. We further provide separate analysis of the computational complexity of OFMU Appendix 7.4 These results provide the theoretical founda- tion for OFMU and validate its design choices.
Penalty Reformulation Enforces Stationarity. We first show that the penalty-based single-level reformulation of the bi-level unlearning problem enforces stationarity of the inner objective, i.e., (a 0, as the penalty parameter increases. This result, presented in following lemma with the proof in Appendix 7.2.1 motivates the use
penalty method in OFMU.
Lemma 1 (Stationarity via Penalty Reformulation). Let E. and &b be continuously differentiable and bounded below. For any sequence {6r} of minimizers of F(0) (,) + ( with p + 00, every accumulation point 6* satisfies Vol(0") =.
Convergence of the Inner Maximization Step. Next lemma, with the proof provided in Ap- pendix 7.2.2 analyzes the convergence of the inner maximization loop, showing that gradient ascent, which is used to maximize (0) achieves sublinear convergence in the convex setting.
Lemma 2 (Convergence Inner Maximization). Let Φ(0) be convex and differentiable with I.- Lipschitz continuous gradient. Then, applying T steps of gradient ascent: g((++1) = g((1) + The (Q((()) with step size The 1/L yields the bound:
110 1
(( - ((0(T))
2T 2T
1jin
where 0% arg maxg (0))

---

Published as a conference paper at ICLR 2026
Convergence of the Full Two-Loop Algorithm. Finally, we establish convergence guarantees for the penalty- based OFMU algorithm in both convex and non-convex regimes in the following lemma with the proof provided in Appendix 73
Lemma 3 (Convergence of Penalty-Basee -aaed OFMU). Under Assumptions 7.3 the penalty-bssed OFMU algorithm converges in both convex and non-convex settings:
Convex case: If (0) and (()) are convex and I-smooth, then after K outer iterations with T inner steps per iteration, the suboptimality satisfies
F(((K)) - F =0x)
Setting - ensures c-optimality of the penalty objective.
Non-convex case: If either (0) or (0) is non-convex but L-smooth, then OFMU converges to an e-stationary point of F()), with the expected squared gradient norm bounded by
min
E VF )1² S $ +0 +0+))
where o² captures the variance of stochastic gradients.
5 EXPERIMENTS
In this section, we describe the experimental setup and evaluate our proposed method, OFMU, on both language and vision tasks to assess its effectiveness and generality. Our experiments are designed to examine three key aspects: (i) whether OFMU achieves strong unlearning efficacy while preserving model utility in LLMs; (ii) how OFMU compares against state-of-the-art unlearning baselines across different benchmarks, and (iii) whether OFMU extends effectively to non-language tasks such vision-bascd classification.
5.1 EXPERIMENTAL SETUP
We conduct all experiments on two NVIDIA H100 80GB GPUs and two NVIDIA H100 NVL. 96GB GPUs. For LLMs, consider two widely used benchmarks: (i) TOFU (Maini al. 2024
a synthetic QA dataset on fictitious authors designed to test entity-level unlearning; (i) WMDP Li
al. 2024), which evaluates unlearning in high-stakes domains such as biosecurity, cybersecurity, and chemical safety. For vision tasks, we use CIFAR-10 and CIFAR-100 (Krizhessky & Hinton, 2009) and evaluate OFMU under two settings: (i) class-wise forgetting. where all examples from one or more classes are removed and (ii) random forgetting, where a randomly selected subset spanning all classes is removed. Sections [..2 and 5.3 present results on the TOFU and C-FAR-10 benchmarks respectively, while results for WMDP and CIFAR-100, together with complete details of the experimental sctup, evaluation metrics, baselines, and models, are deferred to Appendix 7.5
5.2 TOFU RESULTS
For TOFU benchmark, we evaluate OFMU across three forgetting scenarios: forget01, forget05, and forget O, which correspond to removing 1%, 5%, and 10% of the dataset, respectively, using two model architectures: LLaMA-2 3- Instruct? We report performance using three key metrics: forget quality (FQ), model utility (MU), and torget truth ratio (FTR), where higher values indicate more effective forgetting, better utility retention, and stronger reliability of unlearned outputs, respectively.
and
Forgetting Quality. On both architectures, OFMU achieves strong FQ, comparable to or exceeding prefcrence-based methods such NPO (Bourtoule et al.. 202I). In forget 01, OFMU achieves slightly lower FQ than NPO on LLaMA-2 but matches or surpasses it on LLaMA-3.2, while main- taining stronger MU and FTR. This highlights that our framework prioritizes balance rather than over-optimizing a single metric. Unlike Gradient Ascent (GA) (Thudi et al. 2022) and Gradient Difference (GD) (Maini et al.] 2024), which aggressively maximize the forget loss collapse to near-zero utility, OFMU enforces forgetting without destabilizing updates.
Utility Preservation. MU is where many basclines diverge. Methods like RMU (Li 2024 pre- serve utility well but at the cost of incomplete forgetting, while GA achieves near-perfect forgetting
http:h//ggggggngero.oooooaaaaaar 3.2-188- 15.2-18-Instrut__fff nnsruct_full

---

Published as a conference paper at ICLR 2026
TOFU Ove all Normllzzd Rarfomance: LLaMAr22 vs LLaMA-3
ferge133
Partod
Seftre
Figure 2: Overall normalized performance of unlearning methods on LLaMA-2 and L under different forget scenarios (1 %,, 5%, 10%). The overall score is computed by normalizing FQ. MU, and FTR and then averaging them. Higher scores indicate better balance between forgetting efficacy and utility preservation. Details about the calculation are provided in Appendix 7.5.3
but eliminates MU entirely. In forget although GA attains higher raw FQ, its utility com- pletely collapses (MU 0.00), whereas OFMU sustains competitive FQ with substantially higher MU (0.65 on LLaMA-2), Similarly, in forget 10, OFMU preserves robustness across all metrics, while GA and GD sacrifice utility entirely, and NPO degrades sharply. Overall, OFMU maintains MU close to the Retain bascline while still enforcing strong forgetting.
Truth Ratio. FTR further confirms OFMU's balanced behavior. Whereas GA and GD degrade truthfulness due to unstable optimization, and NPO variants sometimes inflate FTR by overfitting to retain data. OFMU consistently maintains high FTR across all scenarios. This indicates that unlearned models continue to provide reliable responses, rather than memorized or distorted ones.
Table I: Performance of unlearning methods on TOFU using LLaMA- 2-7B-he Sa0-hf-chae chat (mean std over 5 runs).
fonget10 MLT
Method
HUT
MUT
HVT
POT
FIR!
Finctused
1.32=0
5.6800.1 8e-14
0.63=00001
4.1+6.31ee25
0.58
Reuain
1.0050.000
0.6029000
0.701=0001
10060100
0.63:0000
0.77511222
1.00000.01
0.022-1.022
0.601:0022
GndlAscmn
1.01-0.10--1
0.55+0000
0.37+0.04
866-0.7e-118
0:00-0:00
8.11+0.10-00
111200006-225
0.00+0.00
1.16+0. 1l-32
3.14-0.14-33
0.57+0.02
0.42-0008
2001-0086-118
0.50+0.05
4.20+011 Te-06
966+0.06--220
1,0++000667
HADPO
0.121..02
0.57.003
0.68.10.02
4.00L0.20e
0.04110011
0.67100.02
5.001...25::1
0.041...01
0.6410000
NPO
0.0021.04
0.56:9002
0.60:00022
0.52:5.00
0.71
0.42:00.04
0.54::1.112
0.7420002 02
SimNPO
1.00=0.10e-
0.58:9002
0.41::000
1:10:00055--00 306
0.60120.02
1.8-=0.17e-5
1.52210.tee 198
0.00210.11
3.102:1.15e-4
RMU
0.41+0.04
0.2+0.00
0.65+0002
9:61-0.26-100
0.02+0.01
0:51+0002
6.98+00221 p-21
0.08+0.01
0.2++0.22
OLMC (ours)
0.4110003
0.6310.01
0.68.0002
0.14.10:02
0.65.10.02
0.82.0.01
0.42.10.03
0.61.1002
0.77.0002
To capture
holistic view of unlearning FQ. MU, FTR into a
efficacy, we
aggregate
the three (Figure
core metrics
single normalized score
This unified view highlights the balance between forgetting and retention across different forget sce- narios and shows how OFMU strikes the balance to achieve overall better results in unlearning.
CIRAALO Performance
5.3 CIFAR-10 RESULTS
Feteerege
For CIFAR-10, we evaluate OFMU under two set- tings: class-wise forgettin,, where entire class is removed, and random forgetting, where 10% the training data is randomly selected as the forget set.
Figure 3: Overall normalized performance of unlcarning methods on CIFAR-1O. 10. The score is obtained by normalizing four key metrics-UA. RA, TA, and MIA Efficacy- within each scenario and averaging them into unified value. Details of the calculation are given in Appendix 7.5.3
The results are summarized Table 3 We report Unlearning Accuracy (UA), Retain Accuracy (RA), Total Accuracy (TA), and Membership Inference At- tack efficacy (MIA-Efficacy), where higher values indicate better unlearning performance and robust- ness.
Class-wise Forgetting. Retraining from scratch achieves perfect unlearning (100% UA) and strong overall utility (94.80% RA, 91,91.800 91.82% TA), but is com-
8

---

Published as a conference paper at ICLR 2026
putationally infeasible. Among approximate methods, Fisher Forget (FF) (Golatkar et al. 2021 and Influence Unlearning (IU) (Mehta et al. 2022) preserve utility reasonably well, with IU in particular showing the highest UA (8).31%) However, IU is computationally expensive, requiring repeated influence function estimations and parameter adjustments, which makes it impractical for L.MSs with billions of parameters. In contrast, OFMU achieves balanced trade-off: 81.51% % UA, 93.51% % RA, and 86.8% TA. While its UA is slightly lower than IU, OFMU generalizes more effectively, as shown by its higher MIA-Efficacy 76). This indicates that OFMU not only forgets the targeted class but also improves robustness against membership inference attacks, a critical security measure. Unlike IU, OFMU scales naturally to deep non-convex settings without prohibitive computational cost, making it more suitable for practical deployment in LLMs.
Table 2: Performance of unlearnin methods on TOFU using LLaMA- Ier---nntrruce (mean - std over 5 runs).
EorgettS MI71
fopeet1
Methed
FQ2T
MUT
FTRT
FQT
FTR:
Finetined
0:01--0.005
0.00-0002
0:65-0:05
1.28-0:000-13
0:60-0001
0:48+0002
1.00-0.07--21
0:60+001
0:19+0002
Retrain
0,0010.02
0.9610.02
10001.000
0.0010011
1,0010.00
0.3910002
0,6411022
0.2810004
0.35.00:2
0.35.0.03
1.8_00.08 110
0:00.0000
2.4710.12e
2.2010.11e
0.75=0005
0.41=4008
0.
0.54=000
832250376-34
L10-0055-22
0.50=9000
MADPO
0.01+0.005
0.51-0.03
0.90=0.05
1.10+0.05e-5
0:07=0002
0.02:0005
4.60-0.22--12
0.23+000
0:60++008
NPO
0.92-6.04
0.57-0.02
0.06+0.02
0.1:++0.02
0:46-0.08
0,70-6002
0.02+0.005
0-46+0/03
0,70+0009
SimNPO
0.58_00.0
0,46.00:3
0.56..0.03
5.00L.025e 100
0.58.10002
4.18.10.20e
2.45 203
0.54100.02
1.0611000e 5
RMU
0.17:00.03
0.00=0002
0.71-0001
4.91=11223-10
0.69=00002
0.78-
4.19=00110-- 142-16
0.50=00.02
0.77=0002
OFMU (ours)
0.91:0004
0.61-0.02
0.75:0002
O. 0.15::0.02
0.61:0002
0.75=00.02
0:41::0:05
0. 0.60:0002
0.77:00.02
Random Forgetting. The random forgetting task is more challenging, since the forget set is dis- persed rather than concentrated. Retraining achieves the highest UA (6.79%), while most approx- imate methods collapse, with UA close to zero (c.g., GA: 0.78%, FF: 0.51%). These methods struggle to generalize forgetting uniformly across the randomly distributed forget samples, high- lighting their sensitivity the structure of the forget set. OFMU achieves 7.71% UA, slightly out- performing retraining. While its RA (92.25%) and TA (88.61%) are marginally lower than retrain or fine-tuning, OFMU maintains better balance by reducing susceptibility membership inference (MIA-Efficacy: 3.36 versus 1.21 and 1.87 for FF and GA, respectively). This shows that OFMU enforces forgetting more uniformly, even in the scattered random setting, without collapsing utility.
Table 3: Performance of unlearning methods on CIFAR-10 10 under class-wise and random forgetting. Values show mean + standard deviation over 5 runs.
Class-wise Forgetting
Random Forgetting (10% forget set)
Method
CA f
RA 1
TA
MIA 0
UA 1
RA
TA
MIA 1
Retmin
100.00 + 0.0
94.80=022
91.82
+ 0.3
100.00 = 0.0
6.70 + 0.3
100.00 010
92.04 + 0.1
16.08 0.5
Finetuned (FT)
03-43-. 4 2.1
94.19
94.61
0.6
56.51 = 2.8
1.52 4:
98.54 0.2
92.84 2: 0.4
5.06 0.4
GradAacent (GA)
2.2
$6.52 1.7
82.41 2.0
55.03
0.78
92.10
1.87
Pisher Forger (PP)
79.71 1.4
94.12+04
93.96
0.6
46.38
2.4
0.51 + 0.2
88.00 + 18
87.70 1.9
1.21 0.44
Infiaence Unlearning (IU)
89.31 1:31+:10 1.1
92.19
90.63
: 1.0
55.22
+ 2.5
0.62
4 0.3
99.39
0 0.3
94.43
0.6
1.51 0.3
OFMU (ours)
1.3
93.51 0.6
86.88
59.76
7.71
2 0.4
92.25
1: 0.9
18.61
3.36 0.6
The CIFAR-10 results further underscore the robustness of OFMU. Whereas existing baselines overemphasize either unlearning or retention, OFMU achieves stable performance across both sce- narios, validating the advantages of its hicrarchical optimization design. As shown in Figure 3 we also report the overall normalized score for CIFAR-IO, analogous to the TOFU benchmark. In the class-wise setting, Influence Unlearning (TU) attains higher unlearning accuracy, but its heavy com- putational cost limits practicality. In the more challenging random forgetting scenario, where most baselines collapse, OFMU achieves the best overall performance. These results confirm that OFMU maintains a consistent balance across metrics, achieving effective and robust unlearning without sacrificing generalization, unlike methods that over-optimize for a single objective.
CONCLUSION AND FUTURE WORK
In this work, we introduced OFMU, a penalty-based bi-level framework for machine unlearning that explicitly prioritizes forgetting before utility preservation. By combining a scalable two-loop algo- rithm with similarity aawre penalty, OFMU achieves state-of-the-an trade-offs across language and vision benchmarks. Our theoretical analysis provides convergence guarantees in convex and non-convex regimes, and empirical results consistently demonstrate improved stability, robustness

---

Published as a conference paper at ICLR 2026
to hard-to-forget samples, and stronger resilience against membership inference attacks compared to existing approaches.
Although OFMU makes significant progress, several directions remain open. First, extending OFMU to continual unlearning scenarios, where multiple requests arrive sequentially, would en- hance its applicability. Second, investigating adaptive penalty schedules and alternative gradient similarity measures could further improve robustness. Finally, applying OFMU to even larger foun- dation models and diverse modalities such as speech and multimodal learning presents a promising avenue for future research.
REFERENCES
Ludovic Bourtoule, Varun Chandrasekaran, Christopher A. Choquette-Choo Haoran Jia, Adam Travers, Bita Zhang, David Lie, and Nicolas Papernot. Machine unlearning. In 202/ IEEE Sympo- sium on Security and Privacy (SP), pp. 141-159. IEEE, 2021. doi: 10.1 10.1109/SP40001.1 20000000..22..0000000
Zhiqi Bu, Xiaomeng Jin, Bhanukiran Vinzamuri, Anil Ramakrishna, Kai-Wei Chang, Volkan Cevher, and Mingyi Hong. Unlearming as multi-task optimization: A normalized gradient dif- ference approach with an adaptive learning rate. arXiv preprint arXiv:2410.22085 2024.
California Legislative Counsel. Bill text, califormia consumer privacy act (ab-375).
https: htt///gginff...o https://eeinfo. legislature ca. gov/ wtt.aaa.....e////////eeaaaa bill 10=2011111111 7201 72117788118883 2018. Accessed: 2025-09-16.
http/aaee//iiTeettteeeeee x
xhtm1?
Yinzhi Cao and Junfeng Yang. Towards making systems forget with machine unlearning. In 2015 IEEE Symposiun on Security and Privacy, pp. 463-480. IEEE, 2015. doi: 10. 10.9109/P..0.....5.
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramèr, and Chiyuan Zhang. Quantifying memorization across neural language models. In The Eleventh International Conference on Learning Representations (ICLR), Kigali, Rwanda, May 2023. OpenReview.net.
Jiaao Chen and Diyi Yang. Unlearn what you want to forget: Efficient unlearning for llms. arXiv preprint arXiv:2310.20155 2023.
Minseok Choi, Daniel Rim, Dohyun Lee, and Jaegul Choo. Snap:
Unlearning selective knowledge
in large language models with negative instructions. arXiv preprint arXiv:2406. 2024.
Benoît Colson, Patrice Marcotte, and Gilles Savard. An overview of bilevel optimization. Annals of Operations Research, 153(1):235-255 2007. doi: 000077/11479-000-0117666
Quang-Viie Vinh Dang. Right to be forgotten in the age of machine learning. In Advances in Digital Science: ICADS 202/, pp. 403-411, Springer, 2021.
Stephan Dempe and Alain B. Zemkoho. Bilevel Optimization: Advances and Next Challenges. Springer, 2020. doi:
Yijiang River Dong, Hongzhou Lin, Mikhail Belkin, Ramon Huerta, and Ivan Vulic. Undial: Self- distillation with adjusted logits for robust unlearning in large language models. arXiv preprint arXiv:2402. 10052, 2024.
Ronen Eldan and Mark Russinovich. Who's harry potter? approximate unlearning in llms. arXiv preprint arXiv:2310.022.88 2023.
European Union General data protection regulation (gdpr), 2016. URL htttts://errr lex.
europa. http://eeeeeelleeee//////////99
Regulation (EU) 2016/679.
Chongyu Fan, Jiancheng Liu, Licong Lin, Jinghan Jia, Ruiqi Zhang, Song Mei, and Sijia Liu. Sim- plicity prevails: Rethinking negative preference optimization for Ilm unlearning. In Proceedings of the 39th Conference on Neural Information Processing Systems (NewrIPS), 2025.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learming for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning (ICML), 2017. WRRLtttp::.. httt::/pprocedd: ngs. ml www.pppwwwwwwnnnn.ooeo 7a .htm]
10

---

Published as a conference paper at ICLR 2026
Luca Franceschi, Michele Donini, Paolo Frasconi, and Massimiliano Pontil. Bilevel programming for hyperparameter optimization and leatnnl...ning.... In Proceedings of the 35th Internationall Con- ference Machine Learning (ICML), 2018. URL http::/proceeedngss....
press/
v8C/Francess 18a html
Saeed Ghadimi and Guanghui Lan. Stochastic first- and zeroth-order methods for noncon- vex stochastic programming, SIAM Journal on Optimization, 23(4):2341-2368 2013. doi:
10.1137/120880811
Aditya Golatkar, Alessandro Achille, Avinash Ravichandran, Marco Polito, and Stefano Soatto. Mixed-privacy forgetting in deep networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 792-801 2021.
Timothy M. Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-learning in neural networks: A survey. IEEE Transactions Pattern Analysis and Machine Intelligence, 44 (9):5149-5173) 2021.
doi: 10. 1.110//PAM..222133799999999
James Y. Huang, Wenxuan Zhou, Fei Wang, Fred Morstatter, Sheng Zhang, Hoifung Poon, and Muhao Chen. Offset unlearning for large language models. Transactions on Machine Learning Research, May 2025.
Jie Huang, Hanyin Shao, and Kevin Chen-Chuan Chang. Are large pre-trained language models leaking your personal information? In Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 2038 2047, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.
Jinghan Jia, Jiancheng Liu, Parikshit Ram, Yuguang Yao, Gaowen Liu, Yang Liu, Pranay Sharma, and Sijia Liu. Model sparsity can simplify machine unlearning. Advances in Nearal Informa- tion Processing Systems (NeurIPS), 2023. URL http:://c.....1...85555513111110 Spotlight.
2304. 04934
Anastasia Koloskova, Youssef Allouah, Animesh Jha, Rachid Guerraoui, and Sanmi Koyejo. Certi- fied unlearning for neural networks. In Proceedings of the 42nd International Conference on Ma- chine Learning, volume 267 of Proceedings of Machine Learning Research, Vancouver, Canada, 2025. PMLR.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Tech- nical report, University of Toronto, 2009. Technical Report.
Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D. Li, Ann-Kathrin Dombrowski, Shashwat Goel, Long Phan, Gabriel Mukobi, Nathan Hurgerrrger, Rassin Lababidi, Lennart Justen, Andrew Liu, Michael Chen, Isabelle Barrass, Oliver Zhang. Xiaoyuan Zhu, Rishub Tamirisa, Bhrugu Bharathi, Adam Khoja, Zhenqi Zhao, Ariel Herbert- Voss, Cort B. Breuer, Samuel Marks, Oam Patel, Andy Zou, Mantas Mazeika, Zifan Wang, Palash Oswal, Weiran Lin, Adam A. Hunt, Justin fienken-Hardee Kevin Y. Shih. Kemper Talley, John Guan, Russell Kaplan, Ian Steneker, David Campbell, Brad Jokubaitis, Alex Levinson, Jean Wang, William Qian, Kallol Krishna Karmakar, Steven Basart, Stephen Fitz, Mindy Levine, Pon- nurangam Kumaraguru, Uday Tupakula, Vijay Varadharajan, Ruoyu Wang, Yan Shoshitaishvili, Jimmy Ba. Kevin M. Esvelt, Alexandr Wang, and Dan Hendrycks. The wmdp benchmark: Mea- suring and reducing malicious use with unlearning, arXiv preprint 20X3v:2403.032/1 2024.
Chris Yuhao Liu, Yaxuan Wang, Jeffrey Flanigan, and Yang Liu. Large language model unlearning via cachedddin--oorruplee prompts. arXiv preprint arXiv:2406.07933. 2024a.
Chris Yuhao Liu, Yaxuan Wang, Jeffrey Flanigan, and Yang Liu. Large language model unlearning via embeddiing-corruped prompts. arXiv preprint arXiv:2406.0700 079300 2024b.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference Learning Representations (ICLR), 2018. URL https::/ htttps://openzeview net net/forum?id
r

---

Published as a conference paper at ICLR 2026
Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary C. Lipton, and J. Zico Kolter. Tofu: A task of fictitious unlearning for llms. In Proceedings of the Conference on Language Modeling (COLM), 2024.
Ronak Mehta, Siddharth Pal, Vivek Singh, and S. N. Ravi. Deep unlearning via randomized condi- tionally independent hessians. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattem Recognition (CVPR), pp- 10412-10421, 2022.
Anmol Mekala, Vineeth Dorna, Shreya Dubey, Abhishek Lalwani, David Koleczek, Mukund Rungta, Sadid Hasan, and Elita Lobo. Alternate preference optimization for unlearning factual knowledge in large language models. arXiv preprint (tX.Xi:24499134455 2024.
Yizhong Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with reference-free reward. In Advances in Neural Information Processing Systems (NewrIPS), vol- ume 37, pp. 124198 124235, 2024.
Andrei Muresanu, Anvith Thudi, Michael R Zhang, and Nicolas Papernot. Unlearnable algorithms for in-context learning. arXiv preprint arXiv:2402.0075/ 2024.
Alex Nichol, Joshua Achiam, and John Schulman. Reptile: A scalable meta leta-lea algorithm. huttps://arxe wwt.o//g//////ooooo//.ooo/...tt/ 1803. 02999 2018. 1XIv:1803.02999
Office of the Privacy Commissioner of Canada. Announcement: Privacy commissioner seeks federal court determination on key issue for canadians' online reputation. httpps://ww..... www..priv...m .gr.v.g.
[ca/en/ cessed: 2025-09-16.
1810000//11111/11111 Oct 2018. Ac-
Martin Pawelczyk, Seth Neel, and Himabindu Lakkaraju. In-context unlearning: Language models as few shot unlearners. arXiv preprint arXiv:2310.07579, 2023.
Barak A. Pearlmutter. Fast exact multiplication by the hessian. In Neural Computation, volume 6, pp. 147-160. MIT Press, 1994.
Robin Staab, Mark Vero, Mislav Balunovic, and Martin T. Vechev. Beyond memorization: Violating privacy via inference with large language models. In The Twelfth International Conference Learning Representations (ICLR), Vienna, Austria, May 2024. OpenReview.net.
Anvith Thudi, Gabriel Deza, Varun Chandrasekaran, and Nicolas Papernot. Unrolling sgd: Un- derstanding factors influencing machine unlearning. In 2022 IEEE 7th European Symposium Security and Privacy (EuroS&P),, pp. 303-319. IEEE, 2022.
Yuanshun Yao, Xiaojun Xu, and Yang Liu. Large language model unlearning. arXiv preprint arXiv:2310. 106.83333.. 2023.
Binchi Zhang. Yushun Dong, Tianhao Wang, and Jundong Li. Towards certified unlearning for deep neural networks. In Proceedings of the 41st International Conference on Machine Learning (ICML), volume 235 of Proceedings of Machine Learning Research, Vienna, Austria, 2024a. PMLR.
Chenlong Zhang, Zhuoran Jin, Hongbang Yuan, Jiaheng Wei, Tong Zhou, Kang Liu, Jun Zhao, and Yubo Chen. Rule: Reinforcement unlearning achieves forget-retain pareto optimality. arXiv preprint 25Xiv:2506.07171, 2025. Accepted at NeurlPS 2025.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric P. Xing, Laurent El Ghaoui, and Michael I. Jordan. Theoretically principled trade-off between robustness and accuracy. In Proceedings of the 36th International Conference on Machine Learning (ICML), 2019. TtLLhtt:://ppt http:://proceednggs..
mlr. www.ppeesssvvv. 2aap///hhaaggggg 9p...19pp... html
R. Zhang, L. Lin, Y. Bai, and S. Mei. Negative preference optimization: From catastrophic collapse to effective unlearning. In Conference on Learning on Large Language Models (COLM), 2024b,
Jakub tucki, Boyi Wei, Yangsibo Huang, Peter Henderson, Florian Tramèr, and Javier Rando. An adversarial perspective on machine unlearning for ai safety. arXiv preprint (id::2499.8822.. 2024.
12

---

Published as a conference paper at ICLR 2026
7 APPENDIX
7.1 NOTATION SUMMARY
0 E R2. model parameters.
D Dy: full dataset.
D,: retain set (data to preserve).
Dy: forget set (data unlearn).
:): base loss function (c.g., cross-entropy).
(O): empirical loss on the retain set.
* L (0): empirical loss on the forget set.
((0) ( B Sg 9r): inner maximization objective.
F(0)= = C, (0) + V b (f)ll penalty-basee reformulated objective.
VoL,(0); gradient of the retain loss.
(oC,(0): gradient of the forget loss.
Sim (im(VoL,(O), (O)): cosine similarity between VoCy(0) and (O).
(e)( Hessian of the inner objective.
Hv: Hessian-vector product (Pearlmutter trick).
3 > 0: regularization parameter for similarity -wwrr decorrelation.
p 0: penalty parameter enforcing stationarity.
Min learning rate for the inner loop.
Nout: learning rate for the outer loop.
T: number of inner loop steps per outer iteration.
K: number of outer iterations.
B: mini- batch size.
((k;; parameters after inner loop at iteration
g(k); parameters after outer loop at iteration k.
FQ: Forget Quality (lower residual accuracy on forget set).
MU: Model Utility (performance on retain set).
FTR: Forget Truth Ratio (faithfulness of unlearning).
UA: Unlearning Accuracy (CIFAR evaluation).
RA: Retain Accuracy.
TA: Total Accuracy.
MIA- MIA-Efficacy: Membership Inference Attack efficacy.
UDI(z): Unlearning Difficulty Index for sample z. V (x)] gradient norm of forget loss on x.
Af(x): loss margin to the target threshold (used in UDI).
+)::): Spearman correlation coefficient.
7.2
LEMMA PROOFS
7.2.1
PROOF OF LEMMA 1
Proof. Let 0% be a minimizer of F(0) pl for a given p > 0. Assume L, and continuously differentiable and bounded below.
Suppose, for contradiction, that there exists accumulation point 6* of the sequence {0t) oo such that (6*(6") # 0. Then, for sufficiently large p, the penalty term el (8"))) would dominate F(0"), causing diverge infinity, which contradicts the assumption that F(O")) is minimized and bounded below.
Therefore, it must that V, = for any accumulation point H of the minimizers as p
13

---

Published as a conference paper at ICLR 2026
7.2.2
PROOF OF LEMMA 2
Proof. Let d(() == 6 -((()) By convexity of 4 for any A and 6"
((8") S 2 (V 0- 8).
(10)
so
((0)) - (000) s (V d
(11)
The update rule gives:
(1+1) = d Tha (
(12)
so
|aalt111 = =1 2mh (d ( + Thn {1(00))
(13)
Using the convexity bound above, we have:
(d V 2 ((f()) ((
(14)
For L-smooth convex functions, yields:
| < (0() - ((())
(15)
Substituting the two bounds from equation 14 and equation into equation
13 we have:
|1 I || 2mm (1()) (((())) + 27 (20 (((()))
(16)
This simplifies to
|1 < 12 27 (1 Lypa) ( (o(( -
(17)
Summing over t to T - yields:
(p || 21 Lmpa)
D (((()) (()())))
(18)
Since 12 0, we obtain:
T-1
||d 112
Σ (+())) ((((0)) - 27ma (1 Lm7m))
(19)
Thus, the average suboptimality is give by:
min( min( (0%) (( < [10'(0) - 8in
(20)
2 (1 L )T'
For The 11/L, 1 Lina 2 1/2, we achieve:
110(00) 0% 112
((OT.) - ((((T))
<
(21)
21han
7.3 CONVERGENCE GUARANTEES FOR SENALTY-BASED OFMU
In this section, we rigorously analyze the convergence properties of the penalty- OFMU al- gorithm for bi-level unlearning. We consider both convex and non-convex settings, reflecting the diversity of loss landscapes encountered in practice. Our analysis is grounded in the following prob- lem setup and assumptions.
Problem Setup.
We study the optimization of the penalty-based objective
F(0) (0) /
(22)
where (d) is the retain loss, (() is the inner (forgetting) objective, and p > 0 is the penalty parameter. The algorithm alternates between T steps of gradient ascent on (() (inner loop) and single gradient descent step F(0) (outer loop).
14

---

Published as a conference paper at ICLR 2026
Assumptions.
Throughout our analysis, assume:
(0) and (0) are continuously differentiable.
The gradients (0),(() and (o are L-Lipschitz continuous.
The penalty parameter is non-decreasing and bounded below by Pmin > 0.
The inner and outer step sizes satisfy Thn I 1/L and Tout 1/Lp, where Lp is the
Lipschitz constant of V (0)).
Additional assumptions specific the convex or non-convex setting will be stated the correspond-
ing subsections.
We now present detailed convergence analyses for both the convex and non-convex cases.
7.3.1 CONVERGENCE ANALYSIS: CONVEX CASE
We first analyze the convergence the venalty-based OFMU algorithm under the assumption that both the retain loss (#) and the inner objective ((f) are convex and L-smooth. Our goal is to rigorously bound the suboptimality of the penalty-based objective F(0) after K outer iterations, each involving 7 steps of gradient ascent (0)).
Additional Assumptions (Convex Case).
(0) and (0) are convex.
Algorithmic Steps. At each outer iteration A; Starting from ((k), perform steps of gradient ascent on ((0) with step size Vmm 1/L to obtain (k)
1. Inner maximization:
2. Outer minimization: Update g(++1) ((h) Mouu (F(((()), where Your S 1/L.
Step 1: Inner Maximization Error. By Lemma 2 after T steps of gradient ascent the convex, L-smooth function Q we have
10:- ((8)
((()) - -0((()) <
(23)
27 Vin
where 0 = arg maxg (0). This quantifies the inexactness of the inner maximization.
Step Outer Minimization with Inexact Inner Solution. The outer update performed using
g(*) as input. Since F(0) is convex and Lm-ooohh the standard inexact gradient descent analysis yields:
F(g(k+1)) S F(O( Youu 2
(24)
Summing over 1 = -0te K - and rearranging, we obtain
1 K-1
(VF() )1 (((())))) - F*)
(25)
K
Kfouut
km
where F* is the minimum value of F.
Step 3: Bounding the Total Suboptimality. Due to the inexactness of the inner maximization, the update direction be bounded in terms of the inner error. Specifically, the gradient error at each step is
the true minimizer of the inner problem. The error the outer update can
8th)
(O(t() in
VF(O()))
(26)
and, by smoothness, ||8(+)]| < LF
0 in
g(k) Since |10%5 - Øck)|| is controlled by the inner
maximization error, and by Lemma this error is OLITT), we have [| = O(1TT2). Thus, the cumulative emor over K steps scales OKK/T2).
15

---

Published as a conference paper at ICLR 2026
Step 4: Final Rate and Parameter Choices.
Combining the above, the suboptimality after K
iterations is bounded by
[0(0]
8
11 +
K
F(O(K)) FF
(27)
2K Youu
To achieve O(c) suboptimality, it suffices to choose == (11/c) and T O(1/c).
The penalty-based OFMU algorithm, under convexity and smoothness assumptions, converges to an e-optimal solution of the penalty objective at a sublinear rate, with explicit dependence on number of outer and inner iterations. The analysis leverages Lemma for stationarity enforcement and Lemma for the inner maximization rate.
7.3.2 CONVERGENCE ANALYSIS: NON-CONVEL CASE
We now analyze the convergence of the penalty-based OFMU algorithm in the non-convex setting, where either (0) (Ø) (or both) may be non-convex. In this regime, global optimality generally intractable, so our goal is to establish convergence to e-stationary point of the penalty objective F(0).
Assumptions (Non-Convex Case).
Cy (0) and (0) are differentiable (possibly non-convex) and L-smooth.
The stochastic gradients used in the inner loop are unbiased with bounded variance o².
The penalty objective F(0) is Lp-smooth and lower bounded.
V < G, and (()]] < H for all 6.
Step 1: Inner Loop Approximation. By standard results for stochastic gradient ascent on L-
smooth non-convex functions (see, e.g., (Ghadimi & Lan 2013), after T steps we have
Σ B [IVO
< 2( ((((0)))
T
Lon
(28)
ThnT
where 4* is the maximum value of ④. Thus, the expected squared gradient norm at the final inner iterate satisfies
(10)) S o 1 0
(29)
Step 2: Outer Loop Descent and Stationarity. The outer update is ((++1) = 0 -
-Tjout V
FO
Since is L--smooth, the standard descent lemma gives
F(((k+1)) s (()
Mou
IVFO )1
(1)
(30)
2
Summing over K iterations and rearranging, we obtain
2(F(O(0)) - F*)
K Σ
[IF
(31)
K Toou
k=0
Step 3: Explicit Dependence on Inner Loop Error. The gradient of the penalty objective is
FF V VL.(0) 20 200
(32)
Using the inequality ||a+ bll² < 2| 2 + H, we have
bb|² and assuming VC, S G, and [2)(0)] S
E
|V ||
2G + H2² E [|!" {1(]
(33)
Plugging in the bound from the inner loop,
2( - -((((0))
E
VF ( |?]
< 2G2, + 8p²H² H²
+ Tha Io²
(34)
Thm T
16

---

Published as a conference paper at ICLR 2026
Step 4: Final Convergence Guarantee. the inner approximation error, we obtain
Combining the outer-loop descent bound for K steps and
min VF 2(F(010)) -
+ 2G + 8p² 22 -2(0- 2(0(01)) TmnT + Lo
(35)
Kou
This result shows that the penalty-based OFMU algorithm converges to an c-stationary point of the penalty objective. The convergence rate exhibits explicit dependence on the number outer iterations K, inner-loop accuracy T, penalty parameter p. curvature bound H, and gradient noise variance a². Choosing K = (1/ee) and T ((1/c) ensures convergence to an e-stationary point in expectation.
7.4 COMPUTATIONAL COMPLEXITY AND PRACTICAL EFFICIENCY
This section provides a formal yet practical analysis of the computational cost of OFMU and ex- plains why the method remains scalable despite introducing bi-level structure. In particular, we highlight two design choices-(i) the small number of inner steps T, and (ii) the use of Hes- sian-vector products (HVPs)-that keep our overall cost close to standard gradient based single- loop optimization while still enforcing the hierarchical structure introduced in Section
7.4.1 PER-ITERATION COST THE INNER AND OUTER LOOPS
Let d denote the number of trainable parameters, B the minibatch size, and (K, the number of outer and inner iterations, respectively. A standard borward-backward computation on a minibatch incurs
C.. = O(Bd)
floating-point operations. This the dominant unit of cost in both the inner and outer loops.
Inner loop cost. Each inner step requires one gradient of ((),, which itself evaluates VLy, V.,, and their cosine similarity. All of these operations share the same forward-backward structure, so each inner step costs O ((B)), yielding
Cinner (dB)).
Outer loop cost. The outer step requires: (i) gradient of (0) and (ii one HVP term (2 (0) arising from the penalty objective (Section 3p. The HVP computed via a Pearl- mutter's 's method (Appendix 7.7 and therefore costs the same order gradient with detail given in next section:
CнvP ((Bd).
Thus the outer step costs
Couter ((Bd).
7.4.2 WHY HESSIAN- VECTOR PRODUCTS ARE COMPUTATIIONALLY CHEAP
A naive Hessian computation requires (dd) memory and time, which is infeasible for modern LLMs. However, OFMU uses the Hessian only through the vector product
<20(0) V,
which can be implemented using Pearlmutter's method (Appendix 7.7). This trick computes Hu using one additional backward pass without forming the matrix. Consequently.
Chvr R Cgrad
up to small constant factor. Therefore, the HVP introduces negligible overhead compared to standard fine-tuning, and its inclusion does not alter the asymptotic scaling of the algorithm.
7.4.3
WHYY THE INNER LOOP IS COMPUTATIONALLY LIGIIT
Unlike classical li-el optimization, which often requires the inner problem converge nearly to optimality at each outer iteration, OFMU relies on the cenalty-based reformulation from Section This has two practical effects: 1. The inner loop does not need to converge; it only needs to make progress toward reducing the violation of the stationarity condition V 0. 2. A small, fixed inner budget T (e.g., T = 5 or 10) is sufficient, because the penalty term in the outer loop completes the enforcement of objectvve stationarity. Thus, T remains small across all experiments (see Appendix 7.3.9p, ensuring that OFMU's additional cost remains a lightweight correction rather than full inner optimization.
17

---

Published as a conference paper at ICLR 2026
7.4.4 TOTAL COMPLEXITY Combining both loops, the total cost over K outer iterations is
COFMU K (Cinner + Couter)
+ (KTBd),
where typically 7 < K and 7' is a small constant. In practice, 7 contributes negligibly to runtime, and the cost of OFMU close that of standard gradient based methods. Table summarizes the per-iteration computational complexity of different baselines compared to OFMU.
Table 4: Per-iteration computational complexity of OFMU compared to standard unlearning base- lines. Here K' denotes the number of steps, the batch size, d the parameter count, and '' the number of inner penalty updates.
Method
Complexity
Gradient Ascent (GA)
@(KBd)
GradDiff
e Bd)
NPO / SimNPO
OKKBd)
RMU
(d
OFMU (ours)
(KTBd)
7.5 AUXILIARY RESULTS AND ABLATION STUDY
Here we present additional results and ablation studies to support the main findings. These include experiments on the WMDP and CIFAR-100 100 benchmarks, robustness to hard target samples, eval- uation sensitivity, and deeper analysis of OFMU's design. Together, these analyses validate the generality and stability of OFMU across domains, architectures, and unlearning scenarios.
7.5.1 WMDP RESULTS
Table reports QA accuracy on the WMDP benchmark across three domains: Biosecurity, Cybersecurity, and the general-purpon MMLU subset. For MMLU, higher accuracy reflects better utility preservation. For the WMDP domains, we report Unlearning Efficacy as Accuracy, where higher values indicate stronger removal of hazardous knowledge.
Overall Performance. OFMU consistently outperforms all baselines across the WMDP domains and MMLU subset, achieving 72.8% unlearning efficacy on Biosecurity, 70.4% on Cybersecurity, and 74.6% utility accuracy on MMLU, While the performance gains over the strongest baseline (RMU) are moderate (+1.5 on Biosecurity, +2.1 on Cybersecurity, +0.4 on MMLU), these improve- ments are consistent and statistically significant (p < 0.05). This stability highlights OFMU's effectiveness avoiding the degradations exhibited by scalarized or overly aggressive forgetting methods.
Comparison with Preference-Basee Methods. NPO (Bourtoule et al. 2021) and SimNPO (Meng jet al. 2024) achieve competitive forgetting quality other benchmarks but often underperform on WMDP. Their unlearning efficacy lags behind OFMU by 4.0 and 3.9 points on Biosecurity, and by 5.5 and 4.2 points on Cybersecurity, respectively. This demonstrates that while preference- based optimization can guide forgetting effectiivlly, it often harms utility in specialized domains. By contrast, OFMU's hierarchical formulation ensures that utility actively restored after forgetting, enabling it to retain strong reasoning capabilities safety-critical domains.
Comparison with RMU. RMU performs better than NPO variants, particularly in Biosecurity (71.3%). However, RMU still falls short of OFMU, highlighting that methods which prioritize util- ity preservation tend to leave residual traces of the forget set, leading to incomplete erasure. OFMU achieves higher unlcarning efficacy, showing that stability and efficacy are not mutually exclusive when optimization is structured hierarchically.
On Biosecurity and Cybersecurity, where specialized reasoning is crucial, OFMU achieves the largest margins, suggesting that its similarity- penalty is particularly effective preserving domain-speciin specific knowledge while enforcing unlearning. On MMLU, OFMU's gains smaller but consistent, indicating that our framework maintains general reasoning skills rather than overfit- ting narrow benchmarks.
18

---

Published as a conference paper at ICLR 2026
Table 5: Performance of unlearning methods the WMDP benchmark. For Biosecurity and Cy- bersecurity, higher values indicate better unlearning efficacy (1 - Accuracy), For MMLU, higher accuracy indicates stronger utility preservation. Values show mean + standard deviation over runs.
Method
Bio Unlearning f Cyber Unlearning t
MMLU Utility
Retrain
78.6 + 0.3
76.5 + 0.4
82.4 + 0.3
RMU
71.3 + 0.4
68.3 + 0.5
74.2 + 0.3
NPO
68.8 + 0.5
64.9 + 0.6
72.8 + 0.4
SimNPO
68.9 + 0.4
66.2 + 0.5
73.3 + 0.4
OFMU (ours)
72.8 0.3
70.4 + 0.4
74.6 0.3
7.5.2 CIFAR-100 RESULTS
We further evaluate unlearning methods on CIFAR-100, which is considerably more challenging than CIFAR-10 due to its fine-grained class structure and higher inter-class similarity. As with CIFAR-10, 10, we report results under two settings: class-wise forgetting (removing all samples of one class) and random forgetting (removing 10% of training samples at random). The results are summa- rized in Table Class-wise Forgetting. Retraining once again achieves the best possible unlearning (100% UA) while preserving high RA and TA. Among approximate methods, Influence Unlearn- ing (IU) demonstrates competitive UA but incurs significant computational cost. Fisher Forget (FF) and fine-tuning (FT) preserve utility but fail to fully erase the target class. Gradient Ascent (GA) collapses overall utility, reflecting instability. By contrast, OFMU achieves 74.1% UA, 71.9% RA, and 69.3% TA, balancing forgetting efficacy with stable retain performance. Importantly, OFMU also outperforms all baselines in MIA-Efficacy (48.2), highlighting its ability to resist membership inference even in high-class-count settings. Random Forgetting. Random forgetting particularly challenging in CIFAR-100 because the forget set is highly dispersed across fine-grained classes. Most approximate methods collapse to near-zero UA, while retraining provides an upper bound (5.3% UA). OFMU achieves 6. 6.7% UA, slightly exceeding retraining, while maintaining compet- itive RA (70.%% and TA (67.8%). Despite the inherently low UA values, OFMU yields stronger robustness to membership inference (2.8 MIA vs. < 2.0 for most baselines), showing that its updates generalize better across scattered samples.
Table 6: Performance of unlearning methods on CIFAR-100 under class-wise and random for- getting. UA: Unlearning Accuracy, RA: Retain Accuracy, TA: Test Accuracy, MIA: Membership Inference Attack Efficacy, Random forgetting uses a 10% forget set.
Class-wise Forgetting
Random Forgetting (10% forget set)
Method
UA f
RA f
TA
MIA
UA
RA 1
TA f
MIA t
Retrain
100.00
72.4
70.5
100.00
5.3
72.8
71.1
14.2
Finetuned (FT)
32.6
71.9
72.2
41.7
1.2
72.1
71.5
3.9
GradAscent (GA)
28.4
65.8
64.1
39.2
0.6
71.7
70.2
1.1
Fisher Forget (FF)
65.7
71.2
70.9
37.5
0.4
62.1
61.8
0.9
Influence Unlearming (IU)
70.3
69.8
68.5
44.9
0.7
71.5
71.0
1.6
OFMU (ours)
74.1
71.9
69.3
48.2
6.7
70.2
67.8
2.8
Overall, CIFAR-100 highlights the robustness of OFMU in fine-grained and challenging set- tings. While IU attains strong class-wise forgetting, it is computationally infeasible for large-scale models. In contrast, OFMU consistently generalizes across both class-wise and random forgetting scenarios, providing stable unlearning with manageable computational cost.
7.5.3 OVERALL PERFORMANCE SCORE CALCULATION
To enable fair and unified comparison across different benchmarks, we define an Overall Perfor- mance Score that aggregates multiple evaluation metrics into a single normalized value. Let M denote the set of evaluation metrics used for given benchmark. For metric m E M, let mi(i) denote it value for i-th method. Then for method i:
mi)
Mmorm (i)
(36)
max; (m
19

---

Published as a conference paper at ICLR 2026
The overall score for method i is then computed the simple average of all normalized metrics:
1 L
Overall(i)
mnorm (t).
(37)
MEM
This formulation is flexible and adapts to different evaluation protocols:
For TOFU, M (U, where Forget Quality (FQ), Model Utility (MU), and Forget- to-Retain Trade-off (FTR) capture forgetting efficacy, retention stability, and balance between them.
For CIFAR-10/100, M {UA, RA, TA, MIA), where Unlearning Accuracy (UA), Retain Aceu- racy (RA), Test Accuracy (TA), and Membership Inference Attack efficacy (MIA) jointly capture forgetting quality, retention, generalization, and privacy robustness.
This aggregated score provides balanced evaluation that prevents misleading conclusions from focusing on single metric. As demonstrated in Figures B and [ the Overall Performance Score highlights the robustness of OFMU across both language and vision domains, effectively capturing trade-offs between forgetting efficacy, utility preservation, and security.
7.5.4 HARD IN-SCOPE EVALUATION AND ROBUSTNESS
Unlearning is inherently scope- depeedeeend:nt: the target to forget can be expressed through para- phrases, multi-hop reasoning, or cross-lingual variants that are still in scope but harder suppress.
Embedding Similarity with Retrain Model (Easy vs Hard Samplesi
We evaluate robustness on TOFU (forget under three in-scope transformations that do not add new knowledge but rephrase the same tar- get: (i) Paraphrase-semante rewrites; (ii)
Grddurt undioceen
0.85
0.62
0.00
0.80
0.70
that require 2-3 composi- tional steps surface the same fact; (iii) Cross- lingual-prooe prompt in second language and ask for an English answer. These probes align with worst-case/adveesariaa assessments advocated in prior work.
0.80
APO
0.89
6:73
0.75
0:91
0.75
0.70
OPPR
0.65
We report Forget Quality (FQ), Model Utility (MU), and Forget Truth Ratio (FTR). Results are averaged over 3 seeds (tt).
Easy Samples
Hard Samples
GA attains some forgetting but collapses util- ity in hard settings. RMU preserves utility and truthfulness, but under-forgets (lower FQ). NPO/SimNPO are more balanced but degrade on multi-hop and cross-lingual probes. OFMU is not an outlier; it sits between RMU (utility-leaning) and NPO/SimNPO (forgetting-leaning deliver- ing the most consistent balance (best second- best FQ while keeping MU/FTR competitive). This mirrors the main-table story: OFMU avoids both extremes (incomplete forgetting vs. catas- trophic utility loss).
Figure 4: Embedding similarity with the retrain model for easy vs. hard samples. Easy samples correspond to instances where the base model had low initial confidence, while hard sam- ples are high-confidence, entangled instances. Scores are computed as cosine similarity of embeddings with retrained reference model. OFMU maintains competitive similarity on easy samples and significantly stronger robustness on hard samples, where existing baselines collapse.
Table 7: Hard ic-seope robustness on TOFU (forget 05, LLaMA-2-7B-hh--chall Higher is better for FO/MU/FTR.
Paraphrase MU
Multi-hop MU
Cross-lingual MU
Method
FQ
FTR
FQ
FTR
FQ
FTR
GradAscent
0.21110.02
0.12.10.33
0.2810.04
0.18.10.03
0.08.10.03
0.22.10.05
0.17.10.03
0.07.10.02
0.20.10.44
GradDiif
0.35:10.03
0.4910002
0.4620.03
0.30:20.3
0.45:00.3
0.4210.04
0.28:2003
0.43:10.33
0.4020.04
NPO
0.41:20.03
0.50+0.02
0.66=0.03
0.34=00.33
0.47+0.02
0.6120.03
0.33=0002
0.4620.03
0.6020.03
SimNPO
0.39-+0.03
0.534-0.22
0.58-0.03
0.36+00.3
0.51+40.22
0.56+0.03
0.35--0.33
0.50+0002
0.55+0.03
RMU
0.28:20.02
0.58+0.02
0.73=0.02
0.24:0.02
0.57+0.02
0.75:0.02
0.22=0.02
0,5620.02
0.75:0.02
OFMU (ours)
0.44++0.03
0.56+-0.02
0.74-0.03
0.38++0.03
0.54+00.2
0.72++0.3
0.37=00.2
0.55+00.2
0.73+0.03
20

---

Published as a conference paper at ICLR 2026
7.5.5 CIMPLE-SELECTION SENSITIVITY
Randomly choosing forget samples can mask algorithmic weaknesses. We repeat TOFU forget experiments over five random forget sets (single-seed per set) and report the dispersion (mea of FQ/MU.
Table 8: Variance across random forget-set draws (TOFU LLMMA-2---
Method
FQ (te)
MU (meant:std)
GradAscent
0.26 + 0.09
0.18 + 0.12
GradDiff
0.33 + 0.06
0.47 0.05
NPO
0.38 + 0.05
0.49 + 0.04
SimNPO
0.39 + 0.04
0.52 + 0.03
RMU
0.25 + 0.04
0.58 + 0.02
OFMU (ours)
0.41 + 0.03
0.55 + 0.03
Rankings can flip depending the draw (notably between NPO and GDiff), confirming prior observations about selection bias. OFMU shows the lowest FQ variance and low MU vari- ance consistent with its stability claim.
7.5.6 MEASURING UNLEARNING DIFFICULTY
We define simple Unlearning Difficulty Index (UDI) for a forget sample x:
UDI(z) = a ( + X(1 - (1m-sim(Ve simC Vol.)) + Al(x),
(38)
where (i) the gradient norm captures the magnitude of the update pressure, (ii) the similarity term captures forget-retain gradient conflict, and (iii) A&(x) is the loss margin to the target threshold used for termination (higher margin harder). We set X y 1 for simplicity.
We compute the Spearman correlation (r) between UDI and the induced utillity drop (MU degrada- tion on retain tasks) across samples.
Table 9: Correlation of UDI with utility drop (higher т = stronger coupling hetween difficulty and collateral damage).
Method
UDDI MU drop)
Comment
GradAscent
0.71
Strong coupling: hard samples cause large damage
GradDiff
0,58
Coupling reduced but persists
NPO
0.45
Moderate; preference shaping helps Slightly better than NPO Utility-first dampens coupling but under-foree forgets
SimNPO
0.41
RMU
0.36
OFMU (ours)
0.29
Lowest coupling; balanced updates on hard cases
For GA/GDiff, hard samples (high UDI) strongly predict collateral utility loss. OFMU shows the weakest coupling, indicating that its similarity-awar penalty and hierarchical updates regulate gra- dient conflict and prevent overcorrection on hard examples. This aligns with the WMDP and TOFU trends.
7.5.7 EMBEDDING ALIGNMENT WITH RETRAIN
We compare cosine similarity between embeddings produced by unlearned models and a retrained model (gold standard) for easy vs. hard forget samples. All methods look reasonable
easy samples. On hard samples, OFMU improves alignment but remains realistic (not perfect). RMU preserves utility but misaligns with retrain on easy samples due forgettogg These findings are consistent with Tables ??-5 OFMU is stable and balanced rather than an outlier.
7.5.8 COMPONENT- WISE ABLATION OF OFMU
To better understand the contribution of each component in OFMU, we conduct an ablation study on TOFU 05 with ILAaM---7B--h--chat. We remove either the penalty reformulation or
21

---

Published as a conference paper at ICLR 2026
Table 10: Cosine similarity (higher is better) retrain embeddings TOFU (forget05).
Method
Easy Samples
Hard Samples
GradAscent
0.95
0.60
Grad
0.93
0.65
NPO
0.91
0,68
SimNPO
0.92
0.70
RMU
0.88
0.72
OFMU (ours)
0.93
0.76
Table 11: OFMU component ablation (TOFU forget05). Higher is better.
Variant
FQ
MU f Hard-sample Emb. Sim. 1
Penalty only (no similarity-aware)
0,36
0,53
0.71
Two-loop only (no penalty)
0.33
0,54
0.69
Full OFMU
0.38
0.54
0.73
the similarity-aware gradient decorrelation and compare against the full model. Results are shown in Table
The results reveal that both the penalty reformulation and the similarity-aware gradient decorrelation are critical. Removing similarity-are aware decorrelation reduces robustness on hard samples (embed- ding similarity drops from 0.73 to 0.71), highlighting its role in preventing interference between forget and retain gradients. Conversely, removing the penalty reformulation lowers forgetting ficacy (FQ falls from 0.38 to 0.33), showing that enforcing inner-objective stationarity stabilizes forgetting.
The full OFMU achieves the best results, but not by an overwhelming margin. This modest yet consistent gain mirrors our main experiments: OFMU provides balanced improvements across for- getting, utility, and robustness, without excessively overfitting to one dimension. Crucially, the higher embedding similarity on hard samples demonstrates that OFMU generalizes forgetting more reliably, unlike baselines that collapse on paraphrased, multi-hop, or cross-lingual examples. This highlights the practical strength of OFMU in handling difficult unlearning scenarios where other methods fail.
7.5.9 EFFECT OF INNER STEPS AND PENALTY PARAMETER
To further analyze the stability of OFMU, investigate how the number of inner steps and the penalty parameter jointly affect unlearning performance. Figure 3 shows three diagnostic views across CIFAR-10: Unlearning Accuracy (UA), Model Utility (MU), and Forget Quality (FQ).
Unteaming Acouracy
Utility (MH))
Forget Quality (FC)
3.4
0.53
0.43
337
17her Stipe
Inter Steps
I000r Stope
Figure 5: Effect of inner-loop steps and penalty parameter p different evaluation metrics. Each subplot shows performance trends as the number of inner steps increases for {0.1, 0.5, 1.0). Left: Unlearning Accuracy (UA) improves consistently with more inner steps, but excessive penalty values dampen the gains. Center: Model Utility (MU) remains relatively stable, with small im- provements for moderate p. Right: Forget Quality (FQ) increases with inner steps but saturates under large penalties, indicating trade-off between aggressive forgetting and preservation of util- ity.
22

---

Published as a conference paper at ICLR 2026
UA trends. As shown in the left subplot, UA increases monotonically with inner steps for all p. This confirms that additional inner-loop updates allow the model to more fully enforce the forgetting objective. However, when is large (p 1.0). the improvement is noticeably dampened, illustrating that strong penalties constrain forgetting effectiveness.
MU stability. The middle subplot highlights that MU remains relatively flat across inner steps, with only slight upward gains at moderate p. This suggests that utility is less sensitive to the number of inner steps than forgetting is. Importantly, MU stability under small and moderate p demonstrates that OFMU's penalty mechanism prevents overfitting to retain data while still allowing controlled forgetting.
FQ dynamics. The right subplot shows that FQ also benefits from increasing inner steps, but unlike UA, the improvements plateau quickly, especially under higher p. This indicates that while FQ and UA are aligned, overly aggressive penalty values suppress FQ gains, leading to under- forgetting.
Together, these plots highlight three insights: (i) more inner steps consistently strengthen forgetting efficacy, (ii) MU is robust to changes inner loop depth, and (iii) high penalty values suppress forgetting improvements. These findings provide practical guidance for setting OFMU hyperparam- eters: small to moderate p and sufficient inner steps yield the best balance between forgetting and utility.
10 EVALUATION METRICS
We evaluate unlearning performance using a range of metrics tailored to different benchmarks.
TOFU
Forget Quality: Measures how effectively the model suppresses undesired knowledge the forget set. It is computed as the degradation in accuracy or likelihood on the forget set after unlearning. Higher values indicate stronger forgetting.
Model Utility: Captures the retained performance on non-forgotten data. is typically measured accuracy or perplexity on the retain set or general benchmark dataset. Higher values indicate better utility preservation.
Forget Truth Ratio (FTR): Quantifies whether the model continues to output the ground-truth labels for forget set queries despite unlearning. A lower FTR indicates more successful forgetting. since the model is less likely to recall the original truth.
WMDP
QA Accuracy (Bio, Cyber, MMLU): Measures task performance on domain-speciff bench- marks (biological risks, cybersecurity, and general knowledge). These serve proxies for model utility in downstream applications, and higher accuracy indicates better utility preservation.
CIFAR-10 and CIFAR 100 For vision experiments, we adopt four evaluation metrics following prior unlcarning literature.
Unlearning Accuracy (UA). UA measures the accuracy of the unlearned model 0, on the forget set D,, Formally,
UA = Dyl
1
Σ
1arg max fe,, (z) = =y}
(39)
(x.))ED
Lower UA indicates better unlearning, as means the model fails correctly classify the forgotten data. In practice, UA is reported as (1 forget accuracy).
Retain Accuracy (RA). RA is the accuracy of 6, the retain D, (training samples not in
Dy). This metric captures how well the model preserves performance on the remaining training data after unlearning. Higher RA indicates better utility preservation.
Test Accuracy (TA). TA the accuracy 6. on the held-out test set of the original task. Unlike
RA, which is training-set specific, TA reflects the model's 's generalization ability after unlearning. Higher TA means better task utility retention.
MIA-Efficacy. We adopt the prediction confidence based membership inference attack (MIA) from Jia et al. (2023), which consists of training and testing phase. An MIA predictor is trained
23

---

Published as a conference paper at ICLR 2026
balanced dataset sampled from D, and the held-out test set (disjoint from Dy). In the testing phase, the predictor is applied to 0g on Dy. MIA-Efficacy then defined as
TN
MIA-Efficacy |Dyl
(40)
where TN is the number of forgetting samples predicted non-training examples. Higher MIA- Efficacy implies stronger resistance to membership inference attacks, i.e., better unlearning.
7.5.11 BASELINES
We compare OFMU against set of strong unlearning baselines covering multiple methodological families:
Retrriiing-basee Retrain (gold standard) and Finetuned baselines. Gradient-brr baased:: Gradient Ascent (GA)and Gradient Difference (GradDiff). Preference-baueee based:: NPO (Zhang al.,
2024b), SimNPO (Meng et al. 2024), and IdkDPO. Regularization -based Representation Misdi- rection for Unlearning (RMU) (Li et al. 2024). Vision-specific: Fisher Forget (FF) (Golatkar al.,
2021 and Influence Unlearning (IU) (Mehta et al. 2022)
7.5.12 MODELS AND EXPERIMENTAL SETUP
For TOFU, we LLaMA-3.2-10 evaluate two model architectures: LLaMA-2-7B-bff-chaa and While WMDP experiments are carried out on Zephyr-78-bet.d) For CIFAR-10, we adopt ResNet-style backbone, consistent with prior vision unlearning studies. All experiments are conducted using the AdamW optimizer with batch size 32, learning rate 10-5, and a maximum
training epochs.
For OFMU, the penalty parameter follows monotonic schedule Pk++1 = 7Pp with y E 2.05. The initial value Po 0.3 is selected via lightweight grid search (0.1, 0.3, 0.5, 1.0) on small data subset, after which the same Po and schedule are reused across all models and forget ratios without further tuning, and we use T = 5 inner steps per outer iteration unless otherwise specified.
Together, these benchmarks, models, and metrics provide comprehensive testbed for assessing OFMU under diverse conditions, ranging from copyright-sensee sensitive L.LM use cases safety-critical QA and vision classification.
7.5.13 EMPIRICAL RUNTIME AND MEMORY ANALYSIS
We complement the theoretical analysis in Section 7.4 with empirical measurements of both per-step runtime and peak GPU memory usage. Runtime values are normalized to the cost of single Gra- dient Ascent (GA) update (1.0x), while memory values are normalized to the footprint of a single forward-only pass through the same model (1.0x). This allows architecture-agnostie comparison across methods.
Runtime, OFMU introduces an inner maximization loop of length and one Hessian-vector product (HVP) per outer iteration. As expected from our @(KTBd) complexity, runtime grows linearly with T.
Table 12: Normalized per-step runtime (GA = 1.0x).
Method
TOFU (7B)
WMDP (7B)
CIFAR-10
CIFAR-100
GA
1.00x
1.00x
1.00x
1.00x
GradDiff
1.12x
1.10x
1.08x
1.09x
NPO
1.65x
1.70x
1,45x
1.50x
SimNPO
1.78x
1.82x
1.55x
1.60x
RMU
1.35x
1.32x
1.25x
1.28x
OFMU (T= 5)
2.85x
2.90x
2.40x
2.52x
OFMU (T 10)
4.22x
4.30X
3.75x
3.92x
24

---

Published as a conference paper at ICLR 2026
Memory. Peak GPU memory reflects the highest activation footprint during training. First-order baselines (GA, GradDiff, NPO/SimNPO) require storing activations for all layers. RMU, by con- trast, backpropagates only through three layers but maintains frozen representation buffer. OFMU incurs additional activation buffers from the inner loop and HVP, but HVPs remain memory-linaar due to Pearlmutter's method.
Table 13: Peak GPU memory usage normalized to single forward-only pass (1.0x).
Method
LLaMA-7B
ResNet-18
Dominant Memory Components
GA
1.3x
1.2x
Full activations gradients
GradDiff
1.5x
1.3x
Two losses, sequential backprop Preference pairs, logits, full activations
NPO / SimNPO
1.7x
1.5x
RMU
2.3x
2.0x
Frozen representations + gradients for 3 layers Inner/outer activations, penalty HVP
OFMU (ours)
3.1x
2.6x
Overall, OFMU incurs a moderate overhead in runtime and memory due to its bilevel structure, yet it remains practical in practice. The empirical scaling aligns closely with the theoretical ((KTBd) complexity, confirming that OFMU is computationally feasible while providing better forgetting-utility guarantees.
7.6 RELATED WORK
Machine unlearning (MU) was first introduced by Cao and Yang (Cao & Yang 2015) as a framework for removing the influence of specific training instances from trained model. Early approaches of machine unlearning focused on exact unlearning, which requires retraining the model from scratch after excluding the forget set (Bourtoule al. (02T). While these methods provide strong correct- ness guarantees, retraining is computationally infeasible for large-scale models. To overcome this limitation, approximate unlearning techniques were developed, which directly update model param- eters to diminish the effect of the forget set (Eldan & Russinovich] 2023 Fan et al., 2025). Some of these methods prioritize computational efficiency, while others provide statistical guarantees, ensur- ing that the unlearned model is indistinguishable from model retrained from scratch (Zhang al.) 2024a; Koloskova et al., 2025). However, with the increasing adoption of LLMs, unlearning has become not only matter of efficiency but also crucial tool for ensuring privacy. copyright com- pliance, and mitigating harmful behaviors (Lucki et al.) 2024, Carlini et al. 2023). Consequently, researchers have begun developing methods tailored specifically to LLMs, which we categorize into three broad types: input-based, data-based, and model-based approaches.
Input-based methods. Input- based approaches attempt to prevent the model from revealing for- gotten content by modifying queries controlling the generation process. Examples include in- context unlearning, which prepends prompts that steer the model away from sensitive topics (Pawel- czyk at.. 2023), the use of trigger phrases and guardrail classifiers enforce refusals (Muresanu et al. 22224] Liu et al.) 2024a). These techniques are attractive because they require no parameter updates and can be deployed instantly. However, they remain brittle: adversarial queries, paraphras- ing, or prompt injection can bypass the refusal mechanisms, exposing residual memorization (Lucki et al. 2024). Moreover, since the underlying parameters remain unchanged, the model still internally encodes the sensitive knowledge, limiting true unlearning.
Data-based methods. Data-based approaches fine-tune LLMs on curated auxiliary data designed to overwrite or suppress forgotten knowledge. Common approaches include training on refusal-style responses (Eldan & Russinovich] 2023) or replacing facts with negated or counterfactual alterna- tives. (Mekala et (0244 propose Alternate Preference Optimization, which combines negative feedback on forget examples with positive in-domain alternatives, yielding more coherent behavior than refusal-only tuning. These methods require careful data construction for each unlcarning task and risk semantic drift or factual incoherence. Additionally, performance on unrelated domains may degrade when auxiliary data overlaps with retained knowledge.
Model-based methods. Model-based approaches directly modify parameters to remove the in- fluence of the forget set. Early methods apply gradient ascent the forget data (Thudi et al. 2022). but these suffer from instability (e.g., gradient explosion) and catastrophic forgetting of re- tained capabilities. More sophisticated variants introduce regularized objectives, such KL-based
25

---

Published as a conference paper at ICLR 2026
penalties Yao al., 2023), gradient difference (Maini et al. 2024), negative preference opti- mization (NPO) (Zhang et al. 2024b| 2025), which treat forget examples as negative preferences in reinforcement learning- style update. Other approaches include adapter-based unlearning (Chen & Yang 2023), which localizes updates to small modules, and neuron-level interventions (Huang al. 2025), which ablate perturb hidden units strongly associated with the forget set. While these methods are more effective at actually suppressing memorized knowledge, they face a fun- damental optimization challenge: balancing the conflicting objectives of forgetting and retention. Most adopt scalarized formulation with fixed trade-off weights, which often leads to unstable dynamics-aa either catastrophic loss of utility or incomplete forgetting -artoe-aaarllll in the high- dimensional, non-convex setting of LLMs.
Our work advances this line of model-based methods by directly addressing the persistent trade-off between forgetting and retention. We propose OFMU, an optimization-drivee framework that intro- duces principled penalty-based reformulation together with similarity gradient decorrela- tion mechanism. Unlike prior scalarization- based or heuristic bi-level approaches, OFMU explicitly prioritizes forgetting in the inner problem while dynamically restoring utility in the outer loop, as formally presented in Section 3
7.7 HESSIAN-VECE PRODUCT VIA AUTOMATIC DIFFERENTIATION
The penalty term in our formulation requires computing the Hessian-vector product
V in ())()())
(41)
where (o Fd is the gradient of the inner objective and ()0(0() matrix.
in E Rddd is its Hessian
A naive approach would explicitly construct the Hessian and then perform matrix-vector multi- plication, which incurs O(d)) time and memory complexity. This is computationally prohibitive in large-scale machine learning settings, where the parameter dimension d large.
Fortunately, the Hessian-vector product (often abbreviated as Hy-product) can be computed effi- ciently without explicitly forming the Hessian. This is achieved by exploiting the directional- derivative interpretation of second-order differentials. Specifically, for any vector R the product
V30(0)
(42)
can be interpreted as the directional derivative of the gradient Vg in the direction u. This observation underlies the Pearlmutter trick Pearlmutter (1994), which computes Hu the cost of a single gradient evaluation. The penalty parameter Pk is gradually increased during training to enforce the stationarity constraint more strictly as optimization progresses. In practice, pa can be updated according to a predefined schedule or adaptively based on the norm of (
26


Reviews:
[{"review_id": "MmU4xLUvUN", "reviewer": "Reviewer_vkLf", "summary": "This work proposes a bi-level method to tackle the unlearning problem, treating forgetting as inner maximization and retention as outer minimization. A brief convergence theory is provided. Two models and 4 datasets are tested empirically.", "questions": "See weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This work tackles an important problem with a relatively new angle. The motivation is clear. The experiments are extensive. I think the 2-level approach makes sense, and the theoretical analysis supports it.", "weaknesses": "Some parts of this paper is not self-consistent and makes it hard to trust.\n\n1. RMU (Risk Minimization Unlearning) is mis-cited.\nRMU an important baseline throughout this work. However, it is attributed to Bu et al in Section 7.4.11 but also attributed to Li et al in Section 5.2. Both citations are wrong. Bu's method is termed NGDiff and has nothing to do with risk. Li's paper is actually about WMDP benchmark, not an optimization method. Also in Section 5.1, WMDP is again mis-cited to Bu et al. \n\nIn the end, what is RMU? Did you really implement NGDiff?\n\n2. Tables are not trust-worthy.\n\nI am highly suspicious of Table 1 to 4. First of all, the bold texts seem to suggest OFMU is the best, but this is simply misleading. In Table 5, **bold texts clearly mean the best performing method**. However, in Table 1 to 4, the authors always highlight OFMU even if it is not the best under many metrics. There are so many false emphasis so I will only point out a few: Table 1, LLaMA-3.2-1B-Instruct, FTR (RMU is better but OFMU is bolded); Table 2, Random Forgetting, RA and TA columns (FT and GA and IU are much better but OFMU is bolded). \n\nSecondly and more importantly, Table 1 didn't not use fair baselines for comparison. The FQ column is almost all zeros and MU column is very low except OFMU, which is highly unlikely in the normal setting. I cannot list all entries in this big table that are suspicious to me, but I require clarity before I finalize my review score. What did I miss?\n\n3. Learning rate concerns\n\nThere is only one learning rate given in Section 7.4.12. But your 2-level algorithm needs two learning rates, inner and outer. Why not give both learning rates? Did you discuss the extra cost of tuning learning rates? If one sweeps 5 values for one learning rate, one will need 5*5=25 values to sweep for two learning rates, which can be very inefficient and impractical.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LR489GxQoY", "reviewer": "Reviewer_jsTy", "summary": "This paper proposes OFMU, an LLM unlearning method based on bi-level optimization. Specifically, the authors formulate unlearning as a bi-level problem, where the lower level maximizes the forget loss and negative cosine similarity between the forget and retain gradients. The upper level minimizes the retain loss. For efficient optimization, the authors propose to convert the original problem into a single-level problem and leverage a two-loop optimization scheme to update the model in an efficient and stable way. The authors also provide theoretical analyses to show the convergence guarantee of the proposed method. Finally, experiments on both language and vision benchmarks demonstrate the effectiveness of the proposed method.", "questions": "Please see weaknesses", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper formulates unlearning as a bi-level optimization problem, which addresses the issues in prior works where the forget and retain objectives are difficult to balance. The authors further propose a practical algorithm to solve the optimization problem.\n2. Experiments on both language and vision tasks show that the proposed method outperforms the strong baselines.\n3. The paper is well-written. The method is motivated and explained well.", "weaknesses": "1. The method introduces additional costs due to the two-loop optimization and the computation of the Hessian-vector product. How's the speed of the method compared to simple baselines such as GA or NPO?\n2. The method introduces additional hyperparameters such as the penalty parameter and the regularization parameter $\\beta$. How robust is the performance with respect to these hyperparameters?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ko9hZVyGOc", "reviewer": "Reviewer_R451", "summary": "`This paper proposes a bi-level optimization formulation for LLM unlearning, where the inner level optimizes the LLM paramater to unlearn forget data while  decorrelating forget data and retain data gradient and the outer level restores the unlearned model performance on retain data. To avoid heavy bi-level optimization, this paper proposes an iternating optimization scheme on mini-batch for stable training. This paper presents theoratical analysis and also shows performance improvement on LLM unlearning and image classifier unlearning.", "questions": "* Is the outer level optimization always fine-tuned to divergence? Algorithm 1 suggests only 1-step for outer loop.\n* How is the penalty schedule determined in this work?\n* Given that the framework operates in bi-level optimization, involving runtime information can better allow users to estimate its cost.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The bi-level optimization framework is reasonable and easy to follow.\n* The involvement of decorrelating penalty term is well motivated and promising.", "weaknesses": "* Potential retain data robustness issue. The similarity-aware penalty relies on the cosine similarity between $\\nabla_\\theta L_f$ and $\\nabla_\\theta L_r$, which assumes meaningful gradient directions from the retain data. What's the effect of different retain data on the unlearning training?\n* Concerning performance on WMDP. Table 3 in Appendix presents the performance for WMDP. From my understanding on WMDP dataset, Bio Acc. and Cyber Acc. are measuring how the unlearned LLM performs on the desired knowledge to remove (Bio/Cyber), but Table mentions higher the better, is this really correct?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SPmAVqRMhp", "reviewer": "Reviewer_9aX8", "summary": "This paper proposes OFMU, a bi-level optimization framework for machine unlearning in large language models. The key idea is to formulate unlearning as a hierarchical optimization problem where the inner loop maximizes forgetting (with a similarity-aware gradient decorrelation penalty) and the outer loop minimizes retain loss. The authors develop a penalty-based reformulation to avoid expensive nested optimization, provide convergence analysis for both convex and non-convex settings, and validate their approach on TOFU, WMDP (language) and CIFAR-10/100 (vision) benchmarks.\n\nThe method shows consistent but modest improvements over existing baselines like Gradient Ascent, NPO, and RMU, particularly in maintaining stability on hard-to-forget samples and achieving better balance between forgetting quality and model utility.", "questions": "Q1: Statistical significance\n\nCan you provide confidence intervals or statistical tests for the main results in Tables 1-3? Given the small effect sizes, this is essential to determine if improvements are real or within noise margins.\n\nQ2: Computational overhead\n\nWhat is the actual training time and memory consumption of OFMU compared to baselines like GA and NPO? How many additional gradient computations does the Hessian-vector product require per iteration?\n\nQ3: Scalability validation\n\nHave you tested on models larger than 7B parameters? If not, what are the expected computational bottlenecks that would prevent scaling to 70B+ models used in production?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1: Well-motivated framework\n\nThe hierarchical formulation clearly captures the asymmetry between forgetting (must succeed) and retention (restore afterward). The similarity-aware penalty that decorrelates forget and retain gradients addresses a real problem in existing scalarization approaches.\n\nS2: Solid theoretical grounding\n\nLemmas 1-3 provide rigorous convergence analysis for both convex and non-convex settings. The penalty reformulation (Equation 7) is theoretically justified and avoids the computational burden of fully solving the inner problem at each iteration.\n\nS3: Consistent empirical performance\n\nOFMU shows stable improvements across multiple benchmarks and scenarios. I particularly like Figure 1's analysis showing reduced coupling between sample difficulty and utility loss (though this might be from appendix content—focusing on main text results). The method avoids the catastrophic utility collapse of Gradient Ascent while achieving better forgetting than utility-focused methods like RMU.\n\nS4: Comprehensive evaluation\n\nThe paper evaluates across both language (TOFU, WMDP) and vision (CIFAR-10) tasks with multiple metrics capturing different aspects of unlearning quality.", "weaknesses": "W1: Modest and potentially insignificant improvements\n\nLooking at Table 1, the gains are small and inconsistent. For forget05 on LLaMA-2, OFMU gets FQ=0.13 vs NPO's 0.09, but NPO actually beats OFMU on forget10 (0.42 vs 0.41). More critically, the paper provides no error bars or significance tests in main results tables. Without confidence intervals, it's impossible to judge whether these differences are meaningful or just noise.\n\nW2: Computational cost completely unaddressed\n\nThe method requires Hessian-vector products (Equation 9) at each outer iteration, plus T inner gradient ascent steps. This is clearly more expensive than single-loop baselines, yet the paper provides zero analysis of wall-clock time, memory usage, or FLOPs. For a method claiming \"scalability\" (abstract), this is a major omission that undermines practical applicability.\n\nW3: Limited scale and missing ablations\n\nThe largest model tested is LLaMA-2-7B. Given that the method involves second-order computations, scalability to 70B+ models is questionable but unexplored. Additionally, key design choices lack justification—why T inner steps specifically? How sensitive is performance to β and ρ schedules? These are mentioned but not systematically studied in the main paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)