NeurIPS 2025

Ascent Fails to Forget
Ioannis Mavrothalassitis∗
Pol Puigdemont∗
Noam Itzhak Levi∗
Volkan Cevher
LIONS, École Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland
{ioannis.mavrothalassitis, pol.puigdemontplana, noam.levi}@epfl.ch
Abstract
Contrary to common belief, we show that gradient ascent-based unconstrained
optimization methods frequently fail to perform machine unlearning, a phenomenon
we attribute to the inherent statistical dependence between the forget and retain
data sets. This dependence, which can manifest itself even as simple correlations,
undermines the misconception that these sets can be independently manipulated
during unlearning. We provide empirical and theoretical evidence showing these
methods often fail precisely due to this overlooked relationship. For random forget
sets, this dependence means that degrading forget set metrics (which, for the oracle,
should mirror test set metrics) inevitably harms overall test performance. Going
beyond random sets, we consider logistic regression as an instructive example
where a critical failure mode emerges: inter-set dependence causes gradient descent-
ascent iterations to progressively diverge from the oracle. Strikingly, these methods
can converge to solutions that are not only far from the oracle but are potentially
even further from it than the original model itself, rendering the unlearning process
actively detrimental. A toy example further illustrates how this dependence can
trap models in inferior local minima, inescapable via finetuning. Our findings
highlight that the presence of such statistical dependencies, even when manifest
only as correlations, can be sufficient for ascent-based unlearning to fail. Our
theoretical insights are corroborated by experiments on complex neural networks,
demonstrating that these methods do not perform as expected in practice due to
this unaddressed statistical interplay.
1
Introduction
Machine learning models have become an integral part of modern r...