Understanding Fine-tuning in Approximate Unlearning: A Theoretical Perspective
Authors: Meng Ding
Rohan Sharma
Changyou Chen
Jinhui Xu
Kaiyi Ji
Venue: Accepted by TMLR ()

Abstract: Machine Unlearning has emerged as a significant area of research, focusing on `removing' specific subsets of data from a trained model. Fine-tuning (FT) methods have become one of the fundamental approaches for approximating unlearning, as they effectively retain model performance. However, it is consistently observed that naive FT methods struggle to forget the targeted data. 
In this paper, we present the first theoretical analysis of FT methods for machine unlearning within a linear regression framework, providing a deeper exploration of this phenomenon.  Our analysis reveals that while FT models can achieve zero remaining loss, they fail to forget the forgetting data, as the pretrained model retains its influence and the fine-tuning process does not adequately mitigate it. To address this, we propose a novel Retention-Based Masking (RBM) strategy that constructs a weight saliency map based on the remaining dataset, unlike existing methods that focus on the forgetting dataset. Our theoretical analysis demonstrates that RBM not only significantly improves unlearning accuracy (UA) but also ensures higher retaining accuracy (RA) by preserving overlapping features shared between the forgetting and remaining datasets. Experiments on synthetic and real-world datasets validate our theoretical insights, showing that RBM outperforms existing masking approaches in balancing UA, RA, and disparity metrics.

Published in Transactions on Machine Learning Research (11/2025)
Understanding Fine-tuning in Approximate Unlearning: A Theoretical Perspective
Meng Ding
Department of Computer Science and Engineering State University of New York Buffalo
Rohan Sharma
Deportment Computer Science and Engineering State University of New York at Buffalo
Changyou Chen
Department of Computer Science and Engineering State University of New York Buffalo
Jinhui Xu
School of Information Science and Technology Universily of Science and Technology of China
Kaiyi Ji
Department of Computer Science and Engineering State University of New York at Buffalo
Reviewed OpenReview: https: optere///penrrevieu net/ ttr/formm? id= XXXX
Abstract
Machine Unlearning has emerged a significant area of research, focusing on 'removing' specific subsets of data from trained model. Fine-tuning (FT) methods have become one of the fundamental approaches for approximating unlcarning, as they effectively retain model performance. However, is consistently observed that naive FT methods struggle to forget the targeted data. In this paper, we present theoretical analysis the the first to characterize FT unlearning behavior in lincar models, providing deeper exploration of this phenomenon. Our analysis reveals that while FT models achieve zero remaining loss, they fail to forget the forgetting data, as the pretrained model retains its influence and the fine-tuning process does not adequately mitigate it. To address this, we propose a novel Retention-Based Masking (RBM) strategy that constructs a weight saliency map based on the remaining dataset, unlike existing methods that focus on the forgetting dataset. Our theoretical analysis demonstrates that RBM not only significantly improves unlearning accuracy (UA) but also ensures higher retaining accuracy (RA) by preserving overlapping features shared between the forgetting and remaining datasets. Experiments synthetic and real-world datasets validate our theoretical insights, showing that RBM outperforms existing masking approaches in balancing U.A, R.A, and disparity metrics.
1 Introduction
Machine Unlearning has emerged as a prominent area that focuses on protecting individual privacy during the model training process, particularly adhering to legislation such as 'the right to be forgotten' [Rosen] 2011) under the General Data Protection Regulation (GDPR) (Hoofnagle et al. 2019). That is, removes certain training samples from the trained model upon their users' data deletion request. A natural approach to machine unlearning is to retrain the model from scratch, excluding the data that needs to be forgotten;

---

Published in Transactions on Machine Learning Research (11/2025)
Table 1: Cifar-10 Class-wise Forgetting Performance Comparing Retrain and Naive FT Tunnin) Method. The table compares Retrain and FT on CIFAR-10 across multiple evaluation metries: Unlearn- ing Accuracy (UA), Retaining Accuracy (RA), MIA-Efficacy, Test Accuracy (TA), Avg. Disparity, and Run-Time. Values in brackets indicate the gap between FT and the golden model (i.e., Retrain). Further explanations are provided in Section 5.1
Cifar- 10 Class-wise Forgetting
Methods
UA
RA 100.00 00 40.76 (59.244)
AIA-Efficacy
TA
Avg. Disparity
Run Time
Retrain
100.00=0. 0,00
100.00 20.00
94.81
0.00
82.00 2.53
FT
8.36 (911.64))
99.92_0.0s(0.08
04.4110.2000.400
37.84
this is known as exact unlearning. However, this method highly computationally inefficient. To address this challenge, previous research has proposed more relaxed definition of machine unlearning, where the unlearned model only needs to be approximately similar one retrained from scratch. This led to the development of approzimate unlearning methods, such as Fine- Tinn-Tuning (Warnecke et al. 2021 Golatkar
et al. 2020a), Gradient Ascent (Graves et al., 2021 Thudi et al.) 2022). Fisher Forgetting (Becker & Liebig,
2022 Golatkar et al., 2020a), and Influence Unlearning Izzo (Izzo et al. 2021). Fine-tuning, as one of the most
widely used approaches in approximate unlearning, has demonstrated its empirical effectiveness. However, it can be observed in many studies (Kurmanji et al.] [2024] Warnecke el al. [202]] Golatkar el. al., [2000a] Liu et al.] [2024] SSharma et al.) (2024) and our investigations in Table [I that while fine-tuning may maintain the utility of the model remaining data, it struggles to forget the targeted data. This raises natural question:
Why does fine-tuning fail to unlearn the forgetting data?
To answer this question, we revisit the machine unlcarning problem with a simple yet fundamental over- parameterized linear regression model and explore the behavior of fine-tuning through a theoretical perspee- tive. Our main contributions can be summarized as follows.
Theoretical Understanding of Fine-Tuning We provide a theoretical analysis the first to characterize FT unlearning behavior in linear models. Specifically, 1) Based on the assumption of distinct features (Assumption 3.1) our theoretical observations, which align with empirical studies, show that the remaining loss for the fine-tuning model is zero, matching that of the golden model. Moreover, the loss of the fine-tuning model on the forgetting dataset consistently remains zero, di- verging from the performance of the golden model. 2) We extend our analysis to a more complex case when the dataset retained for model retraining shares overlapping features with the forget- ting dataset. This challenges assumptions distinct feature sets across datasets, yet the previous conclusions remain valid in this case. More discussion refers to Scction R
Understanding the Benefits of Masking in Fine-Tuning: Our analysis shows that naive fine-tuning (FTT) methods fail to unlearn the forgetting data because the pretrained model retains information about this data, and the fine-tuning process does not effectively alter that retention. To address this, we propose removing the forgetting component to mitigate its retention in the pre- trained model, an approach that aligns with the masking concept proposed in existing work et al.) (2023), which directly masks the forgetting data. However, one critical case is omilted: when the remaining data and forgetting data share similar features, it becomes unclear whether those shared fcatures should be preserved. In Section our work proves: 1) Masking on the pretrained model can significantly improve unlearning accuracy while preserving the retaining accuracy. 2) When considering overlapping features, retaining them does not substantially affect unlearning accuracy, but discarding them compromises the retaining accuracy.
Retention-Based Masking, Building on the aforementioned analysis, we propose a novel RetenntionBasse Masking (RBM) strategy that constructs the weight saliency map based on the remaining dataset instead of the forgetting dataset.
To validate our theoretical results, we conduct experiments on both synthetic and First, all mask-based methods significantly improve UA compared to the naive FT method. Addi-
datasets.
2

---

Published in Transactions on Machine Learning Research (11/2025)
tionally, RBM preserves overlapping features by constructing masks based on the remaining dataset, achieving higher RA than forgetting-based methods. Furthermore, RBM consistently achieves lower average disparity, effectively balancing unlearning and retaining objectives.
1.1 Related Work
Machine Unlearning Methods. Cao & Yang (0155) first defined "Unlearning" as the removal of sample that produces the same output on the dataset as if the sample had never been trained. The natural way to solve the problem is to retrain a model from scratch in response to each data deletion request. However, retraining is not feasible due to the limited time and constrained resources. Ginart et al. (2019) provided a relaxed definition inspired by Differential Privacy (Dwork et al.) 2014). which only requires the unlearned model to produce results similar to those of retrain-fromm-scatchh models. This led to the development. of
"approximate unlearning" methods, offering more efficient computational designs for machine unlearning. Guo et al. ) Izzo et al. (2021): Neel et al. [2021]; |; Ullah et al. (2021 Sekhari et al. (2021) provide theoretical error guarantees by focusing on the empirical risk minimization problem under this probabilistic notion of unlcarning. Golatkar et al. (2020a) proposed an information-base
procedure to remove knowledge from the trained weights, without access lo the original (raining data. Further, Golatkar et al. (202bb) approximated the weights inspired by NTK theory, addressing situations where the Hessian is not informative about where the model will converge into a null space. Mchta et al. (2022) avoid the computation of Hessian by introducing a method only computing conditional independence, which identifies the Markov Blanket of parameters requiring updates. [Thudi al. (2022) proposed regularizer to reduce the 'verification model.
error,' which represents the distance between the unlearned model and cetrained-from-scratde Kurmanji et al. (2024) bears a novel teacher-student formulation to achieve better performance towards unbounded unlearning problems. Liu et al.] (2024) considers model sparsity by pruning weights before the unlearning process, thereby introducing a new unlearning paradigm. [hhen et al. (2024) incorporates the variational inference and contrastive learning approaches to address the lack supervision information (label- agnostic). Torkzadehmahani et al. (2024) leverages memorization dynamics to design localized masks that better preserve task-relevant representations, while Foster et al. (2024) computes a parameter-importanee ratio between the forget and retain sets using the Fisher information matrix. In contrast, our method derives the masking stralegy directly from the remaining dataset, guided by theoretical insights into feature overlap and retention.
Machine Unlearning Theory. For approximate unlearning, Neel et al. 2021 |5 Thudi et al. (0022) explored algorithms for empirical risk minimization objectives, while Sekhari et al. (2021) studied population risk minimization problems, providing theoretical guarantees on both the effectiveness of unlearning and the privacy of the data subjects. Guo et al. (2019); Zhang et al. (2022) provided the cortified radius with respect to data changes before and after removals, as well as the certified budget for data removals. For exact unlearning,
Cllah et al. (2021) introduced the notion of algorithmic stability, called Total Variation (TV) stability, which is suited for achieving exact unlearning. This concept was further extended to the federated setting by Che eet al. p Tao et al.] (2024). Chien et al. (2024) introduced Langevin Unlearning, which interprets noisy gradient descent as an implicit Bayesian sampling mechanism that promotes forgetting through stochastic noise injection. However, existing thcorctical work has primarily focused on utility guarantees or privacy- driven dynamics, with limited analysis explaining the successes and failures of fine-tuning methods.
Notations: In this paper, we adhere to consistent notation style for clarity. We use boldface lower letters such as x, w for vectors, and boldface capital letters (c.g. A, H) for matrices. Let A|] denote the spectral norm of A and ||v||2 denote the Euclidean norm of v. For two vectors u and v, their inner product is denoted by (u, v) or u v. For two matrices A and B of appropriate dimension, their inner product is defined as (A, B) == tr(AT (ATB) B). For a positive semi-definite (PSD) matrix A and a vector v of appropriate dimension, we write [ := Av. Denote by Pm the projection onto the space of a matrix Xm, i.c.,
P,, X,, (XTT x,m)
3

---

Published in Transactions on Machine Learning Research (11/2025)
2 Machine Unlearning in Linear Models
Let D = [(x,, W=) be a training dataset consisting of n data points, where x; represents the feature vector, and y: is the response variable for each data point in the dataset D. Assume that cach pair (x,, Wr) is a realization of the linear regression model: y = x w.: with w. Rd being the optimal model parameter in the overparameterized regime (n < d). Machine Unlearning aims to remove (or scrub) the influence of specific training data from trained machine learning (ML) model. Let D, - vx)
D represents a subset whose influence we want to scrub, termed the forgetting dataset. Accordingly, the complement of D termed the remaining dataset, is D, [(x,, Uz) ) M= = D,D The forgetting data can be represented by stacking the feature vectors and response variables as follows:
y = X2..,,,.. Xn,] E Ruxni yy == 12,, 32. ...,............................ Fuyl E 1L
Similarly, it also holds for the remaining data:
X, = [Xn 22.......; x,] pd)(n-B)), yy = 11 42, yn] E R(
The overall dataset X and y are composed separately by concatenating X, Xy and Yr.
Learning Procedure We consider the machine unlearning problem based on the fine-tuning method divid- ing the learning process into two distinct phases: Original Training and Sine-tuning (Unlearning). During the original training phase, we train model on data points X d and obtain an original model w, by opti- mizing L(w.. where L(w, D) is defined as the meen-squared-eroo (MSE) loss: L(w, D) - n x w w-yll2. For the fine-tuning (unlearning) phase, we initialize with the original parameter w. and proceed to retrain the model specifically on subset of the remaining dataset D, S D, by optimizing L(w,, Dt), where we is the unlearn model by fine-tuning.
Since we work in the overparameterized regime, where n < d, each w can perfectly fit the dataset. We can express each solution w to the following optimization problems for Original training (OT), Unlearn via fine-tuning (FT) and Golden unlearning (GU) respectively:
OT: Wo = argmin|| w/12,
s.t. y= X w
(1)
FT: W. = argmin| w wolla, 8.l. yt yt= = X, w
(2)
w
GU: Wg = argmin||w/lp s.t. yr y; = w
(3)
w
Our goal is to evaluate how the fine-tuning solution w, differs from the golden model solution wg which refers to retraining the model parameters from scratch over the remaining dataset D,. Existing work has assessed machine unlearning performance from various perspectives (Graves al.] 2021 Becker & Liebig
2022 Golatkar et al. 2020al Song et al., 22019). In this paper, we focus particularly on the Unlearning Loss
(UL) and Remaining Loss (RL), which refers to the model performance the forgetting and remaining dataset respectively. These losses are defined as:
| - vr
UL: L(w, Dy) - nf [ w -yrll2
RL: I(w, Dz)
nr
3 Naive Fine- Tame-Tuning Methods Fail To Unlearn
In empirical sludies (Kurmanji et al. 2024 Warnecke et al. 2021 Golatkar et al. 2020a) and Table 1 it can be observed that fine-tuning may retain the utility of a model but struggles to forget. In this section, we revisit this phenomenon in a simplified sotting, aiming to gain basic understanding of why the vanilla fine-tuning method succeeds in retaining the model's utility on the remaining dataset but fails to forget the targeted data it was trained on.
3.1 Distinct Features
To simplify our analysis, first consider distinct features with the following assumption:
4

---

Published in Transactions on Machine Learning Research (11/2025)
Assumption 3.1. The datasets X, and X, possess distinct non-zero features, which can be denoted x, = |R -R,,00 and ; [0, JT,, where R S ( and F S [Rd4x=s correspond to the non-zero parts, de and dg are the distinct feature numbers for remaining and forgetting data, respectively, and it satisfied that d, + =d
The assumption implics that each of these datasets contains features that are unique to each datasct is no overlap in the features present in X, and X,. The construction of X, and X, can be achieved by rearranging the samples in the matrix to group non-zero features into distinct blocks. Without loss of gencrality, we assume structure where cach matrix contains only one zero block to clear the analysis. It can be obtained immediately from Assumption 3.1 that w. = w! + w where w! and w,, are the optimal solution such that y' =x] x.w! and y" = x, wT. In an ideal scenario for classification tasks, each class possesses its own unique set of features that distinctly differentiates it from other classes. We later extended our analysis to overlapping features in Section 3.2.
Theorem 3.2. Suppose a model trained by the procedure and seporately. Under the Assumption 3.1 it holds that
RL: L(Wi, Dr) = 0, UL L(w., Dy) =0;
RL: L(W,. D,) 0, UL: L(Wg. Dy) = ||w! x
Here, we refers to the unlearned model via fine-tuning, Wg refers to the model parameter retrained from scratch, RL and UL refer to the remaining loss on the remaining data and the unlearning loss on the Jorgelling data.
Theorem 3.2 presents two interesting observations: 1) The fine-tuning model can perform perfectly on the remaining dataset, which indicates that the information from training data has been preserved from the original model, Wo: to the unlearned model via fine-tuning, w,- 2) The loss of the finc-tuning model on the lorgetting datasel consistently remains zero, which diverges from the performance of the golden model. This suggests that the fine-tuning model is unable forget the information it previously acquired from Wo. which may be contradicted by catastrophic forgetting in continual learning (Parisi et al., 2019 Ding et al. 2024
To illustrate the behavior of fine-tuning during the unlearning process more clearly, we consider the projective nature of learning. Firstly, the solution of Equation (2) can be represented as
(4) where P, is the projection space of X,, the I P, is the corresponding orthogonal space, and the w. can be also represented w, Pw, with P being the projection space of X. According to the property of projection Corollary C.I multiplying any data matrix by a projection matrix presorves the components of the data that lie within the subspace defined by the projection. Moreover, under the distinct features assumption 3.1 it holds that
W; = - P.). + P.W..
W Pw, + (P-P,,
(5)
Therefore, the unlearned model w, from Equation process: the first part, w preserves the accuracy on the remaining data, while the second part, w!, also ensures accuracy on the forget data. However, the projection of w! onto the finc-tuning space P, has no effect, ultimately resulting in the unlearned model w, being exactly the same as the pretrained model w.. The proof of Theorem 3.2 is provided Appendix C.2
(2 decomposod into two components for the unlcarning
3.2 Overlapping Features
In practical scenarios, training datasets often deviate from ideal classifications, introducing complexities such as overlapping fcatures between subsets. This challenges assumptions of distinct feature sets across datascts. Therefore, we extend our previous analysis to address the presence of overlapped features. In the following, begin by defining overlapped features.
5

---

Published in Transactions on Machine Learning Research (11/2025)
10.0
10.0
MU Loss
20.0
m
MN Log
W:
MU Loss
20.0
Me
MU Losg
w:
RL with W;
RL with W3
RL with
RL with
10.0
UL with Wy
5.0
UL with W
10.0
UL with W:
5.0
with w
Hy
with W,
w,
10
20
10
20
10
20
10
Ft
Pt
nt
Ft
(a)
(b)
(c)
(d)
Figure 1: Machine Unlearning Performance via (Masked) Fine-tuning with (without) Overlapping Features. Section 3.1 and Figure 1b present the relationship between machine unlcarning loss (i.c. RA, UA) and the number of fine-tuning data samples under distinct features and overlapping feat assumptions, using naive FT method. In contrast, Figure and Figure show the same relationship using masked fine-tuning methods, as discussed in Section
Assumption 3.3. The datasets X, and X, possess d, overlapped features, which can be structured as follows: x,T = [R [RT,Li ,L,, and x = (0, LJ F. where R S [Rd.xnr and F C idd represent the distinct features of the remaining and forgetting data, respectively. L C RduyyXn and L2 C RRduyxn Xay denote the overlapped parts.
Similarly to Assumption 3.1 d, and dy are the distinct feature numbers for remaining and forgetting data, while the equation d, + diep +dy = d holds. It can also indicates that the optimal solution can be decomposod into = ww w.. w such that y = = (w! w!")) and y" = XX (wI + w!"
Theorem 3.4. Suppose a model is trained by the procedure and 3 separately. Under the Assumption 3.3. it holds that
RL: I(We, Dr) = 0, UL: L(W,, Dy) = 0;
RL: L(W D,) 0, UL: L( Df) |..w P.w.or lop - -(w! + Wa my X
Theorem 3.4 shows that the previous conclusions remain valid under the assumptions of overlapping features, as the information from all training data, including forget data, is preserved from the pretrained model, Wo. to the unlearned model through fine-tuning, Wt- Consequently, the loss both the remaining dataset and the forgetting dataset for the fine-tuning model is zero. Additionally, interesting observation is that the number of overlapping features does not impact the unlearning accuracy of the fine-tuning model. The proof of Theorem 3.4 is provided in Appendix C.3
Both Theorem 3.2 and Theorem 3.4 present similar findings regarding the performance of the unlearned model through fine-tuning. We run a synthetic experiment to validate these results (more experimental details in Appendix B). In Section 3.1 and Figure Ib] both distinct and overlapping feature assumptions demonstrate the same results: 1) The remaining loss of fine-tuning model w, and golden model Wg is zero, indicating that the fine-tuning model performs equivalently to the golden model, successfully retaining the model's utility on the remaining dataset. 2) The unlearning loss of the fine-tuning model consistently remains at zero, differing from the golden model, suggesting that the fine-tuning model fails to forget the information obtained from the pretrained model. These empirical findings align well with our theoretical analysis.
4 Eliminating Forgetting Data Features from Pre- Trained Model Enhances Unlearning
Compared to the golden model Wg = P, P.W,, w the unlearned model can be viewed as having an additional second term as
W = Pw + (P P.)w!
This additional term (P P represents the residual influence of the data intended be forgotten on the unlearned model, contributing to the unlearning accuracy (UA) gap between w, and the golden model
6

---

Published in Transactions on Machine Learning Research (11/2025)
Wg. A natural approach to mitigate this gap might involve making the fine-tuning space converge toward the pretraining space-that is, aligning P. with Pr. However, this strategy is inefficient and contradictory, as it would lead to the optimal solution for the fine-tuning dataset becoming identical to that of the entire datasct, undermining the purpose and benefits of inn-tuning.
Inspired by the formulation of the unlearned model:
W. = -I- ( +
To mitigate the UA gap between the fine-tuning model and the golden model, it becomes evident that the remaining portion of the pretrained model does not contribute to UA. Specifically, the components of the pretrained model wa associated with the forgetting data (w!) do not enhance performance on the remaining dataset Dr. Therefore, if we eliminate the forgetting term from whethe divergence can be addressed. Tn the following, we provide a formal description of this modification. Consider the same learning procedure Equation ( to obtain the pretrained model wo. Prior to unlearning through fine-tuning, we modily wo by removing components associaled with the forgetting data. Specifically, we construct modified model wo as follows:
-aaponenntt-eccffllllyy by removing the w!
1. Distinet Fcatures Scenario. When the features of D, and Dy are distinct, we construct w. by retaining only the components corresponding to D, and setting the rest. to zero. Formally, we define w, w'[0 d,] w. d,] or equivalently can be understood
wo]l], if features of Dr:
w
0,
otherwise.
2. Overlapping Features Scenario. When features overlap across D, and Dy, we consider two cases:
Option A (Retaining Overlapping Features): We retain the overlapping features between D, and Dy, which can be expressed as wo[0 d, + dtopl wo de + diopl equivalently
woli]
if i E features of
Wo
D, U overlapping features,
otherwise.
Option B (Discarding Overlapping Features): We discard the overlapping features, which can be expressed as w.[0 d,] =w. : d,] or equivalently
woll], if features of Dr,
w.l
0.
otherwise.
Theorem 4.1. Let Wa be pretrained model obtained the overall dataset D = DD Dy. Before performing unleurning (fine-tuning), we modify wo to remove the components associated with Dj as described above. Then, using the modified models w. (Wa) in the unlearning process yields
1. In the distinct features scenario (i.e., under the Assumption 3.1 RL: I(W,, D,) 00 UT. I(W,, D2) ||w! x
have:
2. In the overlapping features scenario (i.e., under Assumption 3.3 have:
Option A (Relaining Overlopping Features):
RL: L(W,, Dr) 0;
UL: L(W,, Dy) |PW + -Pw!"p (w! 1am x,
Option B (Discarding Overlapping Features):
RL: L( [[I-P.)W!"P
P.))w!ap12 XX
UL: L(W.' Dy) = |Pw; + P.w.ap (w! + w! 1x,x

---

Published in Transactions on Machine Learning Research (11/2025)
0.8
RL with We
0.8
RL with We
RL with Wo
with Wo
0.6
0.6
UL with Wt
UL with We
MU Loss
UL with Wg
with Wg
0.4
0.4
MU
M
0.2
0.2
0.0
0.0
0
20
30
50
10
20
40
50
diap
diap
(a)
(b)
Figure 2: Comparison of Machine Unlearning Loss with and without Overlapping Features. Figure 2a retains overlapping features from the pretrained modcl, showing the matching performance between masked We model and golden model Wgi Figure 2b discards the overlapping features, showing a decline in retaining accuracy.
According to Theorem 4.1 under the distinct features assumption, the masked unlearned model achieves the same remaining and unlearning loss as the golden model. Furthermore, when considering overlapping features, if the overlapped component from the pretrained model is retained, the remaining loss remains zero, with the golden model, while the unlearning loss differs only in the projection component. This difference can be considered negligible when applied to w and w!ap duc to the model assumption. Figure 1c and Figure Td verify our theoretical conclusions. However, if the overlapped component is discarded from the pretrained model, the remaining loss is no longer zero, and there a small change to the unlcarming loss that can be overlooked. These findings offer several insights into the design machine unlearning algorithms: 1) Masking on the pretrained model can significantly improve unlearning accuracy while preserving the retaining accuracy. If related to the forgetting data, applying masking to this component can further enhance UA. Our theorem also explains recent related works, such as Liu et al. (2024); Fan et al. (2023), which apply a mask to the pretrained model either randomly or by regularizing the weights associated with the forgetting data to provide better unlearn performance. These mothods share the same underlying principle discussed here. However, one overlook a critical scenario: when the remaining data and forget ting data share similar features, it becomes unclear whether those shared features should be retained. As shown in Theorem 4.1 our work When considering overlapping features, retaining them does not substantially affect unlearning accuracy, but discarding them compromises the retaining accuracy. As shown in Theorem 4.1 the remaining loss can not retain zero unless the remaining data X, can be fully represented by the fine-tuning space, meaning P,X, = X,. Additionally, as the number of overlapping features increases, the impact on both remaining and unlearning loss becomes more significant. Discarding too many overlapping components can lead to instability in the retaining accuracy, as the model loses essential information needed to represent Dr, which in turn causes the remaining loss increase. Figure and Figure 2 validate our theoretical findings. The proof of Theorem 4.1 is provided in Appendix C.4
identify the component of the pretrained model
provides clear resolution this question. 2)
5 Rethinking Masking in Approximate Unlearning
5.1 Discarding Overlapping Features May Harm Retaining Accuracy
In Section 4 we show that once the components of the pretrained model related to the forgetting data are identified and removed, unlearning accuracy can be significantly improved. In practice, an existing work Fan et al. (2023) shares the same principles as discussed above. Specifically, it constructs the desired weight saliency map by leveraging the gradient of a loss function with respect to the model weights on the forgetting dataset:
my =1 (IV (w; Dyw, 127).
(6)
8

---

Published in Transactions on Machine Learning Research (11/2025)
where 1g 2 7) is an element-wise indicator function that outputs for the i-th element if gi 2 and 0 otherwise, 1.1 denotes the element-wise absolute value operation, and >> 0 is a hard threshold. Variables with larger gradients, which are highly associated with the forgetting dataset, are identified and subsequently masked. Then, the unlearn model can be updated by:
(Aw+ Way m O (Aw + w.)
(1-mi) --mmy Wor
salient weights
original weightss
where w. represents the original model, denotes an all-one vector and Aw is optimized by the following problem:
min.((W Aw
v/)] + (x)) (ko[(,,,,,
(8)
where y/ is the random label of the image different from y, is regularization parameter and for is the cross-entropy (CE) loss for supervised classification.
However, Equation 7 overlooks the scenario where the remaining data and the forgetting data share similar model weights, making unclear whether those weights should be preserved. In Theorem 4.1 we demon- strate that when considering overlapping features (features shared between the remaining dataset and the forgetting dataset), retaining them is more beneficial than discarding them, which contrasts slightly with the approach in Fan et al. (2023). Specifically, show that discarding overlapping features compromises retaining accuracy while retaining them ensures better performance for the remaining dataset.
Based on this analysis, we propose an alternative principle Retention-Bas Based Masking: the weight saliency map should be constructed based on the remaining dataset rather than the forgetting dataset:
m, = 1 - 1 (L L (w; D,)l (27).
(9)
Therefore, the unlearn model will be updatod by
az
(Aw w.)
+
W
(10)
salient weights
original weights
with the retention-based mask, which combines the forgetting loss on the random labels of forgotten data with the retaining loss on the remaining data.
Here, Aw is optimized using Equation S
5.2 Experiment
In the following, we verify our theoretical insights by evaluating the effectiveness of the mask-based FT machine unlcarning mothods through mumerical experiments.
Experimental Setup. We conduct our evaluations using the ResNet-18 backbone across the methods. The network is initially trained for classification over the CIFAR datasets for 182 epochs with an initial learning rate of 0.1 following a cosine decay schedule. For the unlearning procedure, the learning rate is set to 0.02 for our hot and 0.013 for the SalUn method, following the recommendations from the official repository. We set the number of unlearning epochs to 10. However, we observe that both methods benefit from a reduced number of unlearning epochs (5) in the random forgetting scenario on CIFAR-100. We set the sparsity at 50%. Unlearning using the fine-tuning method employs learning rate of 0.01 for 10 epochs. All methods utilize the SGD optimizer.
Datasets and Models. Our baseline methods include the naive fine-tuning approach (Golatkar et al. Warnecke et al.) 2021) and SalUn (Fan al.) 2023), both implemented on ResNet-18 (He et al. 2016) For comparison, we also include the golden retrained model as a reference. Our experiments will focus on image classification using the CIFAR-10 (Krizhevsky et al.] 2009 and CIFAR-100 (Krizhevsky
2020a;
et al. 2009). More details on the experimental setup will be provided in Appendix
Evaluation Metrics. We follow the existing work to assess machine unlearning performance from different aspects (Golatkar et al., (2020a] Graves al., 2021 Thudi al. 2002 Liu et al., 2024 Sharma et al. 2024 Zhu al. 2024). Specifically, we focus the following evaluation metrics:
9

---

Published in Transactions on Machine Learning Research (11/2025)
Table 2: Cifar-10, Cifar-100 Comparison. We evaluate the unlearning performance on Cifar datasets using Dense models only. Evaluation metrics from Section 5.1 are employod. Metries are reported as mean + standard deviation across five seeds. (UA: Unlearning Acc., RA: Retaining Acc., TA: Test Acc.)
Cifar-10 10 Random Forgetting TA
Methods
MIA-Efficacy
RAA
Avg. Disparity
Time (min)
Retrain
5.80+a.12
13.91 +0.15
100.00+0.00
94.30+0.13
0.00
82.15
FT SalUn
0.18 (G.622)
1.70 (122.211)
99.92 (0.08)
94. 25 (0.05)
4.48
2.51
2.38+ a. (2.42)
15.455.0.44...
99. 09.62-0.22 200-0000000000000
94. 11-11--.220 28 (0.19)
1.39
2.50
Ours
3.24 (2..56)
14.22 (0.31)
99.72 (0.28)
93.90 (0.40)
0.88
2.50
Cifar-10 Class-wise
Forgetting
Methods
UA
MIA-Effcacy
RA
TA
Avg. Disparity
Run Time (min)
Retrain
100.00=a 00
100.00.0.00
100.0010.00
94.81
0.00
$2.00
FT
40.76+5.08 os (59.24)
99.92 (0.08)
94.41=0.20 (0.44))
(0.40)
37.84
2.50
SalUn
78. (2.
100.00 (0.000))
99.47 28
93.55 (1.26) 26)
0.50
2.50
Ours
99.9610.o2
10.00000..000..0o0 (0.00)
99.69 19.69_0.30 (0.31)
93. 23.44. 0.30 (1.37)
0.42
2.50
Cifar-100 Random Forgetting RA
Methods
UA
MIA-Bflicacy
TA
Avg. Disparity
Run Time (min.)
Retrain
24.70+0.31
49.68+0.35
99.98
70.07+0.06
0.00
82.33
FT
0.11 0a (24.64)
5.66 47 (44.02)
99.97 (0.01)
75.45 (0.88)
16.85
2.50
SalUn
27.75 s3(3.00)
71.421o.1 (21.74)
08.65 20 (1.33)
68.97 10 (3.60)
7.91
1.3
Ours
23.53g0 22 22)
69.0220 0n (19.84)
78.7. 1a (1.21))
69. 1620 (5.41)
6.80
1.3
Cifar-100 Class-wise Forgetting TA
Methods
UA
MIA Eif-Efficacy
RA
Avg. Disparity
Run Time (min)
Retrain
100.00 100.00...mm no
100.00_a.au
99. i1
73.75
0.00
82, 22
FT
11.00+7.3 48)
59.33 09.33+17.sr (7)
79.7840.05 (0 (0:20)
74.61 (0.)00)
(0.86)
32.55
2.52
SalUn
100.00 00..0....0.0000 (0.00)
100.00 (0.00)
99 63 (0.35)
74.76 a1 (1.001
0.35
2.50
Ours
100.00+0..00 (0.000
100.00+0.co(0.000
99. 00.90+0..0
74. 74.47
0.19
2.50
Unlearning accuracy (UA): We define UA(w,) = 1- - AccD, (wa) Liu et al. (2024), measuring how effectively the model has forgotten the targeted data. Here Accn, (Wa) is the accuracy of the unlearned model on the forgetting dataset.
Membership inference altack (MTA) on Dy (MIA-Ffffcacy): The efficacy of MTA on the forget dataset, which assesses whether the model still retains any identifiable information about the for- getting data.
Retaining accuracy (RA): The accuracy of the model on the remaining dataset D, after unlearning, measuring how well the model retains its performance from the pretrained model.
Testing accuracy (TA): The accuracy of the model on the independent test dataset, indicating its generalization ability after unlearning.
Average Disparity (Avg. Disparity): The mean absolute difference between the performance metrics of the unlearned model and the retrained model, calculated across all evaluation aspects.
Run-time efficiency (RTE): RTE evaluates the computational efficiency of the unlearning process, including the run-time cost taken to exccute the unlearning procedure.
Note that a smaller performance gap between the unlearned model and the golden retrained model indicates the better performance of approximate unlearning.
Masking Strategies Enhance Unlearning Accuracy in Naive Fine Fine-Tuning. The performance of Retention-Based Masking (Ours) and Forgetting Based Masking (SalUn) against Fine- Tuning highlights how masking significantly improves unlearning accuracy, measured as the distance from the retrained model. Specifically, in CIFAR-10 Class-wise Forgetting, naive fine-tuning exhibits significant gap of 919.646 compared to the retrained model, which is drastically reduced by SalUn (0.22) and further refined by RBM to 0.02. Similarly, for CIFAR-100 Class-wiee wise Forgetting, fine-tuning shows a large gap of 87.96, reduced to 0.00 by SalUn and maintained by RBM at 0.00. In random forgetting scenarios, in CIFAR-10 Random Forgetting, fine-tuning achieves gap of 5.62, while SalUn reduces this 3.42 and RBM further minimizes it to 2.56. A similar trend is observed for CIFAR-100 Random Forgetting, where fine-tuning starts with gap of 24.64,
Fine-Tuning
10

---

Published in Transactions on Machine Learning Research (11/2025)
SalUn reduces it to 3.00, and RBM achieves further improvement to 1.22. These results demonstrate that masking strategies, whether leveraging forgetting-basel or relention-based saliency, are essential for enhancing unlearning accuracy in naive fine-tuning,
Retention- Masking Improves Retaining Accuracy. Next, we compare retention based masking (Ours) and forgetting-based masking (SalUn) evaluate how different masking approaches impact retaining accuracy, measured by the distance from the retrained model. For CIFAR-10 Random Forgetting, SalUn achieves gap of 0.38, while RBM improves it to 0.28. For CIFAR-100 Random Forgetting, SalUn shows gap of 1.33, whereas RBM significantly reduces it to 1.24. In wise forgetting scenarios, RBM consistently maintains an advantage: for CIFAR-10 Classwise Forgetting, SalUn achieves gap of 0.53 compared to 0.31 for RBM. Similarly, for CIFAR-100 Class-wise Forgetting, SalUn achieves a gap of 0.35, while RBM further improves this to 0.08. These results underscore that the retention-base saliency approach more effective in preserving the performance of the remaining dataset, aligning with our theoretical insights Theorem 4.1
Retention-Based Masking Minimizes Average Disparity Across Tasks. In addition to improving retaining accuracy, RBM also reduces average disparity across all tasks. For CIFAR-10 Random Forgetting, RBM reduces the disparity to 0.88, compared to 1.39 for SalUn. Similarly, in CIFAR-100 Random Forgetting, RBM achieves a reduced disparity of 6.80, lower than 7.91 for SalUn. In CIFAR-10 Classwise Forgetting, RBM reduces the disparity to 0.42, compared to 0.50 for SalUn. For CIFAR-100 Class-wise Forgetting, RBM achieves an exceptionally low disparity of 0.19, significantly lower than 0.35 for SalUn. These results further validate the robustness RBM in effectively balancing retaining and unlearning objectives.
40
40
40
Remaining Date
t:5NE 2omponent
I:SNE Component
t-SNE Component
Forgetting Data
20
20
0
-20
-20
Remaining Data
Remaining Data
Forgetting Data
Forgetting Data
-40
-20
40
-25
0
40
t-SNE Component 1
t-SNE Component
t-SNE Component
(a) CIFAR-10-Class 3
(b) CIFAR-10-Class 6
(c) CIFAR-10-Class 9
40
Remaining Data
40
Remaining Data
40
Remaining Data Forgetting Data
t:SNE Component 2
Forgetting Data
t:NE Component 2
Forgetting Data
t:SNE 2omponent 2
20
20
20
0
-20
-20
-20
-40
-40
-40
-40
-20
0
20
40
-40
-20
20
40
40
-20
0
20
40
t-SNE Component
t-SNE Component
t-SNE Component
(d) CIFAR-100-Claa 30
(e) MUFAR-100-Class 60
(I) CIFAR- 1AALR-10--CIA Class 90
Figure 3: Visualization of Remaining Data and Forgetting Data Features Across Various Dataset. Figures
3a 3c focus on classes 3, 6, and 9 in CIFAR-10 and Figures 3d focus on classes 30, 60, and 90 in CIFAR-100.
Visualization of Remaining Data and Forgetting Data Features. The visualization in Figure uses I-SNE to project feature representations of the forgetting and remaining data in CIFAR-10 and CIFAR-100 datasets. The red points correspond to forgetting data, and the blue points represent remaining data. This visualization aims to demonstrate that, class-wise datascts, the unlcarning task for a specific class may involve distinct features. In such cases, naive fine-tuning (FT) methods tend to contribute less towards forgetting the class and focus more on retaining features from the pretrained model.
11

---

Published in Transactions on Machine Learning Research (11/2025)
6 Conclusion
In conclusion, present the first theoretical analysis of fine-tuning methods for machine unlearning within linear regression framework. Our analysis, covering Iwo scenarios distinct and overlapping feature sets sets- demonstaaasss that while fine-tuning can achieve optimal retaining accuracy, it fails to fully unlearn the forgetting datasct. This failure arises from the pretrained model retaining information about the forgetting data. To address this, we propose theoretical solution and introduce Retention-Base Masking (RBM), a strategy that constructs masks based on the remaining dataset to preserve overlapping features. Our experi- mental results demonstrate that RBM achieves a better balance between unlearning and retaining objectives compared to torgotten-based masking approaches.
References
Alexander Becker and Thomas Liebig. Evaluating machine unlearning via epistemic uncertainty. arXiv preprint arx00:208.. 10836, 2022.
Yinzhi Cao and Junfeng Yang. Towards making systems forget with machine unlearning. In 2015 IEEE symposium security and privacy, pp. 463-480 IEEE, 2015.
Tianshi Che, Yang Zhou, Zijie Zhang, Lingjuan Lyu, Ji Liu, Da Yan, Dejing Dou, and Jun Huan. Fast federated machine unlearning with nonlinear functional theory. In International conference machine learning, pp. 4241 4268. PMLR, 2023.
Eli Chien, Haoyu Wang, Ziang Chen, and Pan Li. Langevin unlcarning: new perspective of noisy gradient descent for machine unlearning. Advances in newral informalion processing systems, 37:79666 79703, 2024.
Meng Ding, Kaiyi Ji, Di Wang, and Jinhui Xu. Understanding forgetting in continual learning with linear regression. In Forty-first International Conference Machine Learning. 2024.
Cynthia Dwork, Aaron Roth, et al. The aigc foundations of differential privacy. Foundations and Trends@ in Theoretical Computer Science, 9(3-4):211-477, 2014.
Chongyu Fan, Jiancheng Liu, Yihua Zhang, Dennis Wei, Eric Wong, and Sijia Liu. Salun: Empowering machine unlearning via gradient- based weight saliency in both image classification and generation. arXiv preprint XrX::2330.. 12508, 2023.
Jack Foster, Stefan Schoepf, and Alexandra Brintrup. Fast machine unlearning without retraining through selective synaptic dampening. In Proceedings of the AAAI conference on artificial intelligence, volume 38, pp. 12043 12051, 2024.
Antonio Ginart, Melody Guan, Gregory Valiant, and James Y Zou. Making ai forget you: Data deletion in machine learning. Advances in neural information processing systems, 32, 2019.
Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Eternal sunshine of the spotless not: Selective forgetting in deep networks. In Proceedings of the IEEE/CVP Conference on Computer Vision and Pallern Recognition, pp. 9304 9312, 2020a.
Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Forgetting outside the box: Scrubbing deep networks of information accessible from input-output observations. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXIX 16, pp. 383-398. Springer, 2020b.
Laura Graves, Vincel Nagisetty, and Vijay Ganesh. Amnesiac machine learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 11516 11524, 2021.
Chuan Guo, Tom Goldstein, Awni Hannun, and Laurens Van Der Maaten. Certified data removal from machine learning models. arXiv preprint Xim:1911 11000000, 2019.
12

---

Published in Transactions on Machine Learning Research (11/2025)
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognilion, pp. 770-778, 2016.
Chris Jay Hoofnagle, Bart Van Der Sloot, and Frederik Zuiderveen Borgesius. The european union general data protection regulation: what is and what means. Information Communications Technology Law, 28 2019.
Zachary Izzo, Mary Anne Smart, Kamalika Chaudhuri, and James Zou. Approximate data deletion from machine learning models. In International Conference on Artificial Intelligence and Statistics, pp. 2008- 2016. PMLR, 2021.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Meghdad Kurmanji, Peter Triantafillou, Jamie Hayes, and Eleni Triantafillou. Towards unbounded machine unlcarning. Advances in Neural Information Processing Systems, 36, 2024.
Jiancheng Liu, Parikshit Ram, Yuguang Yao, Gaowen Liu, Yang Liu, PRANAY SHARMA, Sijia Liu, et al. Model sparsity can simplify machine unlearning. Advances in Newral Information Processing Systems, 36, 2024.
Ronak Mehta, Souray Pal, Vikas Singh, and Sathya N Ravi. Deep unlearning via randomized conditionally independent hessians. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recoguition, pp. 10422- 10431, 2022.
Seth Neel, Aaron Roth, and Saced Sharif-MMalvajerdi: Descent-to-dellte: Gradient- methods for ma- chine unlearning. In Algorithmic Learning Theory, pp. 931-962. PMLR, 2021.
German T Parisi, Ronald Kemker, Jose I, Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural networks, 113:54 71, 2019.
Jeffrey Rosen. The right to be forgotten. Stan. L. Rev. Online, 64:88, 2011.
Ayush Sckhari, Jayadev Acharya, Gautam Kamath, and Ananda Theertha Suresh. Remember what you want to forget: Algorithuss for machine unlearning. Advances in Neural Information Processing Systems, 34:18075-18086, 2021.
Rohan Sharma, Shijie Zhou, Kaiyi Ji, and Changyou Chen. Discriminative adversarial unlearning. arXiv preprint arXi::2402.000 06864, 2024.
Shaofei Shen, Chenhao Zhang, Yawen Zhao, Alina Bialkowski, Weitong Chen, and Miao Xu. Label-agnostic forgetting: A supervision-free unlearning in deep models. arXiv preprint arXiv:2404005500 2024.
Liwei Song, Reza Shokri, and Pratock Mittal. Privacy risks of securing machine learning modcls against adversarial examples. In Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communi- cations Security, pp. 241 257, 2019.
Youming Tao, Cheng-Long Wang, Miao Pan, Dongxiao Yu, Xiuzhen Cheng, and Di Wang. Communication efficient and provable federated unlcarning. arXiv preprint ar arXiv:2401. 11018, 2024.
Anvith Thudi, Gabriel Deza, Varun Chandrasekaran, and Nicolas Papernot. Unrolling sgd: Understanding factors influencing machine unlearning. In 2022 IEEE 7th European Symposium on Security and Privacy (EuroS8P), pp. 303 319. IEEE, 2022.
Reihaneh Torkzadehmahani, Reza Nasirigerdeh, Georgios Kaissis, Daniel Rueckert, Gintare Karolina Dziu- gaite, and Eleni Triantafillou. Improved localized machine unlearning through the lens of memorization. arXiv preprint Xrx:::241 22.4142.2244 2024.
Enayat Ullah, Tung Mai, Amup Rao, Ryan A Rossi, and Raman Arora. Machine unlearning via algorithmic stability. In Conference on Learning Theory, pp. 4126 4142. PMLR, 2021.
13

---

Published in Transactions on Machine Learning Research (11/2025)
Alexander Warnecke, Lukas Pirch, Christian Wressnegger, and Konrad Rieck. Machine unlearning of features and labels. ar arXiv preprint arXin:218.. 11577, 2021.
Eduardo H Zarantonello. Projections on convex sets in hilbert space and spectral theory: Part i. projections convex sets: Part ii. spectral theory. In Contributions to nonlinear functional analysis, pp. 237-424 Elsevier, 1971.
Zijie Zhang, Yang Zhou, Xin Zhao, Tianshi Che, and Lingjuan Lyu. Prompt certified machine unlearning with randomized gradient smoothing and quantization. Advances in Newral Information Processing Systems, 35:13433-13455 2022.
Jianing Zhu, Bo Han, Jiangchao Yao, Jianliang Xu, Gang Niu, and Masashi Sugiyama. Decoupling the class label and the target concept in machine unlearning. arXie preprint arXiv:2406.0822 2024.
A Discussion
Our analysis is limited to the overparameterizee linear regression setting, which simplifies understanding but does not fully capture the complexity of modern nonlinear models. The assumptions of distinct and overlapping features are idealized and may not directly reflect real-world data distributions. Empirically, our evaluation is restricted to moderate-scale datasets and finetuming-basee comparisons to future work. Extending our framework to nonlinear or large-scale settings and exploring efficient or certified unlearning mechanisms are promising directions for future research.
baselines, leaving broader
B
Experimental Details
B.1 Verification via Simulation
To empirically validate the theoretical findings presented in Theorem 3.2 Theorem 3.4 and Theorem 11 regarding the performance of unlearned models through fine-tuning, first conducted series of synthetic experiments.
Data Generation. We constructed two data matrices, X, and X,, representing the remaining data and the forgetting data, respectively. The remaining data matrix, X was structured as TR,LL] ,LT,, O], and the forgetting data matrix, XJ, as [0, LJ, FJ,, where LJ L to enforce the non-overlapping case. Here, R and F are random matrices corresponding to different feature sets, and the zeros represent the distinct features across the datasets. We set the total number of data points to = 40 and the total number of features to = 40. The remaining data consisted n, = 30 samples with d, = 20 features, while the lorgetting data comprised ny = 10 samples with d, = d, 20 features. To simulate a controlled environment, we fixed the number of overlapping features diop and diap = 8 for non-overlapping case and overlapping case, respectively.
Label Generation. We generated the true coefficient vector w. E & by sampling from a standard normal distribution. The labels were created using linear regression model without added noisery = X w.,. T The labels were partitioned into y, and y,, corresponding to the remaining and forgetting data, respectively.
Model Training. To compare the effects of fine-tuning, we considered two models: the fine-tuning model W, and the golden model Wg. Specifically, w, was obtained by fine-tuning on subset of the remaining data, denoted X,, which consisted of the first ne data points from X,. The value of ni varied from n, - to study the impact of the fine- mue-tuning data size. The initial model w. was derived from the entire dataset X and calculated by the Equation ① Wg was trained from scratch on the entire remaining data X, and computed by solving Equation (2) considering the masked case in synthetic data, the masked pretrained model will be constructed by zeroing out the coefficients corresponding to the forgetting data features with (without) overlapping features.
Evaluation Metrics. The performance of the models was assessed using the Mean Squared Error (MSE) both the remaining and forgetting data:
14

---

Published in Transactions on Machine Learning Research (11/2025)
Remaining Data Loss (RA Loss):MSERA (w) ni X, yr 1 ny | yyl1
Unlcarning Data Loss (UA Loss):MSEUAA (ANSSEEAAWWWWWW- =
Experimental Results Figure Ic and Figure Id illustrate that the masked line-tuning method discussed Section can signilicantly improve unlearning accuracy while preserving the relaining accuracy. Specifically, both the remaining loss and unlearning loss of We perfectly match those of the golden model under both distinct and overlapping feature scenarios. Additionally, Figure present comparisons of machine unlcarning loss for different approaches to handling overlapping features: Figure 2a retains overlapping features from the pretrained model, demonstrating matching performance between the masked w, model and golden model Wg: whereas Figure 2b discards the overlapping features, resulting in decline in retaining accuracy. These empirical results align well with our thcorctical findings.
B.2 Additional Real-world Details
Table 3: Unlearning Results on
Performance comparison on the more challenging TinylmageNet dataset. Random 10% forgetting scenario with 50% masking. (RA: Retaining Accuracy, UA: Unlearning Accuracy, TA: Test Accuracy, MIA Efficacy: Membership Inference Attack Efficacy,
Avg. Disparity: utility privacy imbalance).
Method
RA
UA
TA
MIA Efficacy
Avg. Disparity
Retrain RBM
99.98
39.91
62.21
65.13
96.89
47.23
59.91
69.98
4.39
SalUn
97.20
31.30
59.10
71.08
5.11
We further evaluate approach the TinylmageNett dataset. This dataset is a subset of ImageNet, containing 200 classes with 500 training samples per class and 50 samples each for validation and testing, We consider a random forgetting scenario involving 10% of the data, with a masking ratio of 50% for both methods. The larger class space and higher visual diversity make this a particularly challenging benchmark. Nevertheless, our approach substantially outperforms the baseline method, showing only a moderate decline in test cecuracy-reflectie
the dataset's inherent complexity. RBM maintains high retention accuracy while effectively mitigating MTA attacks to levels comparable to retrained model, thereby achieving lower average disparity relative to this standard.
Table 4: SVHN Unlearning Results using VIT Tiny. Evaluation of unlearning methods on SVHN. RA: Retaining Accuracy, UA: Unlearning Accuracy, TA: Test Accuracy, MIA Efficacy: Membership Inference Attack Efficacy.
Method
RA
UA
TA
MIA Efficacy Avg. Disparity
Retrain
100.00
9.76
89.38
15.27
RBM
94.12
8.15
88.73
13.82
2.39
SalUn
95.93
7.00
88.55
17.89
2.57
To further assess generality, we incorporate a new set of experiments using Vision Transformer (ViT) backbone, extending beyond the conventional ResNet-based architectures commonly employed in prior work. Specifically, we adopt the ViT-Tiny model (approximately 5.7M parameters) and evaluate it the SVHN dataset. We randomly forget 10% of the data using 50% masking ratio. The learning rate for the VIT baseline is set to x 10-3, and we perform a learning-rate ablation for SalUn by varying it between 6 x 10-4 and 2 x 10 obtaining the best performance at 10-3 -3 For our method, the learning rate is adjusted 10-4, yielding more stable convergence and consistent improvements across all metrics. As shown Table our proposed method consistently surpasses the forget-set- based unlearning method SalUn across all evaluated metrics.
15

---

Published in Transactions on Machine Learning Research (11/2025)
C Proofs
C.1 Useful properties
Before presenting the detailed proofs of the theorems, we first introduce several useful propertics of the projection matrix and the minimum norm solution.
Property 1 (Projection properties). Let P be projection operator that projects onto a subspace X C Rdxx, Then, P holds the following properties:
Symmetric: P=P PT;
Idempotent: p2 PP P;
3. I P is also projection operator, projecting onto the subspace orthogonal to X. Therefore,
- (I-P)P 0:
Let v E Rd be an arbitrary voctor, it holds that |I(I - P)v]] v (I- v Plv
||v - ||
5. Contraction:
[Pv|
< |v]], holding in equality if and only if Pv
Proof. See Zarantonello 1971 for the proofs and for more properties.
P,,,P,P be the corresponding projection operator for X, X,, X X2 respectively. Under Assumption 3.1 the remaining(forgettnng) dataset
Corollary C.1 (Projection Matrix properties). Let P X(X
matric be denoted X, R and X,
0
F 1, where R Rd-x(n-ny) and F C Rddxxn correspond
to the non-zero parts. Then, it holds that:
R(R R) R
O
1. P
0
F)
FFT
F
P. PPf
R(R
R
R
0 0
0
O
2. P,
0
and P =
O
F(F F(F 'F
XII-P)=(I-) P\X O, and the conclusion also holds for P,, p Pe with X,, N,,X,, X, respectively;
4. For any matric that is a submatria X, it holds that A = PA, where P is the projection space of X. Moreover, if P is the projection space of A, it holds that PPA = PAr i.c. X,P = =X,,
x = - x,
R
Proof of Corollary
C.1 Firstly, based on the data composition, the overall dataset holds X
Therefore, it follows:
R
R
0
R
0
R
0
P= X
1
1(
][
F
0
F
0
F
0
F
= R
0 F
R
R)
R)
0
R
0
II
F
F)
O
FT
The remaining Projection matrices be obtained by similar computations.
Additionally, we have X(I (I (I-PPPA P)X X(X - 0.
Moreover, since A is a submatrix of X, it can be represented as = XC for some selective matrix C. Therefore, we have:
PA X -X
X
16

---

Published in Transactions on Machine Learning Research (11/2025)
Meanwhile, it also holds that
PPA = X(XX XXX(CCXXXX)-CCCC Xc( PA.
X, and y are submatrices X, each with disjoint spaces. The projection X, onto the space of X, should be zero.
X,P, XXx =0
Corollary C.2 (Minimum Norm Solution 1). Let P.P, P, be the corresponding projection operator for X, X,, K,, X₁ respectively. Then, the solution to the optimization problem Equation Equation be represented by:
X,.
and
Equalion
Under Assumption 3.1
W. Pw., Wt (I P.)., + P.w., and Wa P.W.;
Under Assumption 3.
Wo = Pw., Wt = (I - P,),, + Pi(wT + (o)), and w, Pr(w + !
3. X w = 0 and X, w = O.
According to the method of Lagrange multipliers and the problem setup, it is casy to obtain the first two conclusions. For the last one, we have:
Proof of Corollary C.2
JW.. = 0. X w = R .Ol! =0 and X. = (0, F lw,
C.2 Proof of Theorem 3.2
Let us first focus on the performance of the golden model. Based on the definition of unlearning accuracy and retaining accuracy, we have
RL: L( D,) = - |x yr
_I..P,W - 1]
7hr
Thy
where the second equality arises from the model setting and Proposition C.2 is due to the properties of the projection matrix. According to Corollary
while the penultimate equality
we have
UL: L(Wg. Dy)
X 1 W.
y/11
X
X.P.W.
W X w!
=
nz
nz
| R
R R
0
| |x]wW! w! w X w!
1 ns
nz
0
0
Similarly, for the fine-tuning model, it holds that
1
1
RL: L(we, D.)
X yr - | ( (P)..).. + P...) x w!].
74 1
n4
| -P:)PPP.." + w!) +P.W.) - X.W.!12
nr
1
IX_((P) (P (P- -P)wwW! - x.I.
7r
0.
17

---

Published in Transactions on Machine Learning Research (11/2025)
1 -
J ysll - | (1- P.)w. + P.ww( - X, w!
UL: L(w,, Dj)
Tif
| J[[ P.)Pw. + - -w(112
/
P + ([ Iw
n5
Pw. + X Pw! - X, w!ll
n
0,
where the penultimate equality comes from X, X, = 0, and the last equality follows from X X
C.3 Proof of Theorem 3.4
Due to the assumption overlapping features, the projection properties of the dataset matrix will be slightly different. Specifically, it holds that:
Corollary C.3 (Projection Matrix propertics'). Let P X P. be the corresponding projection operator for X, X,, x X, respectively. Under Assumplion
3.3
it holds that:
R L L ) R
R L L
L L
P
L R R R
0 0
0
0 Lg + L
0
3. P =
L2
L
FF
L L2
FT
F(F L L2) L
F(F'+L,,4
L2)
FT
3. X(I-P)= (I-P) = O, and the conclusion also holds for Pr., pr,P, Pe with X, X,. X, respectively;
For any matrix A is the submatria of X, it holds that A PA, where P is the projection space of X. Moreover, PA is the projection space of A, it holds that PP = PA.
Proof of Corollary
C.3 Proof of Corollary
C.3 follows the proof of Corollary C.1 directly.
j
Now we ready to go through the proof of Theorem 3.4 Similar to the non-overlapping case, the golden model holds that
RL:
L(Wg, D.)
1
X wo yr M4y
X P. (w? w!r)) X, (w, )|
Thy
- x (P, I)(W. + !1 0,
1
n4
where the second equality also arises from the model setting and Proposition C.2
while the penultimate
equality is due to the properties of the projection matrix. According to Corollary C.3 we have
1 UL: L(Wg. Dy) |X] w - ny
1
['P,,(W, + w!op) x (w! !1
where P.P.W wt and ,,P. w tap follows that
R(R R + - R
RRRR-LLLE 0
- F
LR R R R
LRRRLLL 0
LL R R + L L1) -R R TRTw. w.
18

---

Published in Transactions on Machine Learning Research (11/2025)
and
R R + L 'R' RRRR+LL
0
X P. P. = -, F
L R L -R L(R
0
= LL R L1) L w 1np
For the fine-tuning, the retaining accuracy follows:
RL: L(W.,D,) 1
X Wt - yrl12
| ((I - -P,) + P.(w, + w!o - x (w + )!
X (I- -P))(w. -w- w!
1
X (I - --I w
Rr
0.
The last equality derives from that the facts the projection matrix is commutative matrix and the last property holds in Corollary C.3 For the unlearning accuracy, it holds that
1 UL: L(w,, Dy)
[ We y
nz
1x] - P.)w. + Pi(W + w!PP)) - (w( + w|
725
(P), + w!OP w!)) + P,(w.. + w!(P)) -x, (w! + !!
Xx] [X]((P -P)(WW + waa + w!)) P.(w + w!(P)) - x7 (w! + )|
ny
[ 1x]10 (P -Iw Pw. + (P I- P,)w!]l
1
| !!
0,
where the penultimate equality is due to Corollary C.3 and the last equality comes from the fact X, enjoys the same data structure as X, such that:
P.P.
Rr + Lir) Rr + LIT
T
0
0
(0, LJ, F ]
Lir(R L1d) R LR L LiT)
LLT
0
0
0
0
[L] LLT(RRR + Lir) -1 RR L,J LıT(R
(R, Ry + LIT L 0]
0.
19

---

Published in Transactions on Machine Learning Research (11/2025)
C.4 Proof of Theorem 4.1
For the non-overlapping case, we have that the retaining accuracy follows:
1
yrl1? 1 | (I-P.))- + P.w..) w!l?
RL: L(Wi,D,)
x We
n4
1
I (I
P.) - -?
1
x (I- =0.
For the unlearning accuracy, it holds that
1
X3 JWJ. y.112 = - ||] ((I P.)w. + P.wC) - Xj w!
UL: L(ws, Dy)
n25
ni
1
| [II Po)Pww P.w", -w
f
X [ - P,P. + P.Jw - XJw w/]?
P. x7 w!ll?
| xx,
nf
For the overlapping case, it holds that
1
RL: L(we, D.)
X Wr yr112
74
| ((I P.) + P.(w. + w!(p)) X., (w: + !1
74
| (I (P.,) - w- -5-wi w!
74
1
| (I - (P) (IWW - -w!apy11= = 0.
1
UL: L(W,, Df)
| W. y1112
n
1
|X ((I P.).. + P.(W, + w(er)) x7 (w! + )1
nz
|x] (P).)))P(W. + w!PP)) + P.(w + w!rr)) - X3 (w) + w/oP)112
n.
1
|x] ((P - -P.( + w!PP)) P.(,, w!!P)) x, (w! + w!
nj
|x] (P- -I]w!Pp + Pw - ww!ll
ny
1
IP(W + w!ap) - (w! + w!o) 2 X,
The proof is then complete.
20


Reviews:
[{"review_id": "fEdAJyCBt6", "reviewer": "Reviewer_bujA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work makes a number of contributions to the machine unlearning literature, which can be summarized as:\n\na. This work presents the theoretical analysis of FT methods for machine unlearning within a linear regression framework.\n\nb. The analysis reveals that while FT models can achieve zero remaining loss, they fail to forget the forgetting data, as the pretrained model retains its influence and the fine-tuning process does not adequately mitigate it.\n\nc. Building on this understanding, the paper introduces a Retention-Based Masking (RBM) strategy, which constructs a weight saliency map based on the remaining dataset, unlike existing methods that focus on the forgetting dataset. \n\nd. This paper provides experiments on synthetic and real-world datasets validate the theoretical insights.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The claims made in the submission are supported by accurate, convincing, and clear evidence. The paper provides:\n\na. Theoretical analysis of fine-tuning in approximate unlearning, with formal assumptions and theorems (e.g., Theorem 3.2, Theorem 3.4, Theorem 4.1).\n\nb. Synthetic experiments that validate the theoretical results under both distinct and overlapping feature assumptions.\n\nc. Comprehensive experiments on CIFAR-10 and CIFAR-100 demonstrate the effectiveness of the proposed Retention-Based Masking (RBM) method across multiple evaluation metrics.", "audience_yn": "Yes", "audience_explanation": "The work addresses a very timely and important problem in machine unlearning, which is directly connected to privacy regulations such as the right to be forgotten under GDPR. The paper also contributes a theoretical explanation of why fine-tuning fails to fully unlearn.", "requested_changes": "a. The distinct/overlapping feature assumptions (Assumptions 3.1 and 3.3) are central, but the realism of these assumptions in high-dimensional nonlinear models is not sufficiently discussed. A clearer justification of how these cases map to practical settings is essential.\n\nb. It is necessary to provide a more explicit computational complexity analysis of RBM.\n\nc. The paper briefly mentions catastrophic forgetting. It will be more interesting to provide a more explicit connection to continual learning theory.\n\nd. The t-SNE plots (Appendix A.2) could be brought into the main text to better illustrate the intuition.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "hhKUbRvfrm", "reviewer": "Reviewer_GuEV", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper investigates why fine-tuning (FT), a common method for approximate machine unlearning, often fails to forget targeted data despite maintaining performance on remaining data. Through a theoretical analysis within an overparameterized linear regression framework, the authors show that FT models preserve information from the original pretrained model, preventing effective forgetting. They extend the analysis to scenarios with both distinct and overlapping features, confirming that the forgetting failure persists in both cases. To address this, they propose Retention-Based Masking (RBM), which constructs saliency maps based on the remaining dataset instead of the forgetting dataset, thereby preserving overlapping features. Experiments on synthetic and real-world datasets validate that RBM significantly improves unlearning accuracy, retaining accuracy, and fairness compared to existing fine-tuning and masking approaches.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper provides a solid theoretical framework using overparameterized linear regression to explain why naive fine-tuning fails in approximate unlearning, and the proofs are carefully aligned with both distinct and overlapping feature settings. The proposed Retention-Based Masking (RBM) is supported by both mathematical analysis and empirical validation on synthetic and real-world datasets, with results showing clear improvements over existing methods. Overall, the evidence is convincing and clearly presented, though additional experiments on larger-scale or more diverse benchmarks could further strengthen the generality of the claim.", "audience_yn": "Yes", "audience_explanation": "Yes, the findings of this paper would be of strong interest to TMLR’s audience, as machine unlearning is a rapidly growing area motivated by privacy regulations such as GDPR and the “right to be forgotten.” The paper addresses a fundamental question about the limitations of fine-tuning for unlearning, providing both theoretical clarity and a practical improvement through Retention-Based Masking. These insights are valuable not only for researchers in privacy-preserving machine learning but also for those working on reliable and efficient model update strategies.", "requested_changes": "1. The adopted baselines are not very recent. Including baselines from the latest 2024–2025 works would better position RBM within the evolving landscape.\n\n2. The current experiments (mainly CIFAR datasets) are somewhat limited; adding results on larger or more diverse datasets would strengthen the claim that RBM generalizes beyond small-scale benchmarks.\n\n3. The experiments are mostly conducted using ResNet backbone, it would be interesting to see the results on other backbone, e.g., ViT.\n\n4. Analyzing how RBM’s performance varies with different mask thresholds and degrees of feature overlap would provide stronger evidence of its robustness.\n\n5. It would be better to give more details regarding the random forgetting and class wise forgetting scenarios.", "broader_impact_concerns": "I do not see any ethical concerns in this work that would necessitate a Broader Impact Statement. The paper’s contributions focus on theoretical analysis and methodological improvements for machine unlearning, without raising issues related to fairness, misuse, or sensitive applications. As such, the ethical implications appear minimal and sufficiently addressed by the scope of the research.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "hnhWozDKhV", "reviewer": "Reviewer_GWBK", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a theoretical analysis of fine-tuning methods for approximate machine unlearning, focusing on the overparameterized linear regression setting. The authors investigate why naive fine-tuning fails to effectively unlearn targeted data and propose a novel masking strategy called Retention-Based Masking (RBM) to address this issue. The paper provides theoretical and empirical evidence showing that RBM can improve unlearning accuracy while preserving retaining accuracy, particularly in scenarios with overlapping features between the forgetting and remaining datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper provides the first theoretical analysis of fine-tuning for machine unlearning in the linear regression setting. The proposed RBM strategy is a direct and logical consequence of the theoretical analysis. The insight that masking should be based on the remaining data to preserve shared features is novel and well-supported by the theoretical results. The paper includes experiments on both synthetic and real-world datasets (CIFAR-10) to validate the theoretical claims. The synthetic experiments are particularly effective in illustrating the theoretical concepts of distinct and overlapping features.", "audience_yn": "Yes", "audience_explanation": "This paper makes a solid contribution to the field of machine \"unlearning\" by providing a much-needed theoretical foundation for understanding fine-tuning-based approaches. The proposed RBM method is well-motivated and shows promising results.", "requested_changes": "The theoretical analysis is confined to the overparameterized linear regression setting. While this provides a valuable starting point, the extent to which these insights generalize to more complex, non-linear models is not immediately clear. The authors should discuss the potential challenges and future directions for extending their theory to more practical settings.\n\nThe analysis relies on strong assumptions, particularly the \"distinct features\" and \"overlapping features\". While these assumptions are useful for simplifying the analysis, they may not accurately reflect the complexities of real-world data. A discussion of the limitations of these assumptions would strengthen the paper.\n\nThe paper primarily focuses on fine-tuning and masking-based methods. While it provides a good overview of related work, a more direct comparison with other approximate unlearning techniques like gradient ascent, influence unlearning, in the experimental section would provide a more complete picture of RBM's performance.", "broader_impact_concerns": "Unlearning is often motivated by ethical concerns (copyright, bias, etc.) in the training data. Overselling the capability of models to unlearn, could lead to broader impact concerns. There is no Broader Impact Statement section in the paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)