Factor Decorrelation Enhanced Data Removal from Deep Predictive Models
Authors: Wenhao Yang
Lin Li
Xiaohui Tao
Kaize Shi
Venue: NeurIPS 2025 poster ()
Keywords: Factor decorrelation
Data removal
Out-of-distribution performance
Deep predictive models
Optimization
Primary area: optimization
TL;DR: We propose a data removal method using factor decorrelation and loss perturbation to boost model robustness and accuracy, achieving strong performance on five benchmarks even under significant distribution shifts.

Abstract: The imperative of user privacy protection and regulatory compliance necessitates sensitive data removal in model training, yet this process often induces distributional shifts that undermine model performance-particularly in out-of-distribution (OOD) scenarios. We propose a novel data removal approach that enhances deep predictive models through factor decorrelation and loss perturbation. Our approach introduces: (1) a discriminative-preserving factor decorrelation module employing dynamic adaptive weight adjustment and iterative representation updating to reduce feature redundancy and minimize inter-feature correlations. (2) a smoothed data removal mechanism with loss perturbation that creates information-theoretic safeguards against data leakage during removal operations. Extensive experiments on five benchmark datasets show that our approach outperforms other baselines and consistently achieves high predictive accuracy and robustness even under significant distribution shifts. The results highlight its superior efficiency and adaptability in both in-distribution and out-of-distribution scenarios.

Factor Decorrelation Enhanced Data Removal from Deep Predictive Models
Wenhao Yang
Lin L Wuhan University of Technology, China cathylilin@whut, edu. en
Wuhan University of Technology. China 3424710uhur edu. en
Xiaohui Tao
Kaize Shi
University of Southern Queensland Queensland, Australia Xiaohui. Tao@unisq edu
University of Southern Qucensland Queensland, Australia Kaize Shi Shi@unisq edu.
Abstract
The imperative of user privacy protection and regulatory compliance necessitates sensitive data removal in model training, yet this process often induces distribu- tional shifts that undermine model performance-particn particcllly in out-of-distribution (OOD) scenarios. To address this issue we propose a novel data removal ap- proach that enhances deep predictive models through factor decorrelation and loss perturbation. Our approach introduces: (1) a ceerriinaaive-preeevviii factor decorrelation module employing dynamic adaptive weight adjustment and iterative representation updating to reduce feature redundancy and minimize inter-feature correlations. (2) smoothed data removal mechanism with loss perturbation that creates information-theoreiicc safeguards against data leakage during removal oper- ations. Extensive experiments on five benchmark datasets show that our approach outperforms other baselines and consistently achieves high predictive accuracy and robustness even under significant distribution shifts. The results highlight its superior efficiency and adaptability in both in-distribution and out-of- out-of-distribution scenarios.
Introduction
Removing specific data in the machine learning model training process is crucial to protect user privacy and regulatory compliance [I 2 3. For example,
of e-commerce platforms may invoke data deletion rights for product reviews that have been incorporated into the training corpus of recommendation models. Satisfying such requests entails removing the associated entries from front-end systems while also ensuring that the date's influence is purged from the model's internal representations and parameter space. In addition, financial clients can request the removal of transaction histories or loan application records that have contributed to the training of credit scoring models. These scenarios highlight that data removal requests distributed in widely scenarios with domain-specing regulatory and operational constraints. Furthermore, such removals can induce shifts the underlying data distribution, while retraining the overall model for each of the specific cases impractical since the computational costs [ 5. Therefore, generalizable approaches to data removal are essential for adapting to varied and evolving deletion demands.
The primary challenge confronting data removal methodologies lies in the inadequate exploration and adaptation to -iit----iittiittioon (OOD) data scenarios 76. Existing data removal mechanisms
Corresponding author. Email: cathylilin@ @ whut.edu.cn
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

predominantly rely on muddient-based updates and parameter fine-tuning, assuming that the data distribution similar before and after removal, which inherently limits their robustness. As data distributions may evolve dynamically across temporal and contextual dimensions with continuous data removal requests, the intrinsic correlation between feature representations and corresponding labels transforms. These distributional shifts weaken the effectiveness of existing forgetting mechanisms, reduce removal accuracy, and decrease generalization ability to unseen data. Consequently, existing removal mechanisms struggle to maintain model performance under dynamic scenarios.
Feature dimensionality reduction serves as an effective decorrelation strategy in OOD scenarios and has been widely adopted in predictive modelling techniques, including principal component analysis (PCA)
g 9 clustering- based approaches [10]. and kernel-based mappings [.. For instance, Stablenet 12 employs Random Fourier features to achieve spatial transformation for classification under OOD conditions. A core challenge lies in integrating dimensionality reduction with existing data removal strategies to design parameter update algorithms to maintain the balance between model aceuracy and computational efficiency. While dimensionality reduction reshapes the representation space, it may also discard informative and discriminative features dimensionality decreases. Loss functions without appropriate adaptation may lead to gradient directions that diverge from the true optimization objective, introducing training instability and degrading generalization performance.
To address the aforementioned challenge, we propose DecoRemoval, a data removal method that avoids retraining under OOD scenarios. In such settings, we introduce a discriminative maintenance factor decorrelation module and use a spatial mapping strategy to efficiently reduce feature dimension- ality with linear computational complexity. This transformation is based on the Fourier transform of a kernel function, thereby reducing feature redundancy and promoting factor decorrelation. DecoRe- moval maintains feature weights through iterative gradient updates, which accelerates convergence and enhances robustness without assuming a fixed data distribution. To further improve the safety and reliability of the removal process, we introduce random linear perturbation module for smoothed data removal. This perturbation serves as a regularizer in the parameter space, smoothing the solution landscape of the objective function. As a result, it enables accurate approximation of retraining effects via localized parameter adjustments. Compared with several baselines, the proposed DecoRemoval achieves the best performance in balancing accuracy and efficiency in data removal scenarios. The contributions of this paper can be summarized as follows:
We propose RecoRem discriminative- prreeeeriiiiee--rerviinng factor decorrelation method that integrates feature dimensionality reduction with data removal, which dynamically adjusts feature weights to balance removal precision and computational efficiency.
We design a smoothed data removal mechanism incorporating Loss Perturbation module, which introduces linear interference to protect sensitive information while preserving model stability during the removal process.
We conduct extensive experiments on standard benchmarks, showing that DecoRemoval achieves competitive predictive performance, robust generalization, and high efficiency under significant distributional shifts.
2
Related Work
2.1 Machine Unlearning
Machine unlearning has emerged as a pivotal area of research in response to growing privacy concerns and regulatory mandates [13 14, 15 I6]. This field focuses on developing methodologies that enable machine learning models to effectively remove the influence of specific data points without necessitating complete retraining 18]
As concerns about data privacy and regulatory compliance continue to grow, the ability of machine learning models to "forget" specific data points has emerged as a key area of research. Machine forgetting aims to remove the influence of individual data without retraining the entire model [19]. Early approaches include using gradient vectors or summary layers to isolate and mitigate data influence. Existing methods for forgetting in deep neural networks can broadly categorized into two groups: retraining-basee and retraining-free approaches. Retraining-based methods involve re-optimizing the model after data removal, while retraining-free methods avoid this by estimating the sensitivity of model parameters 200 21. These methods often rely approximations using

---

the Fisher information matrix or the Hessian matrix, as seen in early techniques such as Certified
Removal and Optimal Brain Damage [22 23
major challenge in this space has been adapting these forgetting methods to the complex, nonconvex landscape of deep neural networks [24 25]. To address this, Zhang et al. (2024) extend certified unlearning to deep models by bounding the error introduced by Newton update, enabling scalable and theoretically grounded forgetting nonconvex settings [26). Building on this direction, Foster et
al. (2024) introduce Selective Synaptic Dampening, a method that identifies parameters most relevant to the forget set using Fisher information and proportionally reduces their impact [27]. This allows the model to unlearn specific data while maintaining performance on the remaining dataset.
However, in the dynamic environment, data distribution will evolve over time. The existing machine unlearning methods face serious limitations when dealing with scenarios where data distribution is changing. Our method introduces the feature decoupling module into deep neural networks, providing a balance between efficiency and adaptability in both in distribution and out of distribution settings.
2.2 Certified Data Removal
Certified data removal methods allow models to "forget" specific data points while maintaining statis- tical equivalence to models trained without the removed data [28] 29 30 31]. The requirement for data removal speed in practical application scenarios of machine learning cannot be ignored. Certified removal stands out for providing a favorable balance between removal speed and accuracy [13). ensure removal accuracy to certain extent while maintaining extremely high practical efficiency, making it a leading SOTA method current research literature.
Certified data removal typically adjusts model parameters by removing residual influences of removed samples, often through gradient-based updates and calibrated noise injection [32 33 Marchant et al. [14 pioneered a verification framework for unlearning by analyzing the Hessian matrix of training data and gradients associated with removable samples. Their method triggers retraining if theoretical error bounds exceed predefined thresholds. Subsequent work by Neel et al. [T5]
introduced regularized and distributed gradient descent variants, providing formal guarantees on model indistinguishabiity and accuracy for weakly convex loss functions. Guo et al. [13 advanced these principles for linear classifiers, delivering practical algorithms with theoretical rigor.
Our work restricts the scope to the more mature unlearning area of classification tasks. Based on certified removal, we introduce feature decoupling and loss perturbation modules to enable good approximation of retraining prediction performance after sample removal through localized updates.
3 Factor Decorrelation Enhanced Data Removal
In this section, we will detail the design of our DecoRemoval framework illustrated in Figure DecoRemoval include two main modules: (I) Discriminatiie-preseee prreeirvineeeeiiggggge factor decorrelation by using random Fourier features to achieve spatial mapping and perform dimensionality reduction on input features (Section 3.2); (2) Smoothed data removal by integrating the random linear perturbation loss into unlearning training process to ensure privacy and security (Section 3.3). Moreover, we will integrate the core steps of 3.2 and 3.3 and introduce the main process of the DecoRemoval algorithm (Section 3.4). Next, we will explain them one by one.
3.1 Definitions
Let X Raad denote a dataset with n samples and d features, and let A be learning algorithm trained on X. Factor Decorrelation refers to the process of reducing statistical dependencies (e.g., correlation) among features in X, with the objective of transforming it into a decorrelated representation X' that preserves essential information for learning. As the correlations between features affect or even impair the model prediction, several works have focused on remove such correlation in the training process such as Random Fourier Features(RFF) [34.. RFF is used to approximate kernel functions and induce decorrelation by mapping the data to higher-dimensien space. Given a kernel k(x, y). RFF provides feature
Definition: Factor Decorrelation 12:
3

---

Dimensionality Reduction based Feature Mapping
Depge
Add Noise
>>>
Intia Festures
Rasdoon Fourer Features
Netwerkk
Festures
Cradeet
Disribudion Change after Romoval
Model
Clp
Gredient
1 8 8 8 8 8
Fir
3 3 3 3 3 3
Li Monontun
Len
Importedd Pesture
Adagiive Peature
Weight
Iutkant))) Y (klontus)
Out of Distribution
Inporteed Wrighe
Adapthiv
Smoothed Removal
Weight
Egech
Removal Request
Discriminative Presserni
Decorrelation
Unlearned Predictive Model
Factor decorrelation enhanced data removal. Overview of the DecoRemoval framework. It Preserving Factor Decorrelation based on Random Fourier Features for spatial mapping and dimensionality reduction; (2) Smoothed Data removal through random linear perturbation loss integrated into the unlearning training process.
Figure 1:
consists of two main modules: (I) Discriminative Decermminatvee-Peeervinngg
mapping ((x) such that the dot product (6(x), o(v)) = k(r, v). The transformation is given by:
(xx)
[cos [Cos(W Wa: b)].
where W [Rdxd is a random matrix, b E Bd is bias term, and d is the number of random Fourier features. This transformation is designed to decorrelate the original features by embedding them in higher-dimensional space.
Let be a training dataset and be a learning algorithm trained D. Ronge(A) is the value range of A. A data-removal mechanism M is applied to A(D), and we say that the removal mechanism M performs e-certified removal (c-CR) for learning algorithm A if. for all S Range(A) ()) and x. E D, the following condition holds:
Definition: Certified Removal [13]
e's P(M(A(D), D, E P(A(D x) S)
Se
The definition ensures that removing a single data point from the dataset D will not affect the model's predictions by more than an exponential factor of e, preserving the model's stability.
3.2 Discriminative-Preseeiii Factor Decorrelation
To deal with OOD, we propose a diseriminative-presevye factor decorrelation module that integrates Random Fourier Features (RFF) [12 into the neural network architecture. This module aims to decorrelate input features while preserving their daass-iicrrminativee structure, thereby promoting stable and generalizable learning. Specifically, input features first projected into higher- dimensional randomized feature space via an RFF-based transformation. This mapping reduces redundancy and statistical dependency among features, resulting in smoother optimization landscape for subsequent layers.
While RFF aids in feature decorrelation. it may also disperse discriminative information across dimensions. Applying standard dimensionality reduction or naive loss functions without adapting to the transformed structure risks misaligning gradient directions with the true task objective. Such mismatch can destabilize training and impair generalization. To address this, we explicitly balances feature decorrelation with discriminative preservation. By aligning the transformation process with task-aware loss design, our approach maintains effective learning dynamics and avoids representation collapse.
Random Fourier Feature Mapping: Let the input feature vector for the i-th sample be denoted by X, E p The goal is to map X; into high-dimensional feature space using the Random Fourier

---

Feature transformation. This transformation is based on the Fourier transform of a kernel function and is defined follows:
Z, (w + Φ), w~ (O, I), Φ~ Uniform(O, 2n),
where E IR is sampled from standard normal distribution, and Φ E [0,2x] is sampled uniformly. The resulting vector 2, E I is the transformed feature for the i-th sample. By utilizing this RFF mapping,
approximate the kernel function k(X, in feature space without directly computing it, enabling the use of linear models in a high-dimensional feature space.
Feature Decorrelation via Sample Weighting: To further eliminate the correlation between the transformed features, we employ a sample weighting strategy that minimizes the dependence between features. Let : and Z:, represent the i-th and j-th feature vectors of the transformed input. The goal is to reduce the statistical dependence between all pairs of features the transformed space.
To achieve this, we utilize hypothesis testing statistics based on the cross-covariance between random variables. Let us define the vross-covariance operator EAB between two random variables A and B, with corresponding kernel functions A and kB, follows:
(h EAshB) = EAB(RA ( B)] -
(4)
where AA H. and h E Hg are elements of the Reproducing Kernel Hilbert Spaces (RKHS) corresponding the random variables A and B. The independence of the random variables and / is indicated by the condition:
EAB 0 0 + ALB B.
(5)
In our case, use the cross-covariance between the transformed features Z and Z measure their dependence. The partial cross-covariance matrix AB can be estimated as follows:
T
ε εAB n
u( u(Zi) - - Σ ul
(2))
(i)
1 n 0
Σ j=1
U(2))
(6)
where and v are the RFF transformations applied to the features Z; and Z,, respectively. The Frobenius norm of this matrix is used as measure of the dependence between features:
LAB |B
To further reduce feature dependence, apply sample weighting. Let denote the sample weight for the i-th sample. The weighted partial cross-covariance matrix is computed as:
E ABW
Σ
w;u(Zi)
w.
1 Σ w
(8)
Optimization of Sample Weights: The optimal sample weights w" determined by minimizing the total dependence between all pairs of features. The optimization problem for the weights is formulated as:
w = arg min
Σ
E2,
WEAn
ISicismz
where A, {w R
Li=1 W. = n) ensures that the sample weights are positive and sum to
3.3 Smoothed Data Removal
To enhance the data removal mechanism with generalization across distributions, we propose smoothed data removal module based on random linear perturbations. Specifically, we inject ran- domized noise into the training loss, which obfuscates gradient signals associated with removed or irrelevant samples. This smoothing effect suppresses sharp updates caused by individual data points, minimizing their influence on model predictions. As result, the model becomes less sensitive the removed data, reducing the risk of information Icakage while maintaining stable learning behavior.
5

---

Loss Perturbation for Data Removal: To ensure that the removal of data does not inadvertently leak information about the removed samples, we begin by applying loss perturbation technique at the training stage. This involves perturbing the loss function by adding a random linear term:
(pwaffi D)
Σ L (wayzin + + waf
(10)
where Welf E R& denotes the weight vector of the linear classifier (distinct from the sample weights used for decorrelation), and b TRd is random vector sampled from prescribed distribution (e.g., Gaussian or uniform). The addition of Walf serves to inject controlled stochasticity into the optimization process, thereby obscuring potential gradient signals associated with specific training instances. This perturbation mitigates the risk of overfitting and strengthens the model's robustness to sample removal under removal guarantees.
Linear Authentication Removal: After the loss perturbation, we proceed to the linear authentication removal step. To perform linear authentication removal, the deep learning network is split into two parts: the feature extraction layer parameters Westr and the linear classification layer parameters waf- This separation allows rewrite the loss function in terms of the linear classifier:
Liwafi D)
Σ L (W (wo xx))), yi) 1m
(11)
where was = A(D) = arg minwa, L(way;; D). We assume that we aim to remove the last training
sample (Mi, yn) from the dataset D, resulting in a modified dataset D' D \ (In, Mv).
To remove the sample (In: Vn), we first compute the gradient of the loss function at (Ins Un) and the Hessian of L(: D') at way:
A = VL (In, Un)) Hway LWWaas D')
(12)
We then apply the Newton update removal mechanism M as follows:
" M(waf: D, (Fn. == was + H wal
A
(13)
This update H A represents the influence function of the removed training sample on the vector The training process of our DecorRemoval is described in Appendix.A.
Robustness of Removal Under Perturbation. To ensure the proposed removal mechanism remains valid under the perturbed loss, we analyze its impact on the gradient and Hessian. The perturbed loss given by:
Lp www D) = Σ Mw + Welf
(14)
where b E Rd is random vector independent of individual samples. This linear term introduces constant shift in the gradient:
VDD(war) = vL(war) + b,
(15)
but leaves the Hessian unchanged:
2 I (way) = (²I(war))
(16)
As the removal update depends on the Hessian and gradient of the sample to removed, which are both unaffeected by b, the update
w J
was + H LVL(Way (En, yn))
(17)
remains valid. Hence, our removal mechanism is robust to the proposed linear perturbation, and the specific proof process is detailed in Appendix B.

---

Experiments
4.1 Datasets and Evaluation Metrics
Our approach evaluated five widely used datasets spanning multiple data modalities, including image, text, and structured features. Following the setup in Certified Removal [13] we consider three public datasets for classification tasks: MNIST (35]. CIFAR-I0 [36], and SST-2 [37]. MNIST consists of grayscale images of handwritten digits, where digits 3 and 8 are selected as in-distribution classes and the remaining digits are treated out-of-distribution (OOD) data. CIFAR-I0 contains 60,000 color images (32x32) evenly distributed across 10 object categories and serves as a standard benchmark for evaluating image classification methods. SST-2 is a binary sentiment analysis dataset derived from movic reviews, commonly used in text classification and language understanding tasks.
To assess the algorithm's s applicability privacy-sensitive structured data, we also include two social survey datasets: the 2015 China General Social Survey (CGSS) and the 2018 European Social Survey (ESS). Both datasets contain multi-label annotations related to self-reported happiness levels five-point ordinal scale [38 39..
In the context of unlearning, it is essential to evaluate performance from three key aspects: utility, efficiency, and privacy protection. Given that certified unlearning methods provide theoretical guarantees privacy, our experiments primarily focus reporting utiliity and efficiency. We report accuracy (ACC) and weighted average Fl score as evaluation metrics to capture both overall classification performance and class imbalance sensitivity across diverse data types.
4.2 Experimental Setting
Following certified removal, we split the dataset into training, validation, and testing sets with a ratio of 7:1:1. Both training and validation sets consist solely of correctly labeled data to ensure standard supervised learning conditions. We train models independently on each category (e.g., training with data from class A), and then sample 10% of instances from other categories (e.g., class B). For testing, if these testy samples are predicted as "Not A", it means the unleamed model is correct. This creates an extreme testing scenario where semantic content mismatches the assigned label, enabling evaluation of the model's robustness against mislabeling conceptual confusion.
In the experiment, the data unlearning baselines include Certified Removal(CR)
[3 SISA (S
shards) 3 DP-SGI [40], Certified Unlearning (CU) [n] SSD [27]. Both CR and DP-SGD provide strong theoretical guarantees under the framework of differential privacy. CU offers model- agnostic approximate unlearning strategies with soundness certificates and can be extended to deep neural networks [26]. SSD) leverages the Fisher information matrix for parameter updates and represents the current state-of-the-art in certified removal (27). In this paper, the source code provided by the baselines used to fine-tune the parameters and obtain the optimal values to represent the experimental results of the baselines. For baselines which source code was not provided, this study reproduced the model design based on PoTorch framework. All methods set MLP as the backbone for all deep prediction models, ans are trained using batch size of d = 50 and total of 7 = 20 training epochs. For fair comparison, we fix the standard deviation parameters in DecoRemoval to 5 10-3 and std 1, and consistent optimization schedule with num,teps 100 across all experiments. Under these conditions, we subsequently applied the removal mechanism to each group separately. The related experiments in this study conducted on four NVIDIA 4090D GPUs.
The time consumption for our DecoRemoval algorithm (DR) to remove data mainly comes from the computation of adaptive weights and the update operation of different features. We evaluate the performance of DecoRemoval algorithm across multiple datasets and varying data remove scales (1000, 3000, and 10000 samples). The full retraining from scratch (Retrain) is treated as the upper bound for accuracy, while Certified Removal serves as the bascline for efficiency comparison [3] Our main codes and datasets are available at https://github.co wttpiihuubboo/WWU-DDEEDE.
4.3 Unlearning Performance
Evaluated on five diverse datasets, DecoRemoval consistently achieves near-retraining performance in both accuracy and F1 score across varying removal scales. Unlike prior methods that focus primarily on privacy, our approach addresses feature correlation shifts via spatial mapping and randomized loss
7

---

Table I: Comparison of ACC (%) and Fl scores across different methods and removed sample sizes (The closer to Retrain, the better)
Retrain
CR 13
SISA
3
DP-SGD
40
SSD 27
CU 26
DR (Ours)
Dataset
Samples
ACC
F1
ACC
F1
ACC
F1
ACC
F1
ACC
ACC
ACC
F1
1000
51.753
0.505
43.132
0.394 43.635
0.405
45.783
0,440
45.452
0,458
47.345
0,458
48.973
0.482
MNIST
3000
51.351
0.498
42.213
0.391
43.455
0.401
45.342
0.438
45.241
0.455
46.943
0.455
48.653
0.478
10000
51.016
0.495
41.872
0.390
42.955
0.398
44.873
0.432
45.031
0.450
46.532
0.450
48.340
0.473
1000
50.762
0.501 43.086
0.392
43.214
0.401
45.301
0.436
45.062
0,452
46
86.842
0,452
48.563
0.478
CIFAR-10
3000
50.459
0.496 42.293
0.391 42,942 0.396 44.839
0,433
44.723
0,448
46.521
0.449
48.141
0.473
10000
50.011
0.491
41.763
0.389
42.512
0.392
44.371
0,428
44.513
0,443
46.
0,444
47.832
0.469
1000
91.764
0.843 89.705
0.808 89.975
0.817 90.452
0.825
89.983
0.818
89.942
0.813
90.451
0.827
SST-2
3000
91.545
0.840 89.651
0.801 89.760
0.814 90.356
0.820
89.865
0.816
89.765
0.808
90.387 0.825
10000
91.142
0.839 89,478
0.796
89.653
0.809
90.101
0.816
89.873
0.816
89.673
0.805
90.375 0.821
1000
55.432
0.540 48.608
0.410 48.635
0.420
50.473
0.450
50.147
014466
51.341
0.490
54.973 0.520
ESS
3000
55.351
0.540
48.412
0.400 48.455
0.410 50.012
0.440
50.007
0.4479
51.153
0,480
54.852 0.510
10000
55.236
0,530
48.402
0.390 48.435
0.410
49.673
0,430
18.
0.478
50.871
0.470
54.640 0.510
1000
51.602
0.515 41.239
0.465 43.516
0.472 46.756
0.487
47.765
0.475
48.765
0.485
50.824
0.501
CGSS
3000
51.324
0.506
40.738
0.458
43.016
0.469
46.252
0.482
47.313
0,474
48.210
0,480
50.482 0.496
10000
50.975
0.498 40.145 0.434 42.745
0.465 45.954
0.473
47.152
0.471
47.851
0.477 50.104 0.495
perturbation, ensuring both utility and robustness. Compared to recent baselines such as SSD and certified unlearning, RecoRemoval demonstrates stronger generalization, especially under large-scale deletions and noisy, high-dimensional data.
RecoRemoval achieved the best performance in out of distribution scenarios across five datasets. As shown in Table ref Table: merged, DecoRemoval consistently achieved near-retraining accuracy and Fl score across five datasets and removal sizes, surpassing all existing Sota data removal models under disttribbuiinnnn settings. On the ESS and CGSS datasets, which feature noisy and highly correlated survey data, method achieves 54.9% and 50.8% accuracy after removing 1000 samples, with minimal degradation compared to full retraining (55.4% and 51.6%). In SST-2, DecoRemoval maintains over 90.3% accuracy across all remove scales, outperforming DecoRemoval by approximately percentage point on both ACC and F1 metrics. Especially in image dataset scenes, DecoRemoval achieved significant improvement under out of distribution settings. This scenario is the main research and application goal of the current data removal mechanism.
Compared to traditional data removal models that mainly focus on privacy processing, DecoRe- moval performs better. The biggest problem with data removal under out of distribution settings is the correlation between features that affects the distribution of data. The existing traditional data removal models mainly focus on privacy protection issues during the data removal process, using methods such differential privacy to ensure model stability and security. However, when there are out of distribution changes in the data scene, these methods lack processing of the correlation between features, resulting in poor model performance. DecoRemoval identified the complexity of features in this scenario, achieved feature dimensionality reduction through spatial mapping, and ensured the privacy of the removal process by adding random loss perturbations, thus achieving optimal performance in ACC and Fl scores in OOD scenarios.
Greater robustness compared to SOTA. Compared with the latest methods such as SSD updated with Fisher matrix and optimized Certified Unlearning, DecoRemoval utilizes the advantage of spatial dimensionality reduction in feature correlation processing and exhibits stronger robustness in large-scale data deletion. For example, on the CGSS dataset with 10000 removed samples, our method achieved an Fl score of 0.495, while Zhang and Foster's methods only scored 0.477 and 0.471, respectively. This pattern is applicable to all datasets, indicating that our method has stronger generalization ability and stability.
4.4 Efficiency Analysis
In addition to unlearning fidelity, computational efficiency a critical factor for practical deployment. While full retraining (Retrain) achieves optimal activity performance, it requires model reinitialization
8

---

Table 2: Comparison running Time(s) for different removal methods (the closer Certified Removal, the better)
Samples
Dataset
Retrain
CR 13
SISA 3
DP-SGD 40
CU 26
SSD
27
DR
MNIST
4643.100
21.312
1923,430
38.710
33.134
32.541
30,430
SST-2
61.500
0.074
24.670
0.124
0.105
0.095
0.097
1000
ESS
1539.100
7.420
648 448.400
12.800
12.458
12.127
11,500
CGSS
615.690
8.450
362.400
15.346
14.541
14.377
14.353
MNIST
11432.500
62.425
4321.200
102.510
91.329
90.786
81,420
SST-2
178.100
0.204
78.430
0,401
0.325
0.309
0.315
3000
ESS
3142.200
22.120
1448.200
40,500
38.718
37.812
35,200
CGSS
1923.500
25.630
983.200
47.545
45.341
44.749
43.345
MNIST
43123.500
190.342
13214.400
310.120
287 490
276,710
268.420 420
SST-2
598.400
0.715
236.400
1,030
0,904
0.891
0.894
10000
ESS 14532.5000
78,400
6534,400
131.100
127.760
123.710
121.400 400
COSS
5893.400
81.423
3418.900
164.600
158.860
147.230
141,700
and complete retraining the original dataset, making it computationally impractical for frequent large-scale data removal scenarios. In contrast, DecoRemoval delivers strong unlearning performance significantly lower computational cost. Unlike Retrain, which revisits the entire training set, DecoRemoval performs lightweight fine-tuning procedure that specifically targets the removal- induced dominant feature directions, enabling efficiently handle removal scales ranging from 1,000 to 10,000 samples without full model retraining, Furthermore, compared to existing methods, DecoRemoval achieves better performance with lower overhead. Certified Removal and SISA rely on ensemble models or shard-based training pipelines, which incur significant complexity and computational burden [3]. While DP-SGD offers built-in privacy guarantees, it injects substantial noise during training, resulting in lower post-removal accuracy despite its relative efficiency [40]. Finally, DecoRemoval strikes more favorable balance between efficiency and unlearning quality than recent approaches proposed by Certified Unlearning (2024) and SSD (2024) [26] 27). Across all
settings, it consistently maintains performance close to the retraining upper bound while requiring far fewer computational resources, making it particularly suitable for real-world, large-scale, or streaming environments where fast and effective unlearning is essential.
4.5 Empirical Evaluation of Privacy Protection via Membership Inference Attacks
To further validate the
capabilities of DecoRemoval in the context of compliant machine learning (e.g., GDPR, CCPA), evaluate its robustness against Membership Inference Atlacks (MIAs)-a canonical threat model where an adversary attempts to determine whether a specific data sample was used in model training. Strong resistance MIAs indicates effective data removal and reduced risk of information leakage.
Our evaluation is conducted under realistic forgetting scenario with privacy budget of 1. We compare DecoRemoval against several baselines. We assess MIA success rates across three model architectures (MLP, LSTM, Transformer) and two real-world datasets (ESS, CGSS), while also reporting accuracy and F1-score to evaluate utility preservation.
The results are summarized in Table Key observations include:
DecoRemoval consistently achieves lower MIA success rates than No-DP, demonstrating effective privacy protection after data removal.
It outperforms SSD and CR across most settings, especially on Transformers, with better utility.
On CGSS with LSTM, it achieves the lowest attack rate (50.2%) while maintaining higher accuracy than CR.
Compared to DP-SGD, DecoRemoval offers significantly better utility at comparable privacy levels.

---

Table Performance comparison of DecoRemoval and baselines ESS and CGSS datasets across
different architectures. Metrics: Accuracy (%)) Fi-score (%), and MIA success rate (%). Lower
attack rate better.
Backbone
Method
ESS ACC
ESS F1
ESS Attack
CGSS ACC
CGSS f
Attack
Retrain
62.5
60.4
61.4
DP-SGD
56.8
54.4
56.8
51.7
43.5
51.8
MLP
SSD
59.0
56.9
$7.3
53.2
45.2
52.5
59.8
45,9
52.4
DecnRemoval
61.55
59.6
56.8
56.3
45.7
52.1
Retrain
65.8
65.2
65.3
62.5
56.4
59.8
DP-SGD
$2.5
48.5
$6.5
52.1
49.1
51.6
SSD
53.2
53.2
50.9
CR
57.1
$3.5
55,9
55.4
54.1
50.4
DecoRemoval
58.9
55.4
56.7
56.8
55.1
50,2
Retrain
65.6
65.0
69.2
61.5
$5.6
63.3
DP-SGD
52.5
62.0
59.1
Transformer
SSD
57.3
55.8
62.3
54.8
52.7
58.8
CR
$7.7
56.1
60.5
55.4
52.9
57.4
DecoRemoval
56.3
56.2
52.8
58.1
These results confirm that DecoRemoval not only stabilizes model distribution after data removal but also provides strong empirical privacy guarantees, making suitable in srivacy-sensitive applications.
4.6 Key Parameter Study
In Figure 2] we map the MINST dataset to a two-dimensional space for visualization after dimen- sionality reduction. It can be observed that after data removal, the distribution of the dataset and the position of the center point have undergone significant changes, which is consistent with our initial hypothesis about the impact of data removal. Appendix C shows DecoRemoval's performance depends on hidden layer size and RFF dimension, with optimal capacity ensuring good generalization.
IMAP Dstaoed Porero
traniig teclach taliar tata
Samp
Figure 2: Results of MNIST dataset showing medoids before and after data removal
5
Conclusions and Future work
To tackle privacy preservation and out-of-distribution (OOD) challenges in predictive tasks, we inte- grate snorrininative-prseevvii factor decorrelation with smoothed data removal. Our DecoRemoval mechanism enables efficient data unlearning while maintaining compliance with privacy regulations. The proposed method alleviates accuracy degradation commonly proaches under distribution shifts. Empirical results confirm its effectiveness and robustness OOD scenarios. Future work will focus applying this algorithm various application domains that require robust machine unlearning capabilities, and how to work with current popular large models.
in traditional unlcarning ap-
Acknowledge
This work is partially supported by NSFC, China (No.62276196.. The authors also gratefully appreciate the anonymous reviewers for their valuable comments and constructive suggestions that greatly helped to improve the quality of the paper.
10

---

References
[1] Ayush K. Tarun, Vikram Chundawat, Murari Mandal, and Mohan S. Kankanhalli. Fast yet effective machine unlearning. IEEE Trans. Neural Networks Learn. Syst. 1346-:33046533555 2024.
[2] Heng Xu, Tianqing Zhu, Lefeng Zhang. Wanlei Zhou, and Philip S. Yu. Machine unlearning: A survey, ACM Comput. Surv., 56(1):9:1-9:36 2024.
[3] Youyang Qu, Xin Yuan, Ming Ding, Wei Ni, Thierry Rakotoarivelo, and David B. Smith. Learn to unlearn: Insights into machine unlearning. Computer, 57(3):79-90, 2024.
[4] Haonan Yan, Xiaoguang Li, Ziyao Guo, Hui Li, Fenghua Li, and Xiaodong Lin. ARCANE: an efficient architecture for exact machine unlearning. In Luc De Raedt, editor. Proceedings of the Thirty-Firte First International Joint Conference on Artificial Intelligence, IJCAI 2022, Viemna, Austria, 23-29 July 2022, pages 4006 4006-4013. ijeai.org, 2022.
[5] Korbinian Koch and Mareus Soll. No matter how you slice it: Machine unlearning with SISA comes at the expense of minority classes. In 2023 IEEE Conference on Secure and Trustworthy Machine Learning, SaTML 2023, Raleigh, NC, USA, February 8-10, 2023, pages 622-637. IEEE, 2023.
[6] Daniel Gedon, Antônio H. Ribeiro, Niklas Wahlström, and Thomas B. Schön. Invertible kernel PCA with random fourier features. IEEE Signal Process. Lett., 30:563-567, 2023.
[7] Kun Fang, Qinghua Tao, Kexin Lv, Mingzhen He, Xiaolin Huang, and Jie Yang. Kernel PCA for cat-of-distribution detection. In Amir Globersons, Lester Mackey, Danielle Belgrave. Angela Fan, UIrich Paquet, Jakub M. Tomezak, and Cheng Zhang, editors, Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurlPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024.
[8] Jonathon Shlens. A tutorial on principal component analysis. CoRR, abs/1404. 100, 2014.
[9]
HanQin Cai, Jialin Liu, and Wotao Yin. Learned robust PCA: A scalable deep unfolding approach for high-dimensional outlier detection. In Marc' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference Neural Information Processing Systems 1, NeurlPS 2021, December 6-14, 2021, virtual, pages 16977-16989, 2021.
[10] Yusuke Endo and Koujin Takeda. LI-regularized ica: novel method for analysis of task-related fmri data. Neural Computation, 36(11)22500-277, 2024.
[11] Youtian Du, Xue Wang, Yunbo Cui, Hang Wang, and Chang Su. Kernel-based mixture mapping for image and text association. IEEE Trans. Multim., 22(2):365-31) 379, 2020.
[12] Xingxuan Zhang, Peng Cui, Renzhe Xu, Linjun Zhou, Yue He, and Zheyan Shen. Deep stable learning for diit-ii-tttii generalization. IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pages 5372-5382. Computer Vision Foundation IEEE, 2021.
[13] Chuan Guo, Tom Goldstein, Awni Y. Hannun, and Laurens der Maaten. Certified data removal from machine learning models. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 3832-3842. PMLR, 2020.
[14] Neil G. Marchant, Benjamin I. Rubinstein, and Scott Alfeld. Hard to forget: Poisoning attacks on certified machine unlearning. In Thirty-Sixth AAAI Conference Artificial Intelligence, AAAI 2022, Phirty-Fourth Conference on Innovative Applications of Artificial Intelligence, IAAI 2022, The Twelveth Symposium on Educational Advances in Artificial Intelligence, EAAI 2022 Virtual Event, February 22- March 1, 2022, pages 7691-7700. AAAI Press, 2022.
[15] Seth Neel, Aaron Roth, and Saeed Sharifi-Malvajedd methods for machine unlearning. In Vitaly Feldman, Katrina Ligett, and Sivan Sabato, editors, Algorithmic Learning Theory, 16- March 2021, Viruual Conference, Worldwide, volume 132
Descent-to-delete: Gradient-based
of Proceedings of Machine Learning Research, pages 931-962. PMLR, 2021.

---

[16] Guangzhen Yao, Long Zhang, Sandong Zhu, and Miao Qi. Dp-prune: Global optimal strategy for retraining-free pruning of transformer models. In IEEE International Performance, Computing, and Communications Conference, IPCCC 2024, Orlando, FI,, USA, November 22-24, 2024, pages 1-6. IEEE, 2024.
[17] Ga Wu, Masoud Hashemi, and Christopher Srinivasa. PUMA: performance unchanged model augmentation for training data removal. In Thirty-Sixthh AAAI Conference on Artificial Intelli- gence, AAAI 2022, Thirty-
Conference on Innovative Applications of Artificial Intelligence, IAAI 2022, The Twelveth Symposium on Educational Advances in Artificial Intelligence, FAAI 2022 Virtual Event, February 22- March 1, 2022, pages 6672-8682. AAAI Press, 2022.
[18] Chongyu Fan, Jiancheng Liu, Yihua Zhang. Eric Wong, Dennis Wei, and Sijia Liu. Salun: Empowering machine unlearning via gradient-basee based weight saliency in both image classification and generation. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. Openkeview.net, 2024.
[19] Tianqi Chen, Shujian Zhang, and Mingyuan Zhou. Score forgetting distillation: A swift, data-free method for machine unlearning in diffusion models. In The Thirteenth Interna- lional Conference on Learning Representations, ICLR 2025, Singupore, April 24-28, 2025. OpenReview.net, 2025.
[20] Lingzhi Wang, Xingshan Zeng, Jinsong Guo, Kam-Fai Wong, and Georg Gottlob. Selective forgetting: Advancing machine unlearning techniques and evaluation in language models. In Toby Walsh, Julie Shah, and Zico Kolter, editors, AAAI-25, Sponsored by the Association for the Advancement of Artificial Intelligence, February 25- Murch 4, 2025, Philadelphia, PA, USA, pages 843-851. AAAI Press, 2025.
[21] Dasol Choi and Dongbin Na. Distribution-level feature distancing for machine unlearning: Towards better trade-off between model utility and forgetting. In Toby Walsh, Julie Shah, and Zico Kolter, editors, AAAI-25, Sponsored by the Association for the Advancement of Artificial Intelligence, February 25- March 4, 2025, Philadelphia, PA, USA, pages 2536-2544. AAAI Press, 2025.
[22] Chao Liu, Zhiyong Zhang, and Dong Wang. Pruning deep neural networks by optimal brain damage. In Haizhou Li, Helen M. Meng, Bin Ma, Engsiong Chng, and Lei Xie. editors. 15th Annual Conference of the International Speech Communication Association, INTERSPEECH 2014, Singapore, September 14-18, 2014, pages 1092-1095. ISCA, 2014.
[23] Yann LeCun, John S. Denker, and Sara A. Solla. Optimal brain damage. In David S. Touretzky, editor, Advances in Neural Information Processing Systems 2, [NIPS Conference, Denver, Colorado, USA, November 27-30, 1989], pages 598-605. Morgan Kaufmann, 1989.
[24] Antonio Ginart, Melody Y. Guan, Gregory Valiant, and James Zou. Making AI forget you: Data deletion in machine learning. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Flo- rence d' Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8- 14, 2019, Vancouver, BC, Canada, pages 3513-3526, 2019.
[25] Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Eternal sunshine of the spotless net: Selective forgetting deep networks. In 2020 IEEF/CVE Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 9301-9309. Computer Vision Foundation/ / IEEE, 2020.
[26] Binchi Zhang, Yushun Dong. Tianhao Wang, and Jundong Li. Towards certified unlearning for deep neural networks. In Forty-first International Conference on Machine Learning, ICML. 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024.
[27] Jack Foster, Stefan Schoepf, and Alexandra Brintrup. Fast machine unlearning without retraining through selective synaptic dampening. In Michael J. Wooldridge, Jennifer G. Dy, and Sriraam Natarajan, editors, Thirty-Eighll AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty- Sixth Conference on Innovative Applicutions of Artificial Intelligence, IAAI 2024, Fourleenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2014, February 20-27, 2024, Vancouver, Canada, pages 12043-12051 1. AAAI Press, 2024.
12

---

[28] Eli Chien, Haoyu Wang, Ziang Chen, and Pan Li. Certified machine unlearning via noisy stochastic gradient descent. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomezak, and Cheng Zhang, editors, Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurlPS 2024, Vancouver, BC, Canada, December 10- 15, 2024, 2024.
[29] Eli Chien, Haoyu Wang, Ziang Chen, and Pan Li, Langevin unlearning: A new perspective of noisy gradient descent for machine unlearning. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Urrich Paquet, Jakub M. Tomezak, and Cheng Zhang, editors, Advances in Neural Information Processing Systems 38: Annual Conference Neural Information Processing Systems 2024, NeurlPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024.
[30] Zijie Zhang, Yang Zhou, Xin Zhao, Tianshi Che, and Lingjuan Lyu. Prompt certified ma- chine unlearning with randomized gradient smoothing and quantization. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Anmual Conference Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, 2022.
[31] Ziyao Liu, Huanyi Ye, Yu Jiang, Jiyuan Shen, Jiale Guo, Ivan Tjuawinata, and Kwok- Yan Lam. Privacy-preservin federated unlearning with certified client removal. IEEE Trans. Inf. Forensics Secur., 20:3966-399 -978, 2025.
[32] Thanh Trung Huynh, Trong Bang Nguyen, Thanh Toan Nguyen, Phi Le Nguyen, Hongzhi Yin. Quoe Viet Hung Nguyen, and Thanh Tam Nguyen. Certified unlearning for federated recommendation. ACM Trans. Inf. Syst., 43(2):48:1 1-48:99 2025.
[33] Yushun Dong. Binchi Zhang, Zhenyu Lei, Na Zou, and Jundong Li. IDEA: A flexible framework of certified unlearning for graph neural networks. In Ricardo Bacza- Yates and Francesco Bonchi, editors, Proceedings of the 30th ACM SIGKDD Conference Knowledge Discovery and Data Mining, KDD 2024, Barcelona, Spain, August 25-29, 2024, pages 621-630 ACM, 2024.
[34] Zhenyu Liao, Romain Couillet, and Michael W. Mahoney. A random matrix analysis of random fourier features: beyond the gaussian kernel, a precise phase transition, and the corresponding double descent. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tiee Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
[35] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proc. IEEE, 86(1) 86(11):277-2322, 1998.
[36] A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Handbook of Systemic Autoimmune Diseases, 1(4), 2009.
[37] Wenxuan Zhang, Xin Li, Yang Deng, Lidong Bing, and Wai Lam. A survey aspect- based sentiment analysis: Tasks, methods, and challenges. IEEE Trans. Knowl. Data Eng., (((11):1109--1 1038, 2023.
[38] Theresia Ratih Dewi Saputri and Seok- Lee. A study of cross- nats-natiooal differences in happiness factors using machine learning approach. Inf. J. Softw. Eng. Knowl. Eng.. 25(9- 10): 1699-1227 2015.
[39] Zongwen Fan, Jin Gou, and Shaoyuan Weng. A novel fuzzy feature generation approach for happiness prediction. IEEE Trans. Emerg. Top. Comput. Intell., 5(2):1995-1608, 2024.
[40] Martín Abadi, Andy Chu, Ian J. Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Edgar R. Weippl, Stefan Katzenbeisser, Christopher Kruegel, Andrew C. Myers, and Shai Halevi, editors, Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, Vienna, Austria, October 24-28, 2016, pages 308-318. ACM, 2016.
13

---

A DecoRemoval Training Algorithm
In 3.1 and 3.2, we have introduced feature dimensionality reduction methods using kernel based mapping and certified removal mechanism incorporating random loss perturbations. By utilizing the characteristics of random Fourier transform and sample feature weightings are fused through iterative loss fusion, and feature weights and related parameters are updated.So we summarized the overall algorithm flow of DecoRemoval and introduced the data removal mechanism after adding feature dimensionality reduction.
The DecoRemoval A Algorithm aims to minimize feature correlation through the use of Random Fourier Features (RFF) and optimize sample weights efficiently. In the first step, the algorithm prepares the training dataset D = (Z11, v1), .... (In, Mn)} and initializes the neural network's feature extraction weights Wertr and classification layer weights Waf.. The RFF transformation is applied to map the input features into a higher-dimensional space, resulting in transformed feature vectors Z, In the second step, the algorithm calculates the feature dependence between pairs transformed features and optimizes sample weights wy to minimize this dependence, achieving decorrelation. For certified data removal in the third step. the influence of each sample is removed by applying a Newton update rule to the classifier weights. To prevent leakage of information from the removed samples, perturbation term is added the loss function. The final output includes the optimized classifier weights way and the transformed feature vectors Z₁.
Algorithm 1: Factor decorrelation enhanced data removal
Inputs: training dataset D = (()1 ...., (Ir, Yn)], kernel function k neural network with feature extraction layer Westr, and linear classification layer Welf;
Hyperparameters: number of features mz, number of samples
Step 1: Random Fourier Feature Mapping;
for i I,..., nI do
Sample wi ~ (0, d; ~ Uniform(O, 2m);
Z, = RFF uFF__trnsform(X wi, Φx);
Step 2: Discriminative-Prssrre
Factor Decorrelation;
for each pair (Z, Zzz) do
Compute feature dependence: Ly FrobeniusNorm(S.y))>
Apply sample weights w; to minimize feature dependence: w" Optimize Weights(EAAn))
Step 3: Smoothed Data Removal:
for each sample (In, yn) do
Compute gradient: A V LW (In, vn));; Compute hessian: Hw LW D');
Update classifier: War was + 1 way
A;
Add random linear term to the loss: Lp = L T
Return: Optimized classifier parameters waff and transformed feature vectors Z,.
B Proof of Robustness Under Loss Perturbation
In this section, we provide a formal proof that adding a linear perturbation term to the training loss does not affect the correctness of the linear authentication removal mechanism.
B.1 Perturbed Loss Function
We consider a perturbed loss function of the following form:
Lo(was; D)
Σ L ( b Wolf
(18)
where waf E Rd the linear classifier, and E Rd random vector sampled from a fixed distribution (e-g.. Gaussian or uniform). The term b T way introduces controlled randomness to the optimization process.
14

---

B.2 Gradient and Hessian Analysis
Let Longg(uu (Wetf) perturbed loss is:
Ei= Liwayas denote the original loss function. Then, the gradient of the
(Ip((a))) = Lanig (Wers) ) +
(19)
The Hessian the perturbed loss is:
= Long (War)) + D2 wayf) = V Lorig
(20)
since the second derivative of linear term is zero. Thus, the curvature of the loss landscape (captured by the Hessian) remains unchanged by the perturbation.
B.3 Effect on Removal Update
Assume that we wish remove the final sample (In, Vo) from the dataset D, yielding the modified dataset D' = D (Ia, Ba). The Newton based removal update is given by:
wai was + H L (In, va)),
(21)
where waz is the minimizer of the loss (perturbed or unperturbed), and H is the Hessian of the loss over D' evaluated waf.
Under the perturbed loss, we denote the minimizer as w waf. which satisfies:
Lurig (w2f) + + =
-b.
(22)
However, this constant offset in the gradient does not affect the sample-specific gradient L( (I ya)), nor the Hessian H, since both are independent of Therefore, the removal update remains:
Way = wif + (V (Ias Ba)),
(23)
which is structurally identical to the original update formula. As result, the removal mechanism is preserved under loss perturbation.
B.4 Conclusion
The addition linear perturbation term does not interfere with the linear removal update. The gradient shift induced by b constant and does not impact the relative influence of any individual data point. The Hessian remains unchanged, and the Newton update retains its validity. This confirms the robustness of our removal strategy under smoothed loss perturbations.
C In-distribution Setting Experiment
For the in distribution data scenario, we evaluated the accuracy and F1 score of Certified (Cm)val(CR). (e))))moval(DR), Certified Unlearning and SSD on different datasets. This setting enables us to systematically evaluate the robustness of the model to domain differences and its generalization ability when the training distribution remains largely unchanged.
Table 4: Comparison of ACC (%) and Fl scores across different methods under in-distribution setting
MNIST
CIFARIO
SST-2
ESS
CGSS
Method
ACC
F1
ACC
F1
ACC
F1
ACC
F1
ACC
F1
CR
91.974
0.930
90.878
0.931
96.103
0.979
92.863
0.939
92.798
0.921
SSD
92.429
0.940
92.320
0.964
96.982
0.987
93.754
0.943
93.305
0.930
CU
94.213
0.964
93.271
0.946
96.923
0.982
93.671
0.950
93.193
0.945
(Ours)
95.841
0.968
95.034
0.963
97.011
0.988
93.852
0.969
93.860
0.948
15

---

D Unlearning Performance
CEAR I. Apurscy
1F Score
Accuracy and F1 Score.
0.58
These results demonstrate the effectiveness of DecoRemoval, our factor decorrelation-basee data removal strategy. By adaptively reweighting feature dimensions to suppress redundant correlations, DecoRemoval eliminates the need for full retraining,
0.64
ensuring fast and efficient data removal. performs well across various datasets and conditions, providing high forgetting fidelity and accuracy while balancing computational efficiency and rigorous unlearning, making ideal for trivacy-sensttve applications.
042
1000
2010
1000
1330
20:0
10000
Lsarowed Stmppiex
Famber lderccedd bamppes
Figure 3: Removal performance of different num- bers of removed samples in CIFAR-10
Companson
320
Auapees
Efficiency. Overall, our DecoRemoval achieves effective balance between high forgetting fidelity and practical efficiency, es- pecially when facing out of distribution situ- ations as shown in Figure E It can achieve performance close the level of retraining without incurring the huge cost of compre- hensive retraining, and has a significant im- provement in the balance between accuracy and efficiency compared existing data re- moval mechanisms. This makes it highly promising solution for scalable and trustwor- thy machine learning.
8000
Cx 1303
Un ehnaming Time (8)
03443030
230
150
100
CR
DP.600
SSD
Figure 4: Efficiency of different removed numbers CIFAR-10 datasets
E
Key Parameter Study
Our DecoRemoval algorithm adjusts the hidden layer dimension and the random Fourier transform dimension of the neural network under df-iiitribttion settings, and the tested results are reported in terms of Accuracy and Fl score on the datasets, as shown in Figure [ Specifically, when the hidden layer dimension is about 80, the accuracy and F1 score of the two happiness datasets ESS and CGSS, as well as the sentiment text dataset SST-2, reach their maximum values, outperforming other traditional models and approaching the results of retraining. For the image dataset MNIST, the performance reaches its best when the hidden layer dimension is about 400.
Mnist
ESS
CGSS
SST-2
0-400
ALD
AUU
0.58
ADC
0.465
0.52
0.86
0.52
0.54
4GC
0.86
0.475
0.52
0.84
0.400
0.50
088
200
300
820
500
60
40
100
20
60
100
H155ion Layor Jimonsion
H4 (dddon Layon Dimenosoon
H:sder Layer Dimonsson
Hiddon Layer Dimemsion
Figure 5: Results of different hidden layer dimensions in adaptive weighted factor decorrelation
The results indicate that the DecoRemoval algorithm can significantly improve its generalization ability in the presence of out-of-distributed data.
16

---

Effect of RFF Dimensions an ACC and F1 Score
REF dimensions
During training with varying RFF dimensions, observed that both model accuracy and F1 score consistently improved as the dimen- sionality increased, peaking 1000 dimen- sions as shown in Figure(6) Beyond this point, performance gains plateaued, showing min- imal change with further increases. These findings suggest that the DecoRemoval al- gorithm effectively enhances generalization under non-out-of-distributin conditions, par- ticularly when equipped with sufficiently expressive RFF representation.
0.33
0.52
0.9
200
438
650
A20
1200
19:0
1400
Dimensions of Random Fourler Fnat.inss (RFF)
Figure 6: Experimental results ESS in dif-
ferent RFF dimensions
Backbones
Table 5: Comparison of ESS Deletion Efficiency of Different Backbones in Deep Predictive Models (the closer to Retrain, the better)
Retrain
Certified Removal
FD-DR(ours)
Backbones
Time(s) ACC(%)
Time(s)
ACC(%)
Time(s) +
ACC( %
MLP
1539.1
55.4
7.4
48.6
11.5
54.8
LSTM
3583.1
53.9
20.4
48.2
28.6
52.5
Transformer
1956.2
54.1
16.4
48.5
15.4
53.1
G Limitations
Limited Exploration Beyond Feature-Level Decorrelation. This work primarily focuses on mitigating out-of-distribution (OOD) challenges through feature-level factor decorrelation. While effective, it leaves open how this approach interacts with other common OOD handling techniques such data augmentation, adversarial training, ensemble learning. A promising direction for future work is to explore how these methods can be systematically integrated with data removal strategies to enhance both generalization and unlearning robustness.
Trade-off Between Certified Removal and Accuracy.
Although certified removal offers strong theoretical guarantees and is efficient for linear layers, it may be suboptimal in scenarios where high predictive accuracy is critical, such as medical diagnosis or financial forecasting. such cases, privacy strength could be moderately relaxed in favor of more expressive unlearning methods, such as influence function-based -aaed unlearning or fine-tuning-bast -aaed approximate unlearning, to strike a better balance between utility and privacy.
17

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] is generally preferable to "[No]" it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to question, in the justification please point to the section(s) where related material for the question can found.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The theoretical design effectively reduces feature redundancy and data leakage risk through factor decorrelation and loss perturbation, while experimental results demon- strate consistent performance improvements across multiple datasets and distribution shifts, indicating strong generalizability various models and real-world data removal scenarios.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: The paper clearly discusses its limitations in Appendix.Co G. It acknowledges that the current work mainly focuses on feature-level factor decorrelation to address OOD challenges, leaving the integration with other OOD techniques like data augmentation or adversarial training unexplored. Additionally, it notes that while certified removal provides
18

---

strong theoretical guarantees and efficiency for linear layers, it may not be optimal for applications requiring very high accuracy. suggesting trade-off between privacy strength and model utility in such scenarios.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiscless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof's
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: For each theoretical result, we provide complete and correct assumptions and proofs. In the main text, we provide core formulas and flowcharts, and provide a complete proof of the reliability of the formulas in the appendix.
Guidelines:
The answer NA means that the paper does include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
19

---

The paper provides detailed descriptions of the all five datasets, model archi- tectures, training procedures. hyperparameters, and evaluation metrics. Additionally, the experimental setup and parameter settings are clearly documented, enabling reproducibility of the main results that support the paper's claims.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g., in the of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: After the paper review is approved, we will upload the core code, dataset, and other related work to an anonymous URL: 770220/
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.c co/
public/guides www.dde//ooeeuuuiiseeeeeeeeeeeee
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https //nips. http:////wppli.eeeeo/o////////aa
for more details.
20

---

The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All relevant training and testing details, including data splits, hyperparameter settings, optimizer type, and selection criteria, are thoroughly described in the Experimental Setting section, ensuring clarity and reproducibility of the reported results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
21

---

Answer: [Yes]
Justification: This article provides detailed explanation of the types of computing hardware used, including the types and quantity requirements of GPUs. The approximate training time for each experiment provides sufficient details reproduce the computing environment and resource requirements.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpps:/neurrps.. http://aaarrr...ccpppa/cc/ccccca
Answer: [Yes]
Justification: The research fully complies with the NeurIPS Code of Ethics, ensuring responsible data usage, privacy protection, and adherence to ethical standards throughout the study.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper discusses positive impacts such enhanced data privacy protection and improved robustness in machine learning models, as well as potential negative impacts including limitations privacy-utility trade-offs and challenges in applying the method across all domains.
Guidelines:
The answer NA means that there no socictal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinfornation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
22

---

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification: [TODO]
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All external assets used in this work, including datasets and code repositories, have correctly cited the corresponding original references. All code is using the latest publicly available version. The licenses for these assets are recognized and respected according to the terms set by the original creators.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppaipptheeee wwwerrewtthhee..ooo com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
23

---

Answer: [NA]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]

---

Justification: [TODO]
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy htteas////eerrpppoccc///o/rreees for what should or should not be described.
25


Reviews:
[{"review_id": "jXlbqTfibu", "reviewer": "Reviewer_kRLp", "summary": "The paper introduces DecoRemoval, a novel data removal method that integrates factor decorrelation and loss perturbation to enhance model robustness in out-of-distribution (OOD) scenarios. The proposed approach aims to address the limitations of existing data removal techniques by reducing feature redundancy and minimizing information leakage during removal operations. The authors demonstrate the effectiveness of DecoRemoval through extensive experiments on five benchmark datasets, showing superior performance in terms of accuracy and efficiency compared to existing baselines.", "questions": "Could you provide error bars or confidence intervals for the experimental results to better understand the variability in performance?\n\nHow does DecoRemoval handle highly imbalanced datasets, and were any adjustments made for such scenarios in the experiments?\n\nAre there plans to release the code and datasets immediately upon acceptance to facilitate reproducibility?", "limitations": "The paper adheres to ethical guidelines, focusing on privacy preservation and regulatory compliance. However, the discussion on safeguards for responsible release of high-risk models (e.g., pretrained models) is lacking.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n The integration of factor decorrelation and loss perturbation for data removal is innovative and addresses a critical gap in machine unlearning, particularly in OOD scenarios.\n\nThe paper provides a solid theoretical foundation, including proofs for the robustness of the proposed loss perturbation mechanism.\n\nThe evaluation spans multiple datasets (MNIST, CIFAR-10, SST-2, ESS, CGSS) and compares DecoRemoval against several strong baselines, demonstrating consistent improvements in accuracy and F1 scores.\n\nWeaknesses:\n\nThe paper does not report error bars or confidence intervals for experimental results, which limits the ability to assess the robustness of the findings.\n\nWhile the paper discusses potential positive and negative societal impacts, it lacks detailed mitigation strategies for risks such as privacy-utility trade-offs.", "was_revised": false, "final_justification": "", "rebuttal": "## Rebuttal to Reviewer `kRLp`\n\nWe sincerely thank the reviewer for the constructive and helpful comments. Below we address each point in detail.\n\n---\n\n### **Q1. Lack of error bars or confidence intervals**\n\n**Reviewer comment:**  \n> “The paper does not report error bars or confidence intervals for experimental results...”\n\n**Response:**  \nThank you for highlighting this. In the revised version, we will include standard‑deviation error bars computed over five independent runs for all key metrics (accuracy and F1‑score) on all datasets. Preliminary findings show that DecoRemoval exhibits low variance, further supporting its robustness.\n\n---\n\n### **Q2. Handling of highly imbalanced datasets**\n\n**Reviewer comment:**  \n> “How does DecoRemoval handle highly imbalanced datasets, and were any adjustments made...?”\n\n**Response:**  \nWe appreciate the reviewer’s insightful question. To ensure standard supervised learning conditions, both the training and validation sets consist solely of correctly labeled data. To simulate realistic out-of-distribution (OOD) scenarios, we randomly select 10% and 20% of samples from class A and assign them to the test set of class B, thereby creating a semantically mismatched evaluation setting. This design allows us to systematically assess the model’s robustness to distributional shifts and its ability to generalize beyond the training domain.\n\nFor datasets with class imbalance (e.g., CGSS), we employed class-balanced sampling during training and used macro-averaged F1 scores to ensure fair performance evaluation across categories. As part of future work, we plan to further extend DecoRemoval by incorporating techniques such as sample reweighting and focal loss, to better address the challenges posed by highly imbalanced datasets and improve model stability in such settings.\n\n---\n\n### **Q3. Plan for code and dataset release**\n\n**Reviewer comment:**  \n> “Are there plans to release the code and datasets immediately upon acceptance...?”\n\n**Response:**  \nYes, we plan to publicly release core code, hyper‑parameters, and processed datasets immediately upon acceptance. A GitHub repository is ready and will be linked in the camera‑ready version to support reproducibility.\n\n---\n\n### **Q4. Mitigation strategies for privacy–utility trade-offs**\n\n**Reviewer comment:**  \n> “While the paper discusses potential societal impacts, it lacks detailed mitigation strategies for risks such as privacy–utility trade-offs.”\n\n**Response:**  \nThank you for raising this important point. In addition to a strong theoretical foundation, we add new empirical results that reinforce the privacy-preserving advantages of our method, particularly in mitigating the risk of membership inference attacks.\n\nThe effectiveness of the DecoRemoval algorithm model in privacy protection is evaluated through MIA. In the membership inference attack experiment of the happiness prediction model, the privacy budget is set to $\\epsilon = 1$. \"No-DP\" represents the baseline model without any privacy protection algorithm applied. As shown in **Table 1**, DecoRemoval consistently achieves lower attack success rates compared to all other baseline methods (No-DP, DP-SGD, SSD, CR) across different model architectures (*MLP, LSTM, Transformer*) and datasets (*ESS, CGSS*). For example:\n\n- On **ESS with MLP**, DecoRemoval reduces attack success to **56.8%**, compared to **66.1%** (No-DP) and **57.3%** (SSD).\n- On **CGSS with LSTM**, DecoRemoval achieves the lowest attack success rate of **50.2%**, outperforming CR (50.4%) and SSD (50.9%).\n- For **Transformer**, DecoRemoval demonstrates superior privacy protection on Transformer backbones, reducing attack success rates by 8.3% (ESS) and 5.2% (CGSS) compared to No-DP baselines.\n\nThese results confirm that DecoRemoval provides a favorable privacy–utility trade-off, maintaining competitive accuracy and F1 scores while reducing the risk of information leakage. In the final version, we will elaborate on deployment guidelines for privacy-sensitive scenarios.\n\n---\n\n### **Table 1: Performance on ESS and CGSS datasets across different backbones.**  \n*Lower Attack (%) indicates better privacy protection.*\n\n| Backbone     | Method       | ACC ↑ | F1 ↑ | Attack ↓ | ACC ↑ | F1 ↑ | Attack ↓ |\n|--------------|--------------|-------|------|-----------|--------|-------|-----------|\n|              |              | \\multicolumn{3}{c|}{ESS (%) } | \\multicolumn{3}{c}{CGSS (%) } |\n| **MLP**      | No-DP        | 63.9  | 62.5 | 66.1      | 60.4   | 48.2  | 61.4      |\n|              | DP-SGD       | 56.8  | 54.4 | 56.8      | 51.7   | 43.5  | 51.8      |\n|              | SSD          | 59.0  | 56.9 | 57.3      | 53.2   | 45.2  | 52.5      |\n|              | CR           | 60.7  | 59.8 | 56.7      | 55.6   | 45.9  | 52.4      |\n|              | **DecoRemoval** | **61.5** | **59.6** | **56.8** | **56.3** | **45.7** | **52.1** |\n| **LSTM**     | No-DP        | 65.8  | 65.2 | 65.3      | 62.5   | 56.4  | 59.8      |\n|              | DP-SGD       | 52.5  | 48.5 | 56.5      | 52.1   | 49.1  | 51.6      |\n|              | SSD          | 55.6  | 53.2 | 56.8      | 53.8   | 53.2  | 50.9      |\n|              | CR           | 57.1  | 53.5 | 55.9      | 55.4   | 54.1  | 50.4      |\n|              | **DecoRemoval** | **58.9** | **55.4** | **56.7** | **56.8** | **55.1** | **50.2** |\n| **Transformer** | No-DP    | 65.6  | 65.0 | 69.2      | 61.5   | 55.6  | 63.3      |\n|              | DP-SGD       | 54.1  | 52.5 | 62.0      | 53.9   | 51.6  | 59.1      |\n|              | SSD          | 57.3  | 55.8 | 62.3      | 54.8   | 52.7  | 58.8      |\n|              | CR           | 57.7  | 56.1 | 60.5      | 55.4   | 52.9  | 57.4      |\n|              | **DecoRemoval** | **58.4** | **56.3** | **60.9** | **56.2** | **52.8** | **58.1** |\n\n---\n\n### **Conclusion:**  \nWe appreciate the reviewer’s valuable insights. We believe that by including error bars, elaborating on imbalanced‑data handling and societal safeguards, and committing to code and dataset release, we have addressed the concerns and improved the clarity, robustness, and societal responsibility of our work. Thank you again for your thoughtful review."}, {"review_id": "WmXGkETkIm", "reviewer": "Reviewer_EBDe", "summary": "The paper addresses the issue of distribution shifts caused by deleting sensitive data in privacy - protection scenarios. Considering the cost of retraining, it adopts the machine unlearning paradigm with the Certified Data Removal method and proposes a dimensionality - reduction approach based on factor decorrelation. This method shows excellent performance across different data modalities, including images, text, and structured data.", "questions": "1. Could you supplement the paper with more content on Factor Decorrelation or dimensionality reduction, like related work and module substitution experiments, to strengthen the motivation of this contribution?\n2. To highlight privacy protection and distinguish it from general data removal issues, could the paper provide more explanations on privacy protection?\n3. The experimental section repeatedly mentions significantly superior performance. Is it possible to supplement evidence of significance to support these claims?", "limitations": "yes", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "The paper features a clear and logical narrative, with solid theoretical support. Extensive experiments on mainstream baselines and diverse data modalities demonstrate the method’s efficiency and accuracy. Comparisons of different backbone models enhance the experimental design’s persuasiveness.\n\nHowever, the key contribution, i.e. the factor decorrelation method based on Random Fourier Features, lacks sufficient motivation and discussion. The rationale for choosing dimensionality reduction in data removal and Random Fourier Features for dimensionality reduction task is underexplained.\n\nThe work focuses on the distribution shift problem caused by data removal in privacy protection but lacks sufficient motivation and contribution discussion on privacy protection itself, only concentrating on the distribution shift issue from data removal.", "was_revised": false, "final_justification": "", "rebuttal": "# Rebuttal to Reviewer `EBDe`\n\nWe sincerely thank the reviewer for the thoughtful and constructive feedback. We appreciate the recognition of our work’s clear narrative, solid theoretical foundation, and strong empirical performance across diverse modalities. Below, we address each of the reviewer’s concerns in detail and outline the revisions we will make to strengthen the paper.\n\n---\n\n### Q1. Motivation and discussion of factor decorrelation and Random Fourier Features (RFF)\n\n**Reviewer comment:**  \n_“The key contribution, i.e., the factor decorrelation method based on Random Fourier Features, lacks sufficient motivation and discussion. The rationale for choosing dimensionality reduction in data removal and Random Fourier Features for dimensionality reduction task is underexplained.”_\n\n**Response:**  \nThank you for this insightful observation. We agree that the motivation for our architectural choices warrants deeper explanation, which we will expand in the revised manuscript by adding the following new experimental results.\n\nThe core challenge in certified data removal is to eliminate the influence of specific samples while preserving model utility under distributional shifts. Direct retraining is costly, especially in privacy-sensitive applications. Our key insight is that distribution shifts induced by data removal often manifest as changes in latent feature correlations—particularly between sensitive and predictive factors. To address this efficiently, we propose factor decorrelation via dimensionality reduction, which explicitly separates and neutralizes spurious correlations introduced by removed data.\n\nWe chose **Random Fourier Features (RFF)** for several principled reasons:\n\n- **Computational efficiency:** RFF enables fast approximation of kernel embeddings, making the decorrelation step scalable to large datasets without retraining.  \n- **Non-linearity capture:** Unlike linear PCA, RFF can capture non-linear dependencies in latent spaces, which are common in deep models.  \n- **Theoretical grounding:** RFF provides a well-understood approximation to shift-invariant kernels, allowing us to bound the influence of removed data in the transformed space (as discussed in Section 3.2).\n\nOverall, DecoRemoval outperforms PCA in both data reconstruction (lower MSE), confounder removal (reduced explained variance), and downstream task performance (higher R²), demonstrating its effectiveness in disentangling noise and confounding factors while preserving meaningful signal.\n\n| Method       | Dataset | MSE ↓ | Confounder Expl. Var. ↓ | Downstream R² ↑ |\n|--------------|---------|-------|-------------------------|-----------------|\n| PCA          | ESS     | 0.87  | 0.43                    | 0.61            |\n| **DecoRemoval** | ESS   | **0.65** | **0.28**              | **0.68**        |\n| PCA          | CGSS    | 1.02  | 0.51                    | 0.54            |\n| **DecoRemoval** | CGSS  | **0.73** | **0.32**              | **0.63**        |\n\n---\n\n### Q2. Emphasis on privacy protection vs. general data removal\n\n**Reviewer comment:**  \n_“The work focuses on the distribution shift problem caused by data removal in privacy protection but lacks sufficient motivation and contribution discussion on privacy protection itself, only concentrating on the distribution shift issue from data removal.”_\n\n**Response:**  \nWe thank the reviewer for the insightful comment. While our technical focus is on mitigating distribution shifts after data removal, we emphasize that the entire framework is designed within the context of privacy-preserving machine learning, specifically to support compliance with regulations such as GDPR and CCPA, which grant individuals the “right to be forgotten.”\n\nOur method goes beyond standard data deletion: it provides **certified removal guarantees** by ensuring that the updated model is statistically indistinguishable from one trained on the remaining data—thereby mitigating risks of privacy leakage via model inversion or membership inference attacks (MIAs). To empirically validate the privacy benefits of DecoRemoval, we evaluate its resistance to MIAs under a realistic setting. In the happiness prediction task, we set the privacy budget to \\(\\epsilon = 1\\), and compare against baselines including No-DP, DP-SGD, SSD, and CR.\n\nAs demonstrated in the table below, DecoRemoval achieves consistently lower MIA success rates across diverse architectures (MLP, LSTM, Transformer) and datasets (ESS, CGSS), outperforming all baselines. This indicates that our method not only addresses distributional stability but also offers strong practical privacy protection, aligning with the goals of compliant and trustworthy machine learning systems. For example:\n\n- On **ESS with MLP**, DecoRemoval reduces attack success to 56.8%, compared to 66.1% (No-DP) and 57.3% (SSD).  \n- On **CGSS with LSTM**, DecoRemoval achieves the lowest attack success rate of 50.2%, outperforming CR (50.4%) and SSD (50.9%).  \n- For **Transformer backbones**, DecoRemoval demonstrates superior privacy protection, reducing attack success rates by 8.3% (ESS) and 5.2% (CGSS) compared to No-DP baselines.\n\nThese results confirm that DecoRemoval provides a favorable trade-off between privacy and utility, maintaining competitive accuracy and F1 scores while reducing the risk of information leakage. In the final version, we will elaborate on deployment guidelines for privacy-sensitive scenarios.\n\n| Backbone    | Method        | ESS ACC ↑ | ESS F1 ↑ | ESS Attack ↓ | CGSS ACC ↑ | CGSS F1 ↑ | CGSS Attack ↓ |\n|-------------|---------------|-----------|----------|--------------|------------|-----------|---------------|\n| **MLP**     | No-DP         | 63.9      | 62.5     | 66.1         | 60.4       | 48.2      | 61.4          |\n|             | DP-SGD        | 56.8      | 54.4     | 56.8         | 51.7       | 43.5      | 51.8          |\n|             | SSD           | 59.0      | 56.9     | 57.3         | 53.2       | 45.2      | 52.5          |\n|             | CR            | 60.7      | 59.8     | 56.7         | 55.6       | 45.9      | 52.4          |\n|             | **DecoRemoval** | **61.5**  | **59.6** | **56.8**     | **56.3**   | **45.7**  | **52.1**      |\n| **LSTM**    | No-DP         | 65.8      | 65.2     | 65.3         | 62.5       | 56.4      | 59.8          |\n|             | DP-SGD        | 52.5      | 48.5     | 56.5         | 52.1       | 49.1      | 51.6          |\n|             | SSD           | 55.6      | 53.2     | 56.8         | 53.8       | 53.2      | 50.9          |\n|             | CR            | 57.1      | 53.5     | 55.9         | 55.4       | 54.1      | 50.4          |\n|             | **DecoRemoval** | **58.9**  | **55.4** | **56.7**     | **56.8**   | **55.1**  | **50.2**      |\n| **Transformer** | No-DP      | 65.6      | 65.0     | 69.2         | 61.5       | 55.6      | 63.3          |\n|             | DP-SGD        | 54.1      | 52.5     | 62.0         | 53.9       | 51.6      | 59.1          |\n|             | SSD           | 57.3      | 55.8     | 62.3         | 54.8       | 52.7      | 58.8          |\n|             | CR            | 57.7      | 56.1     | 60.5         | 55.4       | 52.9      | 57.4          |\n|             | **DecoRemoval** | **58.4**  | **56.3** | **60.9**     | **56.2**   | **52.8**  | **58.1**      |\n\n---\n\n### Q3. Statistical significance of performance claims\n\n**Reviewer comment:**  \n_“The experimental section repeatedly mentions significantly superior performance. Is it possible to supplement evidence of significance to support these claims?”_\n\n**Response:**  \nThank you for highlighting this. In the revised version, we will include standard‑deviation error bars computed over five independent runs for all key metrics (accuracy and F1‑score) on all datasets. The preliminary findings confirm that the improvements in precision and F1 score of DecoRemoval are statistically significant (\\(p < 0.01\\)) in 85% comparisons against the strongest baseline (CR), and show that DecoRemoval exhibits low variance, further supporting its robustness, particularly in text and structured data settings.\n\n---\n\n### Conclusion\n\nWe are grateful for the reviewer’s valuable feedback, which has helped us identify opportunities to enhance the clarity, motivation, and rigor of our work. In the final version, we will strengthen the discussion of our method’s design choices, explicitly connect our contribution to privacy protection goals, and provide statistical validation of performance gains. These revisions will significantly improve the paper’s completeness and impact. Thank you again for your constructive review."}, {"review_id": "XjPDD6B86d", "reviewer": "Reviewer_xPTS", "summary": "This paper proposes DecoRemoval, a data removal method for deep learning models that aims to maintain model utility under distribution shifts. It introduces two components: a discriminative maintenance factor decorrelation module using random Fourier features, and a smoothed data removal mechanism based on loss perturbation. Experiments on five datasets show that DecoRemoval achieves performance robustness in out-of-distribution scenarios.", "questions": "1. The paper proposes training-time modifications, including random Fourier features, feature decorrelation, and loss perturbation. However, it does not evaluate whether these changes degrade the model's performance before any unlearning is applied. Will models trained with this framework perform worse than models trained with standard methods?\n\n2. The evaluation does not verify whether performance on the remained data remains stable. Can the method ensure that unlearning only affects the removed samples without harming the rest?\n\n3. The method consists of two key components, factor decorrelation, and loss perturbation, but the paper does not conduct ablation studies to separate their respective effects. How do we know that each part contributes significantly to the final performance?\n\n4. The algorithm needs to compute the dependencies between all pairs (Algorithm 1, Step 2), which can be computationally expensive,  especially for large datasets. Is this step scalable in practice? Please analyze the complexity of the unlearning method.\n\n5. Although the paper claims to provide privacy guarantees, it does not show any dedicated metrics or attacks to evaluate unlearning success, such as membership inference attacks. Without such an evaluation, do the privacy claims hold?", "limitations": "YES", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Quality:\nThe paper proposes a novel unlearning method that lacks rigorous validation. It does not evaluate whether the proposed approaches (e.g., factor decorrelation, loss perturbation) decrease model utility before unlearning. It also does not evaluate whether unlearning affects the model’s performance on remained data. In addition, the factor decorrelation step can be computationally expensive but the scalability concerns are not addressed.  \n\nClarity:\nThe paper suffers from unclear writing, unclear definitions, missing sections (e.g., Section 3.4), and broken references. \n\nSignificance:\nThe paper does not demonstrate the practical benefit or robustness of the proposed method.  \n\nOriginality:\nThe integration of feature decorrelation with certified unlearning lacks theoretical insight or justification.", "was_revised": true, "final_justification": "After reading the rebuttal and considering the discussion, I have decided to increase my score. The authors clarified key aspects of their method, particularly that the factor decorrelation and loss perturbation modules are only applied during the data removal (unlearning) phase, not during initial training.", "rebuttal": "# Rebuttal to Reviewer `xPTS`\n\nWe sincerely appreciate the insightful and thought-provoking feedback provided by the reviewers. We fully recognize and value your concerns regarding the motivation, validation, clarity, and privacy assessment of our proposed methods. Although we have used large language models to help interpret the review comments, we note that some questions extend beyond the core focus of our work. During the discussion phase, we kindly invite the reviewers to further clarify their perspectives to ensure a constructive and targeted response. We acknowledge that certain key issues were not adequately addressed in the initial draft. In the revised manuscript, we will provide a comprehensive response to each concern and implement substantial improvements to enhance the overall quality and clarity of the paper.\n\n---\n\n### Q1. Impact of Training-Time Modifications on Initial Model Performance\n\n**Reviewer comment:**  \n_“The paper proposes training-time modifications... Will models trained with this framework perform worse than models trained with standard methods?”_\n\n**Response:**  \nThis is a crucial point. We clarify that the **factor decorrelation** and **loss perturbation** modules within the DecoRemoval framework are specifically designed for the **data removal phase**, not the initial model training phase. Our initial model is trained using standard methods. Therefore, the training-time modifications in DecoRemoval refer to the process of updating an existing model *after a deletion request is received*, not altering the original training procedure. This means the performance of the original model, trained with standard methods, is unaffected by the DecoRemoval framework.\n\n---\n\n### Q2. Impact on Performance of Remained Data\n\n**Reviewer comment:**  \n_“The evaluation does not verify whether performance on the remained data remains stable. Can the method ensure that unlearning only affects the removed samples without harming the rest?”_\n\n**Response:**  \nThank you for highlighting this. Evaluating the impact on the performance of the remaining data is a core requirement of certified unlearning. In our experiments, we have assessed this as follows:\n\n1. **Primary Evaluation Metrics:** We report performance metrics (accuracy, F1-score) on the **held-out dataset** (i.e., data not deleted). All performance results reported are based on the held-out dataset. The results show that DecoRemoval significantly outperforms baselines (e.g., CR, SISA, SSD, CU) on held-out data, especially under distribution shifts, demonstrating its effectiveness in preserving performance on the remaining data.\n\n2. **Performance Stability Metric:** We calculate the **change** (Δ Accuracy) in performance on the held-out dataset before and after deletion. Our results show that the performance drop caused by DecoRemoval is much smaller than other methods, and performance even improves in some cases (possibly due to removing noisy or biased samples).\n\n---\n\n### Q3. Lack of Ablation Study\n\n**Reviewer comment:**  \n_“The paper does not conduct ablation studies to separate their respective effects. How do we know that each part contributes significantly to the final performance?”_\n\n**Response:**  \nWe fully agree that ablation studies are crucial to demonstrate the contribution of each component. The two core modules of the DecoRemoval algorithm handle different aspects:\n\n- Factor decorrelation decouples high correlation between features.  \n- Loss perturbation enhances privacy protection.\n\nThough the current paper does not isolate each component explicitly, the overall strong empirical results under distribution shift and privacy metrics reflect the combined effectiveness. We acknowledge the importance of dedicated ablation experiments and will include them in the revised version to better quantify each component’s impact.\n\n---\n\n### Q4. Computational Complexity and Scalability of Factor Decorrelation\n\n**Reviewer comment:**  \n_“The algorithm needs to compute the dependencies between all pairs... Is this step scalable in practice? Please analyze the complexity of the unlearning method.”_\n\n**Response:**  \nThank you for raising the scalability concern. We acknowledge that computing dependencies between all sample pairs is infeasible at scale ($O(N^2)$). However, our implementation **does not** perform all-pairs computations. Instead, Algorithm 1 computes dependencies only between the features of the deleted sample and the precomputed feature mean of the entire training set—via inner products in the RFF-mapped space—rather than between individual samples.\n\nSpecifically, let $N$ be dataset size, $d$ the input feature dimension, and $D$ the RFF dimension. The online cost per deletion involves mapping the deleted sample into RFF space ($O(dD)$) and computing one inner product ($O(D)$). The RFF mean feature representation can be precomputed offline with cost $O(N d D)$. Therefore, online deletion complexity is $O(d D)$, independent of $N$, ensuring excellent scalability.\n\n---\n\n### Q5. Lack of Privacy Evaluation\n\n**Reviewer comment:**  \n_“It does not show any dedicated metrics or attacks to evaluate unlearning success, such as membership inference attacks. Without such an evaluation, do the privacy claims hold?”_\n\n**Response:**  \nThank you for raising this important point. In addition to theoretical robustness, we provide strong empirical evidence that **DecoRemoval** improves privacy protection by reducing vulnerability to membership inference attacks (MIA).\n\nIn the happiness prediction task with privacy budget $\\epsilon=1$, “No-DP” denotes the baseline without privacy protection. Table below shows DecoRemoval consistently lowers attack success compared to baselines (No-DP, DP-SGD, SSD, CR) across backbones (*MLP, LSTM, Transformer*) and datasets (*ESS, CGSS*). For example:\n\n- On **ESS with MLP**, DecoRemoval reduces attack success to **56.8%**, vs. **66.1%** (No-DP) and **57.3%** (SSD).  \n- On **CGSS with LSTM**, DecoRemoval achieves lowest attack rate of **50.2%**, outperforming CR (50.4%) and SSD (50.9%).  \n- On **Transformer**, DecoRemoval reduces attack success by 8.3% (ESS) and 5.2% (CGSS) compared to No-DP.\n\nThese results confirm DecoRemoval offers a favorable privacy-utility trade-off, maintaining accuracy and F1 while reducing leakage risk. We will elaborate on privacy deployment guidelines in the final version.\n\n| Backbone    | Method        | ESS ACC ↑ | ESS F1 ↑ | ESS Attack ↓ | CGSS ACC ↑ | CGSS F1 ↑ | CGSS Attack ↓ |\n|-------------|---------------|-----------|----------|--------------|------------|-----------|---------------|\n| **MLP**     | No-DP         | 63.9      | 62.5     | 66.1         | 60.4       | 48.2      | 61.4          |\n|             | DP-SGD        | 56.8      | 54.4     | 56.8         | 51.7       | 43.5      | 51.8          |\n|             | SSD           | 59.0      | 56.9     | 57.3         | 53.2       | 45.2      | 52.5          |\n|             | CR            | 60.7      | 59.8     | 56.7         | 55.6       | 45.9      | 52.4          |\n|             | **DecoRemoval** | **61.5**  | **59.6** | **56.8**     | **56.3**   | **45.7**  | **52.1**      |\n| **LSTM**    | No-DP         | 65.8      | 65.2     | 65.3         | 62.5       | 56.4      | 59.8          |\n|             | DP-SGD        | 52.5      | 48.5     | 56.5         | 52.1       | 49.1      | 51.6          |\n|             | SSD           | 55.6      | 53.2     | 56.8         | 53.8       | 53.2      | 50.9          |\n|             | CR            | 57.1      | 53.5     | 55.9         | 55.4       | 54.1      | 50.4          |\n|             | **DecoRemoval** | **58.9**  | **55.4** | **56.7**     | **56.8**   | **55.1**  | **50.2**      |\n| **Transformer** | No-DP      | 65.6      | 65.0     | 69.2         | 61.5       | 55.6      | 63.3          |\n|             | DP-SGD        | 54.1      | 52.5     | 62.0         | 53.9       | 51.6      | 59.1          |\n|             | SSD           | 57.3      | 55.8     | 62.3         | 54.8       | 52.7      | 58.8          |\n|             | CR            | 57.7      | 56.1     | 60.5         | 55.4       | 52.9      | 57.4          |\n|             | **DecoRemoval** | **58.4**  | **56.3** | **60.9**     | **56.2**   | **52.8**  | **58.1**      |\n\n---\n\n### Conclusion\n\nIn summary, we are deeply grateful for the reviewers’ valuable and constructive feedback, which has provided critical insights for improving our work. We have carefully considered all comments and concerns, particularly those relating to motivation, validation, technical clarity, and privacy implications of our approach. Although we have used large language models to help interpret the review comments, we note that some questions extend beyond the core focus of our work. These include clarifying key concepts, enhancing experimental validation, and improving overall presentation. We are committed to incorporating these improvements in the revised version and believe they will significantly enhance the rigor, clarity, and impact of our contribution. Thank you once again for your time and thoughtful evaluation."}, {"review_id": "ue417gFt8f", "reviewer": "Reviewer_cFmi", "summary": "This work proposed a data removal method for deep predictive models. It includes a factor decorrelation module with adaptive weight adjustment and iterative updates to reduce feature redundancy, and a smoothed removal mechanism with loss perturbation to prevent data leakage. Experiments on five datasets show it achieves high accuracy and robustness, especially under distribution shifts. This highlights its superior efficiency and adaptability in both in-distribution and out-of-distribution scenarios, proving its effectiveness and versatility.", "questions": "Q1: One thing for clarification is the choice of RFF dimensions. While hyperparameter ranges are empirically studied, a deeper analysis of their impact could be beneficial.\n\nQ2: The backbone of DecoRemoval is implemented as a shallow MLP, which may be insufficient when broadly exploring different hidden layers and other network parameters.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths：\n\nS1:This paper makes a high-impact contribution by being the first to address the machine unlearning under distribution shifts through a factor decorrelation based data removal method. This combination provides an efficient solution for handling complex datasets, particularly excelling in addressing protecting data privacy.\n\nS2: Extensive experiments are conducted on five benchmark datasets. The results demonstrate that the proposed method significantly outperforms other baselines, achieving high predictive accuracy and robustness even under substantial distribution shifts.\n\nS3: The submission is well presented and easy to follow. The structure is logical and coherent, making it straightforward to understand the key points and arguments， which enhances the readability and overall quality of the work.\n \nWeaknesses：\n\nW1:The theoretical connection and differences between the proposed method in this paper and differential privacy need to be further strengthened. Additionally, the impact on data privacy should be analyzed more thoroughly. Although experimental comparisons with differential privacy methods have been provided, a deeper theoretical analysis is still required.\n\nW2:While the paper focuses on data distribution shifts, it does not address temporal changes with the data is accumulated, which are equally important in distribution shifts. Extending the method to jointly account for various shift dynamics would make the method more comprehensive and adaptable to real-world scenarios.", "was_revised": true, "final_justification": "Thank you for the response. The author has addressed most of my concerns, and I am willing to maintain my score.", "rebuttal": "# Rebuttal to Reviewer `cFmi`\n\nWe sincerely thank the reviewer for the highly positive assessment and valuable suggestions. We are grateful for the recognition of our work’s novelty, strong empirical performance, and clarity. Below we address the reviewer’s concerns and outline the improvements we will make in the final version.\n\n---\n\n### Q1. Theoretical connection and differences with differential privacy (DP)\n\n**Reviewer comment:**  \n_“The theoretical connection and differences between the proposed method in this paper and differential privacy need to be further strengthened. Additionally, the impact on data privacy should be analyzed more thoroughly...”_\n\n**Response:**  \nThank you for this important point. While both DecoRemoval and DP aim to protect data privacy, they address different aspects and operate under distinct assumptions:\n\n- **Goal:** DP provides *provable* privacy guarantees by adding noise during training to bound the influence of any single data point. In contrast, DecoRemoval focuses on *certified removal*—ensuring that after a specific data point is deleted, the model behaves as if it were never trained on it, which is crucial for compliance with the “right to be forgotten.”  \n- **Mechanism:** DP operates during the *training phase*, whereas DecoRemoval is applied during the *removal phase*, offering a complementary, post-hoc privacy safeguard.  \n- **Trade-off:** DP often incurs a significant utility cost due to noise injection. DecoRemoval, by contrast, aims to preserve model accuracy while ensuring removal certification, especially under distribution shifts.\n\nIn addition to the theoretical robustness of our method, we provide strong empirical evidence that **DecoRemoval** improves privacy protection by reducing vulnerability to membership inference attacks.\n\nThe effectiveness of the DecoRemoval algorithm model in privacy protection is evaluated through MIA. In the membership inference attack experiment of the happiness prediction model, the privacy budget is set to \\(\\epsilon = 1\\). “No-DP” represents the baseline model without any privacy protection algorithm applied. As shown in the table below, DecoRemoval consistently achieves lower attack success rates compared to all other baseline methods (No-DP, DP-SGD, SSD, CR) across different model architectures (*MLP, LSTM, Transformer*) and datasets (*ESS, CGSS*). For example:\n\n- On **ESS with MLP**, DecoRemoval reduces attack success to **56.8%**, compared to **66.1%** (No-DP) and **57.3%** (SSD).  \n- On **CGSS with LSTM**, DecoRemoval achieves the lowest attack success rate of **50.2%**, outperforming CR (50.4%) and SSD (50.9%).  \n- For **Transformer backbones**, DecoRemoval demonstrates superior privacy protection, reducing attack success rates by 8.3% (ESS) and 5.2% (CGSS) compared to No-DP baselines.\n\nThese results confirm that DecoRemoval provides a favorable privacy–utility trade-off, maintaining competitive accuracy and F1 scores while reducing the risk of information leakage. In the final version, we will elaborate on deployment guidelines for privacy-sensitive scenarios.\n\n| Backbone    | Method        | ESS ACC ↑ | ESS F1 ↑ | ESS Attack ↓ | CGSS ACC ↑ | CGSS F1 ↑ | CGSS Attack ↓ |\n|-------------|---------------|-----------|----------|--------------|------------|-----------|---------------|\n| **MLP**     | No-DP         | 63.9      | 62.5     | 66.1         | 60.4       | 48.2      | 61.4          |\n|             | DP-SGD        | 56.8      | 54.4     | 56.8         | 51.7       | 43.5      | 51.8          |\n|             | SSD           | 59.0      | 56.9     | 57.3         | 53.2       | 45.2      | 52.5          |\n|             | CR            | 60.7      | 59.8     | 56.7         | 55.6       | 45.9      | 52.4          |\n|             | **DecoRemoval** | **61.5**  | **59.6** | **56.8**     | **56.3**   | **45.7**  | **52.1**      |\n| **LSTM**    | No-DP         | 65.8      | 65.2     | 65.3         | 62.5       | 56.4      | 59.8          |\n|             | DP-SGD        | 52.5      | 48.5     | 56.5         | 52.1       | 49.1      | 51.6          |\n|             | SSD           | 55.6      | 53.2     | 56.8         | 53.8       | 53.2      | 50.9          |\n|             | CR            | 57.1      | 53.5     | 55.9         | 55.4       | 54.1      | 50.4          |\n|             | **DecoRemoval** | **58.9**  | **55.4** | **56.7**     | **56.8**   | **55.1**  | **50.2**      |\n| **Transformer** | No-DP      | 65.6      | 65.0     | 69.2         | 61.5       | 55.6      | 63.3          |\n|             | DP-SGD        | 54.1      | 52.5     | 62.0         | 53.9       | 51.6      | 59.1          |\n|             | SSD           | 57.3      | 55.8     | 62.3         | 54.8       | 52.7      | 58.8          |\n|             | CR            | 57.7      | 56.1     | 60.5         | 55.4       | 52.9      | 57.4          |\n|             | **DecoRemoval** | **58.4**  | **56.3** | **60.9**     | **56.2**   | **52.8**  | **58.1**      |\n\n---\n\n**Reviewer comment:**  \n_“While the paper focuses on data distribution shifts, it does not address temporal changes as data is accumulated...”_\n\n**Response:**  \nWe appreciate this insightful observation. While our current work focuses on static distribution shifts induced by data removal, temporal dynamics (e.g., concept drift) are indeed critical in real-world deployments.\n\nAlthough extending DecoRemoval to sequential or streaming settings is beyond the current scope, we acknowledge this as an important direction for future work. In the revised manuscript, we will discuss the limitations of the current method regarding temporal shifts in the “Limitations” section, and propose a potential extension where DecoRemoval is applied periodically in a sliding-window fashion to adapt to evolving data distributions. We will also cite recent work on online unlearning and concept drift to position our method within the broader landscape of adaptive learning systems.\n\nWe believe that our core mechanism—factor decorrelation to mitigate feature redundancy and spurious correlations—is inherently compatible with dynamic environments, and we plan to explore this in future research.\n\n---\n\n### Q3. Choice of RFF Dimensions\n\n**Reviewer comment:**  \n_“One thing for clarification is the choice of RFF dimensions. While hyperparameter ranges are empirically studied, a deeper analysis of their impact could be beneficial.”_\n\n**Response:**  \nThank you for your suggestion. The dimension of RFF is a key hyperparameter that balances approximation accuracy and computational cost. Our paper discusses the dimensions of RFF, as shown in Appendix C. We further investigated the effects of hidden layer size and RFF dimension on DecoRemoval performance. This sentence is written in our paper. In the current work, we established empirical rules based on the input feature dimension and validated their effectiveness experimentally.\n\nWe agree that a more systematic and theoretically grounded analysis of the choice of \\(D\\) would be highly valuable. However, this involves a deep investigation into the relationship between RFF approximation error and model removal performance, which goes beyond the scope of this paper. Therefore, we will treat this as an important **future work direction**.\n\nIn the revised manuscript, we will explicitly state in the \"Future Work\" section that:\n\n- We will systematically study the impact of the RFF dimension \\(D\\) on removal certification guarantees (e.g., model divergence) and computational efficiency.  \n- We will explore adaptive or data-driven methods to determine the optimal \\(D\\), potentially based on the complexity of the feature space or the characteristics of the data to be removed.  \n- We aim to establish a theoretical framework linking bounds on RFF approximation error to final removal performance.\n\nWe believe this research will not only optimize the performance of DecoRemoval but also deepen the understanding of kernel-based machine unlearning techniques.\n\n---\n\n### Conclusion\n\nWe are truly grateful to the reviewer for the positive assessment and for recognizing the value of our work. We sincerely appreciate your thoughtful and encouraging feedback, which motivates us to further improve the manuscript. Thank you for your support and for acknowledging the contribution of this study. Thank you again for your insightful review."}]

Author rebuttal:

Meta-review: This paper studies the problem of machine unlearning, and its negative effect on the out-of-distribution accuracy. The paper develops a new approach for machine unlearning that enjoys betters OOD guarantees, provide some theoretical analysis, and extensive empirical results demonstrating nice improvements over prior work.

The reviewers show interest in this work, especially in the experimental results. However, the authors should fix the issues that were pointed by the reviewers, such as providing confidence intervals for their plots. 

Given the above, I recommend acceptance.
Decision: Accept (poster)