Title: Factor Decorrelation Enhanced Data Removal from Deep Predictive Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Factor decorrelation

Data removal

Out-of-distribution performance

Deep predictive models

Optimization
Abstract: The imperative of user privacy protection and regulatory compliance necessitates sensitive data removal in model training, yet this process often induces distributional shifts that undermine model performance-particularly in out-of-distribution (OOD) scenarios. We propose a novel data removal approach that enhances deep predictive models through factor decorrelation and loss perturbation. Our approach introduces: (1) a discriminative-preserving factor decorrelation module employing dynamic adaptive weight adjustment and iterative representation updating to reduce feature redundancy and minimize inter-feature correlations. (2) a smoothed data removal mechanism with loss perturbation that creates information-theoretic safeguards against data leakage during removal operations. Extensive experiments on five benchmark datasets show that our approach outperforms other baselines and consistently achieves high predictive accuracy and robustness even under significant distribution shifts. The results highlight its superior efficiency and adaptability in both in-distribution and out-of-distribution scenarios.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposed a data removal method for deep predictive models. It includes a factor decorrelation module with adaptive weight adjustment and iterative updates to reduce feature redundancy, and a smoothed removal mechanism with loss perturbation to prevent data leakage. Experiments on five datasets show it achieves high accuracy and robustness, especially under distribution shifts. This highlights its superior efficiency and adaptability in both in-distribution and out-of-distribution scenarios, proving its effectiveness and versatility.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths：\n\nS1:This paper makes a high-impact contribution by being the first to address the machine unlearning under distribution shifts through a factor decorrelation based data removal method. This combination provides an efficient solution for handling complex datasets, particularly excelling in addressing protecting data privacy.\n\nS2: Extensive experiments are conducted on five benchmark datasets. The results demonstrate that the proposed method significantly outperforms other baselines, achieving high predictive accuracy and robustness even under substantial distribution shifts.\n\nS3: The submission is well presented and easy to follow. The structure is logical and coherent, making it straightforward to understand the key points and arguments， which enhances the readability and overall quality of the work.\n \nWeaknesses：\n\nW1:The theoretical connection and differences between the proposed method in this paper and differential privacy need to be further strengthened. Additionally, the impact on data privacy should be analyzed more thoroughly. Although experimental comparisons with differential privacy methods have been provided, a deeper theoretical analysis is still required.\n\nW2:While the paper focuses on data distribution shifts, it does not address temporal changes with the data is accumulated, which are equally important in distribution shifts. Extending the method to jointly account for various shift dynamics would make the method more comprehensive and adaptable to real-world scenarios.\n\nQUESTIONS:\nQ1: One thing for clarification is the choice of RFF dimensions. While hyperparameter ranges are empirically studied, a deeper analysis of their impact could be beneficial.\n\nQ2: The backbone of DecoRemoval is implemented as a shallow MLP, which may be insufficient when broadly exploring different hidden layers and other network parameters.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes DecoRemoval, a data removal method for deep learning models that aims to maintain model utility under distribution shifts. It introduces two components: a discriminative maintenance factor decorrelation module using random Fourier features, and a smoothed data removal mechanism based on loss perturbation. Experiments on five datasets show that DecoRemoval achieves performance robustness in out-of-distribution scenarios.\n\nSTRENGTHS AND WEAKNESSES:\nQuality:\nThe paper proposes a novel unlearning method that lacks rigorous validation. It does not evaluate whether the proposed approaches (e.g., factor decorrelation, loss perturbation) decrease model utility before unlearning. It also does not evaluate whether unlearning affects the model’s performance on remained data. In addition, the factor decorrelation step can be computationally expensive but the scalability concerns are not addressed.  \n\nClarity:\nThe paper suffers from unclear writing, unclear definitions, missing sections (e.g., Section 3.4), and broken references. \n\nSignificance:\nThe paper does not demonstrate the practical benefit or robustness of the proposed method.  \n\nOriginality:\nThe integration of feature decorrelation with certified unlearning lacks theoretical insight or justification.\n\nQUESTIONS:\n1. The paper proposes training-time modifications, including random Fourier features, feature decorrelation, and loss perturbation. However, it does not evaluate whether these changes degrade the model's performance before any unlearning is applied. Will models trained with this framework perform worse than models trained with standard methods?\n\n2. The evaluation does not verify whether performance on the remained data remains stable. Can the method ensure that unlearning only affects the removed samples without harming the rest?\n\n3. The method consists of two key components, factor decorrelation, and loss perturbation, but the paper does not conduct ablation studies to separate their respective effects. How do we know that each part contributes significantly to the final performance?\n\n4. The algorithm needs to compute the dependencies between all pairs (Algorithm 1, Step 2), which can be computationally expensive,  especially for large datasets. Is this step scalable in practice? Please analyze the complexity of the unlearning method.\n\n5. Although the paper claims to provide privacy guarantees, it does not show any dedicated metrics or attacks to evaluate unlearning success, such as membership inference attacks. Without such an evaluation, do the privacy claims hold?\n\nLIMITATIONS:\nYES", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the issue of distribution shifts caused by deleting sensitive data in privacy - protection scenarios. Considering the cost of retraining, it adopts the machine unlearning paradigm with the Certified Data Removal method and proposes a dimensionality - reduction approach based on factor decorrelation. This method shows excellent performance across different data modalities, including images, text, and structured data.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper features a clear and logical narrative, with solid theoretical support. Extensive experiments on mainstream baselines and diverse data modalities demonstrate the method’s efficiency and accuracy. Comparisons of different backbone models enhance the experimental design’s persuasiveness.\n\nHowever, the key contribution, i.e. the factor decorrelation method based on Random Fourier Features, lacks sufficient motivation and discussion. The rationale for choosing dimensionality reduction in data removal and Random Fourier Features for dimensionality reduction task is underexplained.\n\nThe work focuses on the distribution shift problem caused by data removal in privacy protection but lacks sufficient motivation and contribution discussion on privacy protection itself, only concentrating on the distribution shift issue from data removal.\n\nQUESTIONS:\n1. Could you supplement the paper with more content on Factor Decorrelation or dimensionality reduction, like related work and module substitution experiments, to strengthen the motivation of this contribution?\n2. To highlight privacy protection and distinguish it from general data removal issues, could the paper provide more explanations on privacy protection?\n3. The experimental section repeatedly mentions significantly superior performance. Is it possible to supplement evidence of significance to support these claims?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces DecoRemoval, a novel data removal method that integrates factor decorrelation and loss perturbation to enhance model robustness in out-of-distribution (OOD) scenarios. The proposed approach aims to address the limitations of existing data removal techniques by reducing feature redundancy and minimizing information leakage during removal operations. The authors demonstrate the effectiveness of DecoRemoval through extensive experiments on five benchmark datasets, showing superior performance in terms of accuracy and efficiency compared to existing baselines.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n The integration of factor decorrelation and loss perturbation for data removal is innovative and addresses a critical gap in machine unlearning, particularly in OOD scenarios.\n\nThe paper provides a solid theoretical foundation, including proofs for the robustness of the proposed loss perturbation mechanism.\n\nThe evaluation spans multiple datasets (MNIST, CIFAR-10, SST-2, ESS, CGSS) and compares DecoRemoval against several strong baselines, demonstrating consistent improvements in accuracy and F1 scores.\n\nWeaknesses:\n\nThe paper does not report error bars or confidence intervals for experimental results, which limits the ability to assess the robustness of the findings.\n\nWhile the paper discusses potential positive and negative societal impacts, it lacks detailed mitigation strategies for risks such as privacy-utility trade-offs.\n\nQUESTIONS:\nCould you provide error bars or confidence intervals for the experimental results to better understand the variability in performance?\n\nHow does DecoRemoval handle highly imbalanced datasets, and were any adjustments made for such scenarios in the experiments?\n\nAre there plans to release the code and datasets immediately upon acceptance to facilitate reproducibility?\n\nLIMITATIONS:\nThe paper adheres to ethical guidelines, focusing on privacy preservation and regulatory compliance. However, the discussion on safeguards for responsible release of high-risk models (e.g., pretrained models) is lacking.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-06"}