Title: FedRW: Efficient Privacy-Preserving Data Reweighting for Enhancing Federated Learning of Language Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: LLM Privacy

Data Deduplication

Federated Learning

Secure Multi-Party Computation
Abstract: Data duplication within large-scale corpora often impedes large language models' (LLMs) performance and privacy. In privacy-concerned federated learning scenarios, conventional deduplication methods typically rely on trusted third parties to perform uniform deletion, risking loss of informative samples while introducing privacy vulnerabilities. To address these gaps, we propose Federated ReWeighting (FedRW), the first privacy-preserving framework, to the best of our knowledge, that performs soft deduplication via sample reweighting instead of deletion in federated LLM training, without assuming a trusted third party. At its core, FedRW proposes a secure, frequency-aware reweighting protocol through secure multi-party computation, coupled with a parallel orchestration strategy to ensure efficiency and scalability. During training, FedRW utilizes an adaptive reweighting mechanism with global sample frequencies to adjust individual loss contributions, effectively improving generalization and robustness. Empirical results demonstrate that FedRW outperforms the state-of-the-art method by achieving up to $28.78\times$ speedup in preprocessing and approximately $11.42$\% improvement in perplexity, while offering enhanced security guarantees. FedRW thus establishes a new paradigm for managing duplication in federated LLM training.


Peer reviews:
comments: SUMMARY:
This paper introduces Federated ReWeighting (FedRW), a privacy-preserving framework designed to address the challenges of data duplication in the federated training of large language models (LLMs). Traditional deduplication methods in federated learning often rely on trusted third parties and involve deleting redundant data, which can lead to the loss of valuable information and introduce privacy risks. FedRW proposes a "soft deduplication" approach that reweights data samples based on their global frequency, rather than deleting them, and does so without requiring a trusted third party.

STRENGTHS AND WEAKNESSES:
Strengths
- The paper focuses on the data duplication issue in federated learning, an important topic in the field.
- The paper clearly illustrates the framework in the method part.

Weaknesses
- The paper does not clearly show the difference between the proposed method and the baseline, EP-MPD.
- The paper does not explain the FL experiment settings. What FL settings are used currently? Is the proposed method still effective under different FL settings, such as IID, non-IID, or personalized FL?
- In the experiments, only GPT-2 Large and DistilGPT-2 are used for the method. It would be better to include some more recent LLMs in the experiments.

QUESTIONS:
See above.

LIMITATIONS:
There is no discussion on limitations in the paper.
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-06-06

comments: SUMMARY:
FedRW has proposed an innovative privacy preserving federated learning framework that replaces trusted third parties with a frequency aware sample reweighting mechanism, securely calculates global sample frequencies using the PPMPR protocol, and combines a layered parallel strategy to reduce preprocessing complexity, achieving a preprocessing acceleration of up to 28.78 times while improving model generalization ability.

STRENGTHS AND WEAKNESSES:
Strengths: 
1. Balancing privacy and efficiency
2. By using logarithmic functions to dynamically adjust sample weights and preserve low-frequency sample information, an average confusion reduction of 11.42% was achieved on datasets such as Haiku and Sonnets, which is particularly better than the performance of hard deletion methods in data scarcity scenarios.
3. MPC protocol based on semi honest model provides provable security
Weaknesses:
1.The innovation of FedRW is mainly reflected in the first soft deduplication implementation and efficiency optimization in federated scenarios (such as pre-processing acceleration), but its protocol, weight mechanism, and parallel strategy heavily rely on existing technologies (PSI, centralized soft deduplication).
2.This framework provides an efficient and scalable solution for privacy sensitive federated training, but further optimization is needed for semantic redundancy processing and dynamic client adaptation capabilities.

QUESTIONS:
See Weakness.

LIMITATIONS:
Yes, the proposed method, while promising, reveals certain limitations in the Experiments section.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn FL settings, there is a fundamental dilemma for deduplication: local fails to detect inter-client duplication. Existing method may remove some domain-specific information and introduce additional overheads. This paper propose a reweighting method to softly deduplicate samples without a trusted third party. This framework reweight based on the frequency for a more nuanced control based on the multi-party computation.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n\n1. FedRW introduces a pioneering soft deduplication method for FL, avoiding data deletion and preserving dataset integrity, which is a significant advancement over hard deduplication techniques.\n2. By eliminating the need for a trusted third party and using secure 2PC with PSI, FedRW ensures robust privacy guarantees, addressing critical vulnerabilities in decentralized settings.\n\nWeakness:\nSee questions\n\nQUESTIONS:\n1. Due to the different data collection fields or geographical locations of different clients, cooperation is meaningful. In this case, it is difficult for two clients to have exactly the same samples. Therefore, when the two repeated samples are not exactly the same, is this method still effective?\n2. I also have doubts about the significance of data deduplication in real FL scenarios. Is there such a problem in real datasets? Have different clients collected a large number of similar datasets?\n3. The experimental settings mention hyperparameters (e.g., learning rates, epochs), but the paper does not discuss sensitivity analyses or how hyperparameter choices impact performance, which could affect generalizability.\n4. This paper also did not conduct experiments with the current mainstream models. Therefore, whether partial repetition will still be harmful when using mainstream models is also an unknown question\n\nLIMITATIONS:\nSee above", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes FedRW, a novel framework for privacy-preserving soft deduplication in federated LLM training. The PPMPR protocol avoids the need for a trusted third party, scales efficiently through a parallel orchestration strategy, and integrates sample-level loss weighting in training.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\nThe technical contributions are clearly defined, and the paper is well-written with a logical structure.\n\nWeaknesses\n\n1.\tIt would be helpful if the authors could include training loss curves or convergence comparisons to illustrate whether the proposed reweighting mechanism slows down or accelerates model convergence relative to deletion-based approaches.\n\n2.\tThere are a few writing-related issues that could improve the clarity of the manuscript. First, using vague terms such as “nuanced” is discouraged in scientific writing, as it does not convey precise meaning. Second, Figures 1 and 2 are visually engaging but lack adequate explanation of key elements. Third, the system formulation in Section 3 is not fully adapted to the federated learning setting, it would benefit from a clearer illustration.\n\n3.\tThe evaluation is limited to synthetic duplication. It would strengthen the paper to include results or discussion on how FedRW performs in real-world scenarios with naturally occurring duplication, such as instruction-tuning data, which often contains repeated patterns.\n\n4.\tThe protocol assumes a semi-honest threat model. However, in practice, malicious clients may attempt to manipulate the protocol by misreporting sample frequencies or injecting misleading data into the intersection process. It would be valuable to discuss whether FedRW could be extended or adapted to remain robust under such adversarial behavior.\n\nQUESTIONS:\nCheck Weaknesses\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}