FedRW: Efficient Privacy-Preserving Data Reweighting for Enhancing Federated Learning of Language Models
Authors: Pukang Ye
Junwei Luo
Jiachen Shen
Saipan Zhou
Shangmin Dou
Zhenfu Cao
Hanzhe Yao
Xiaolei Dong
Yunbo Yang
Venue: NeurIPS 2025 poster ()
Keywords: LLM Privacy
Data Deduplication
Federated Learning
Secure Multi-Party Computation
Primary area: social_and_economic_aspects_of_machine_learning

Abstract: Data duplication within large-scale corpora often impedes large language models' (LLMs) performance and privacy. In privacy-concerned federated learning scenarios, conventional deduplication methods typically rely on trusted third parties to perform uniform deletion, risking loss of informative samples while introducing privacy vulnerabilities. To address these gaps, we propose Federated ReWeighting (FedRW), the first privacy-preserving framework, to the best of our knowledge, that performs soft deduplication via sample reweighting instead of deletion in federated LLM training, without assuming a trusted third party. At its core, FedRW proposes a secure, frequency-aware reweighting protocol through secure multi-party computation, coupled with a parallel orchestration strategy to ensure efficiency and scalability. During training, FedRW utilizes an adaptive reweighting mechanism with global sample frequencies to adjust individual loss contributions, effectively improving generalization and robustness. Empirical results demonstrate that FedRW outperforms the state-of-the-art method by achieving up to $28.78\times$ speedup in preprocessing and approximately $11.42$\% improvement in perplexity, while offering enhanced security guarantees. FedRW thus establishes a new paradigm for managing duplication in federated LLM training.

FedRW: Efficient Privacy-Preserving Data Reweighting for Enhancing Federated Learning of Language Models
Pukang Ye*
Junwei Luo*
Jiachen Shen 9
Saipan Zhou
Shangmin D
Zhenfu Cao
Hanzhe Yao
Xiaolei Dong Yunbo Yang
East China Normal University Wuhan University *Zejiang University *ZStack
Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security
Abstract
Data duplication within large-scale corpora often impedes large language models' (LLMs) performance and privacy. privacy-conceree federated learning scenar- ios, conventional deduplication methods typically rely trusted third parties to perform uniform deletion, risking loss of informative samples while introducing privacy vulnerabilitiis. To address these gaps, we propose Federated Weiehtting (FedRW), the first privacy-preserving framework, to the best of our knowledge, that performs soft deduplication via sample reweighting instead of deletion in federated LLM training, without assuming a trusted third party. At its core, FedRW proposes secure, frequency-aware reweighting protocol through secure multi-party com- putation, coupled with parallel orchestration strategy to ensure efficiency and scalability. During training, FedRW utilizes adaptive reweighting mechanism with global sample frequencies to adjust individual loss contributions, effectively improving generalization and robustness. Empirical results demonstrate that Fe- dRw outperforms the state-of-the-arr method by achieving up to 28.78x speedup in preprocessing and approximately 11.42% improvement in perplexity, while offering enhanced security guarantees. FedRW thus establishes a new paradigm for managing duplication federated LLM training.
Global Deduplication
Trusted Third Party
Global Data Deduplication
Sa
Pg
Private
Data
Clients
Deduplication challenges in FL
Hard deduplication solution
Figure 1: Deduplication in Federated Learning (FL). (a) Challenges of global deduplication in decentralized settings: privacy constraints prohibit direct data sharing. (b) State-of-the-ee solution utilizing hard deduplication over enerypted data, requiring trusted third party.
S Main Contact: 512759020288stu. ecnu. wenu.edu.ee.
Correspondence authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
have driven remarkable progress across wide range of appli- cations (6.9. However, their performance fundamentally depends on data quality, yet real-world corpora often suffer from noise, bias, and especially redundancy. Among these issues, duplicated sequences are particularly widespread in large text datasets [10 weakening generalization and encouraging memorization. This not only hinders downstream performance but also increases vulnerability to privacy attacks such as model inversion, prompt injection, and membership infer- ence [1H As result, data deduplication has become standard preprocessing step in training pipelines. Existing techniques fall into two categories: hard deduplication, which removes duplicates via exact or fuzzy matching (e.g., suffix arrays, MinHash) [16] 17]; and soft deduplication, which reweights samples to preserve dataset integrity and avoid brittle thresholding [1].
Large language models (LLMs) 115
Meanwhile, the growing scareity of high-quality public data and rising concerns over data privacy 23 have brought federated learning (FL. 24 to the forefront as a compelling alternative for L.M training. By enabling collaborative learning across decentralized clients without local data sharing. FL naturally supports privacy preservation and improved utilization of high-value private data. Yet, FL introduces unique challenges for deduplication, presented in Figure a). Unlike centralized settings, global redundancy across clients cannot be directly resolved due to privacy constraints. A fundamental dilemma emerges: local deduplication fails to detect inter-client duplicates, while global mechanisms cannot bypass privacy silos, leaving redundancy unresolved in federated settings.
Abadi et al. 25 's EP-MPD represents the most state-of-the-arr work for federated hard deduplication, a robust cryptographic framework built on group private set intersection (26]. as illustrated in Figure [b). Nonetheless, key challenges remain unresolved: (1) strict removal of samples may discard informative or domain-specific content beneficial to model training; (2) multi-round key agreement and encryption introduce significant computational and communication overhead; and (3) reliance on a trusted third party for both encryption and duplicate counting reduces feasibility in stricter privacy settings.
To address the issues mentioned above, we propose Federated Re WeWeighting (FedRW), to the best of our knowledge, the first framework that enables privacy
privacy-preservnng preserving soft deduplication in federated LLM training without relying on any trusted third party. Unlike state-of-the-arr method that discards duplicated samples, FedRW pioneers new paradigm of secure, frequency-avee aware sample reweighting, enabling fine-grained control over sample redundancy while ensuring strict privacy guarantees. At the Multi-Party Reweighting (PPMPR), which
core of FedRW lies novel protocol, Privacy-Preservve securely identifies global duplication patterns across clients through a series of lightweight, third-party- free two-party interactions. To ensure scalability, we further introduce a parallel orchestration strategy that organizes the pairwise interactions into a hierarchical schedule, significantly reducing protocol complexity. Comprehensive experiments demonstrate that FedRW improves both preprocessing efficiency and model generalization, particularly in data-scare scarce and resource-comee coouuce-constraieed federated settings. In summary, the key contributions are:
FedRW Framework. Duplicate or overly frequent samples in federated LLM training lead to inefficiency and privacy leakage, especially when deletion solutions impractical. To the best of our knowledge, we propose FedRW, the first framework to achieve Divaacyppeeerving soft deduplication in federated LLM training. Unlike hard deletion methods, FedRW introduces secure, frequency- sample reweighting, establishing a new paradigm that bridges privacy protection and data-centric optimization.
PPMPR Protocol. We design PPMPR, a secure protocol for global frequency estimation without relying on trusted third party. To scale to practical settings, we further introduce a parallel orchestration strategy that reduces the total protocol complexity from O(n²) O log, "I), achieving 17.61-28.7 78x acceleration on large datasets and 4.09-28.78x speedup in preprocessing when scaled to 50 parties.
Experimental Evaluation. We conduct extensive empirical studies across diverse datasets and model configurations. By adaptive reweighting, FedRW yields approximately 11.42% perplexity reduction over the baseline, with particularly enhanced robustness under data- scarce and resource-constrained federated settings, where hard deduplication methods often exhibit apparent limitations.
2

---

2
Related Work
This section reviews data deduplication, categorizing centralized and distributed approaches. We emphasize the limitations distributed settings, which motivates our proposed FedRW framework.
Centralized Deduplication. Centralized deduplication is crucial for large text corpora, which often contain substantial exact or near-exact samples [TO] [T] that degrade model performance and compromise privacy [TT 15 25 Techniques for exact matching commonly include suffix arrays [7] while fuzzy matching typically employs MinHash for syntactic similarity [[] 17]
Semantic duplication can be identified using pretrained reference models 22 28
Instead of removing duplicates, soft deduplication methods reweight training data to mitigate redun- dancy while preserving the integrity and valuable diversity of datasets. For instance, RedPajama- Data-v2 129 leverages over 40 quality metrics for systematic filtering and reweighting. DoReMi [20] derives domain-specific weights estimated by proxy model. Methods like SoftDedup 22 and
DSIR 21 quantify sample commonness or importance via n-grams. DrICL (30) uses differentiated
learning and cumulative advantages for dynamic reweighting. RHO-1 [31 employs token-level
scoring with Selective Language Modeling. However, these centralized strategies are not directly applicable to privacy-conee concered FL environments, which effectively leverage high-quality private
data.
Distributed Deduplication. Deduplication in FL faces unique challenges due to privacy constraints and data silos. Existing work DupLESS [32] proposes encrypted deduplication using dual-server architecture, one for encryption key derivation and one for ciphertext deduplication. The state-of- the-art, EP-MPD 255 introduces a group private set intersection framework built on symmetric-key encryption [26] and oblivious pseudorandom functions [33] but still relies on trusted third party. Critically, these methods focus solely on hard deduplication, neglecting the benefits of reweighting strategies that better preserve data utility and potentially enhance model performance.
These limitations highlight the need for decentralized soft deduplication solution that ensures privacy without relying on trusted third parties. To this end, we propose an efficient, secure, and third third-party- ceal-pprtt----eee free reweighting framework for federated LLM training, delivering enhanced scalability, performance, and robustness while also ensuring stronger privacy guarantees.
3 Preliminaries
Causal Language Models. Causal language models are autoregressive architectures that estimate the joint probability of a token sequence by expressing it into a chain of conditional probabilities:
P(
II P(a, mz)), i=1
where P(æx (ee)) is probability of token x; given its historical context de-- Model training minimizes the cross-entropy loss maximize the likelihood of contextually consistent sequences:
L h= -=--X
log P (x; 0),
where n is the sequence length and 0 the model parameters. Perplexity is the standard evaluation metric, calculated as the exponentiated average negative log- likellhoo over the sequence:
Perplexity = cxp
log P(r: xes)
(3)
:=1
Lower perplexity signifies reduced prediction uncertainty and better data distribution alignment.
Security Definition. In cryptographic protocol design, the ideal functionality f models the desired behavior of a protocol in an idealized setting. serves as trusted third party that collects inputs from all parties, performs the computation securely, and returns the outputs. A protocol is considered secure its real-world execution computationally indistinguishable from the ideal execution with f. Due to space constraints, formal definitions are deferred to Appendix
3

---

Framework
This section details the design and implementation of FedRW. We start by formalizing the PPMPR protocol, followed by a practical construction using cryptographic primitives and parallel orchestra- tion acceleration strategy for efficiency and scalability. Finally, describe the integration of the derived weights into the FL training pipeline. An overview of key stages the FedRW framework is illustrated in Figure
Global Data Reweighting
Buseline
FedRW
Parallelism
1 1160.35s
Sio #1 12 223
Protocol Runtime
40.32s I
2.89
Perplexity
Hierarchical
2.56
Structure
I[2cC
Third-Party-Free
Figure 2: FedRW Framework: Parallel Hase- Reweighting for Efficient FL. The overview is divided into three parts: (Left) The parallel orchestration of the hhird-party-free TT protocol. (Center) The frequency-aware reweighting scheme that dynamically assigns weights (reflected by color) samples while preserving data integrity. (Right) A comparison between FedRW and the baseline approach.
4.1 Formal definition of PPMPR
Consider a federated setting with n clients PI..... ......,,... Pn. where each client P. holds local dataset X, [zi..... =mm_] consisting of m, text samples. The objective of our proposed PPMPR protocol manner, based on how often it appears
is to assign weight to each sample in a across all datasets. This functionality freMPR can be formally defined as:
(XPMPR(X1,-00 X,) + (W1,-., W.).
where W, [w....., mn]] refers to the weight vector for the samples in X,. Specifically, each sample 2² associated with an individualized weight w² reflecting its global frequency.
Subsequently, the derived weights are applied to enhance the federated training of LLMs, providing fine-grained pattern to handle duplicated data. To quantify the relative informativeness of each sample æ, we employ an intuitive yet effective heuristic: the weight w(x) is inversely proportional to its global frequency:
w(x) freqgiobal(x))
(5)
Here, [reggiobat (2) denotes the occurrence frequency of the sample x within the entire dataset, specifically, the concatenation of all clients' local datasets. This formulation naturally turns the reweighting task into a challenge of securely deriving global frequencies without revealing local data. To solve this problem, we leverage secure multi-party computation (MPC) approach.
To avoid reliance on trusted third party, the procedure is decomposed into multiple rounds of secure two-party computation (2PC), sub-issue of MPC. In 2PC protocol, two clients, P; and P,. jointly compute a specific function based on their private inputs, X, and X,, without directly disclosing the inputs each other. The functionality favc defined for this situation is:
fzvc(X,, X,) - (Č,, Cj).
(6)
where C₂ vector of length m,, containing the counts of samples in X, that are identical each sample x in X,. Since there are no privacy concerns client-side, each unique sample æ will be maintained only once X,, along with local frequency, freqx, which can easily collected and securely shared with other clients that hold the same sample. Through this iterative pairwise 2PC protocol, each client computes and obtains the global frequency for its local samples, allowing them to adjust the sample weights without exposing private data.

---

4.2 Efficient construction of PPMPR
To realize the defined functionalities, we utilize two-party private set intersection (PSI) as the cryptographic foundation of our 2PC protocol. PSI enables two parties to compute the intersecting elements of their datasets without revealing any additional information beyond the agreed-upon rules. The protocol involves only the two participating parties as sender and receiver. In the semi-honest setting, the protocol reveals solely the shared samples and how often they appear in each local dataset, as specified by f2Pc. The detailed procedure is outlined in Protocol
Table 1: Protocol N2ce in the comi-honest setting model
Protocol Two-Party Computation (2PC)
Input:
Client Pi holds input Xı (ug }. and client P2 holds input X2 (zi,... ",..,, Lm J. Both input sets are preprocessed local data samples.
Output:
Pi outputs C₁ and P2 outputs C2, as defined in Eq. (6).
Protocol:
1. P1 and P2 initiate two-party Private Set Intersection (PSI) protocol, where:
Pi acts as sender, and receives nothing.
P2 acts as receiver, and receives the intersection set I of Pi Pi's data.
2. For each sample x in I, P2 extracts the local frequency freqx, (It), and creates the frequency F2. P2 then sends I and F2 to Pt.
3. Upon receiving I and F22, P1 extracts the local frequency freqx, (z), and creates the frequency set F1. P1 then sends Fi to Ps.
4. P1 outputs C1 [[rexx [freqx, (4?), fregx, (2
(1 )1 and P3 outputs C2
The 2PC protocol provides an efficient and secure method for pairwise exchange sample frequencies between clients. We now extend this building block to construct the full PPMPR protocol.
Table 2: Protocol ПPрPP in the semi-honest setting model
Protocol Full Protocol (PPMPR)
Input:
Each client P. holds a local dataset X; = {xf,.... .efi.....ae sm where i E {l...., n). All datasets are preprocessed.
Output:
Each P, outputs frequency vector C₁ containing freggtobba (1) for every x in X,, as defined in Eq. 5).
Protocol:
Each P. initialize C, using its local frequencies freqx, (z) for all X,.
2. Each P; performs N2pe once with every other client P3 (all n of them).
After each run, P; outputs C, and updates its global vector: C, + C, + C.
3. After n rounds. P. outputs the final C.
As presented in Protocol Each client P; starts by initializing its frequency vector with local counts, then iteratively executes 1lzc with every other clients to progressively build C, the vector of global frequencies. The formal security definitions and proofs are available in Appendix
4.3 Parallel Acceleration
The formula "N choose 2" represents that the full protocol involves each pair of the clients performing H2zc, which results in (*) executions, leading to an overall time complexity of ((n?) when run sequentially. This quickly becomes inefficient as a growing number of clients. To address this scalability bottleneck, we introduce parallel orchestration strategy that reorganizes the execution schedule to minimize overall runtime. We start with toy example where n 8 in Figure and the detailed procedure is provided in Appendix B
5

---

(Ps, (t,
Tlpe(Ps P.)
Tlpc(Ps
Iare Pa)
Maro(P,,
Level
(lpc(P; Pt)
Teee(Ps Ps)
Tlac(Pt Ps)
Dape: Pg)
Dare Ps)
(P,
Dare(P, Ps)
(avo(P.,P)
Iwu(P,
Mav((Pe Ps)
Level
Tp Pd)
Up Ps)
Level 1
Ha.e(P. Ps)
(Ps
Pg.
Taro(Ps Ps)
Figure 3: A toy example for the parallel orchestration when 8.
The key insight is that multiple I2p instances can be performed concurrently, provided their participating sets do not overlap. As shown in Figure 3 from the left-hand side of level 1, adjacent pairs of clients perform 1lare independently. At the next level, these client pairs grouped into disjoint blocks (e.g.. (Pr. P2] with P3. P.),, and inter-block protocols are executed in parallel. This hierarchical process forms progressively larger blocks, such [P1, P2, P3. Pa) and (Ps, Ps. P,, Ps) at level 3. The structure resembles a binary tree and can be viewed as recursive wa- merge that manages all (>) sub-protocols efficiently.
To organize this orchestration, the client pairings at each level are structured into pairing matrices, with partial examples highlighted in the dashed-line areas of Figure When is power of two, these matrices perfectly arrange all Ilre executions, maximizing parallelism. Each matrix is constructed by element-wip wise pairing of two client blocks. For instance, at level 3, matrix M3 is formed as follows:
á = (1, 4),
T = (5, 6, 7,8)
8 MotL(b,0) 0),
rown {(a), D/) 1, 2,344)
P < RotL(b, 1),
rows + ((7,, 87 1, 2,2,3,4)
F < RotL(E, 2),
row3 < ((at, 00); 1,
σ 3)), TOWA < ((T,, 1,2,3,4)
Here, ă and 5 contain the indices of clients from interacting blocks. In each step, D' is generated by eyclically left-shifting 5 by & positions using RotL(b, k). Client pairs then formed by matching elements from á; and 17, allowing I[2c to run concurrently across each row. For 2 the hierarchical structure remains valid by simply ignoring the unused blocks, thus maintaining optimality and full coverage of client interactions. This parallel approach reduces the total runtime complexity of the full protocol from O(n?) to O logg n 1).
< 2mm
4.4 Enhanced Training
To integrate duplication awareness into model optimization, FedRW employs a frequency based sample reweighting strategy. Given the global frequency vector č, where each element represents the occurrence count of local sample across all clients, the corresponding weight vector W is defined as:
W
(8)
In(C + +5
Here, E is small constant for numerical stability. This formula penalizes frequent samples using logarithmic function, reducing their impact on optimization without complete exclusion. The logarithm, shifted by 1, ensures that the weights decrease moderately and prevents extreme weights for infrequent samples (e.g., when č = 1). Compared to linear or hard-threshold formulas, this scheme offers a smoother and adaptive adjustment across varying duplication levels, leveraging the observation that moderate redundancy promote better model generalization.

---

These derived weights, W, are then applied during training via a sample-wise reweighted loss. Instead of modifying the model architecture, each sample's loss contribution is rescaled by its assigned weight. For batch of B samples, with W; as the weight and p!!) as the token-level average loss of the i-th sample, the aggregated batch loss calculated as:
Dratch EE W. 20 B Σ W;
This method diminishes the impact of frequent samples while balancing the influence of less frequent underrepresented ones. By adapting statistical redundancy across clients, it preserves informative samples and mitigates overfitting to specific patterns. This provides a lightweight yet effective sample-level reweighting mechanism, particularly advantageous in federated settings with skewed or redundant data. Model updates are then aggregated using the standard FedAvg 24 algorithm.
5 Experiments
5.1 Experimental Settings
Environments. For protocol evaluation, we implement the H2ee prototype based on 3 and benchmark its runtime under varying configurations. For FL experiments. we use eight public datasets: Haiku [34), Rotten Tomatoes (35), Short Jokes (36), Poetry (37], IMDB (38), Sonnets [39], P4ays[40), and Twitter Sentiment Analysis(41] To simulate redundancy, duplicates are synthetically added into the training set at different rates and distributed uniformly across 10 clients. The final performance of models is evaluated using perplexity on the test sets. More details can be found in
Appendix C
Baseline Setting. We choose EP-MPD 25 as the primary baseline, the most state-of-the-art hard deduplication solution for federated LLM training via trusted third party. We follow their original experimental settings and directly use their reported runtime and perplexity results for comparison.
5.2 Main Results
Preprocessing. This part evaluates the efficiency and scalability of our proposed PPMPR pro- tocol against the baseline across three key factors: dataset size, client number, and duplication percentage, with the results shown in tables 3 and 4 and figure
Table 3: Effect of dataset size with 30% duplication percentage on H2pc running time.
Method
Protocol Running Time (ms)
Dataset Size
210
212
211
216
218
220
Setup
47. 07.020.002
48.6 48.640.003
54.640.078
76.020.178
167.8 200.4788
715.8+1.811
Execution
0.4 Lo.00.
1.0.0.006
5.9 1.919.019
23.5 10.325
118.7 11.78
713.3 713.317.60
-opc-total
47.410.055
49.710.118
60.9 10.43
11.500
291.8 16.25
147.18127.141
The runtime of the basic 2PC protocol increases with dataset size due to the underlying frequency counting mechanism. For small datasets (e.g., 210_214), runtime differences are minimal, mainly because the cryptographic setup overhead of two-party PSI is significant compared with the actual execution time, which grows linearly with the dataset size. Noticeably, the execution time scales rapidly beyond a certain dataset size, and begins to dominate the total runtime. For instance, processing 22 samples per client takes approximately 1.45 seconds, as illustrated in table 3
Table examines how the duplication percentage affects N2v runtime when each client holds 29 samples. The results show a negligible effect, with the protocol maintaining near-constant performance even at extreme duplication levels. For instance, with 90% duplication, the runtime remains stable at 0.666 seconds, differing by only 6.9% from the 0.620-second runtime with 10% duplication. These small variations are attributed to the increased amount of frequency information exchange (Fi, F2 in B the intersection (I in 3 cardinality grows.
7

---

Table Effect of duplication percentage with 21 data size in each client on U2pe running time.
Method
Protocol Running Time (ms)
Duplication Percentage
10%
30%
50%
70%
90%
Setup
g.092
322.440.966
337.4 10.988
339.8 1391811.015
3/3.440.992
Execution
265.910.250
293.3 13.456
284.9
304.2
310.6
ttoall
12.21
626.9 13.091
633.6:144
656.5 18.177
665.9g18.913
Figure 4 analyzes the effect of client number the runtime of the baseline and PPMPR under consistent experimental settings. The baseline proposes two variants that trade off performance and leakage, and utilize EP-MPD(), which prioritizes efficiency while introducing more privacy leakage. PPMPR demonstrates superior efficiency and scalability, achieving a 17.61x to 28.78 speedup over the baseline with 10 to 50 clients. This advantage is primarily due to the efficient 12 protocol as its basic component, with the parallel orchestration strategy, which reduces the overall computational complexity to 0(m 1) for E (2m-1, 2m clients.
Efffect Client Number on Protocol Rumning Time
Effect of Dataset Size on Protocol Running Time
1200
1000
PMMB
Pritccal Rarning Time (4)
Protccal Ranning Time (5)
$06422
100
600
500-33
400
300.9
10
1.99
1922
12.42
35:31
40.32
21s
2/1
24
2
2.7
2
Size (Number of samples) per client
Figure 4: We evaluate the effect of client number and dataset size protocol running time. For clients (10-50) with 219 data per client and 30% duplication, PPMPR exhibits 17.61-28.7 78x speedup. For 50 clients, PPMPR outperforms the baseline by 4.09-28.76 7 with increasing dataset size.
Furthermore, we evaluate the impact of datasct size per client with 50 clients. While PPMPR initially lags EP-MPD on smaller datasets, its parallel strategy quickly becomes dominant as data size scales. With 21 samples per client, PPMPR achieves a 23.04x speedup. This dual advantage in scaling across both client counts and data volumes positions PPMPR as highly efficient and scalable solution for real-world federated environments.
Model Performance. This section evaluates model performance across eight text datasets with diverse linguistic structures. To simulate realistic data redundancy in FL, we introduce different levels of artificial duplication (10%, 20%, and 30%) into the training data. Initially, we assess the robustness of each method under two foundational models utilized in the baseline, GPT-2 Large and DistilGPT2 43 with perplexity as the evaluation metric.
Table Model perplexity (4) on test under various duplication settings with GPT-2 Large
Method
Dataset
Duplication
Haiku
Rotten Tomatoes
Short Jokes
Sonnets
Percentage
30%
20%
10%
30%
20%
10%
30%
20%
10%
30%
20%
10%
Raw Data
3.26
3.25
2.98
2.65
2.61
2.53
4.11
4.03
3.94
4.39
4.34
4.31
Baseline
2.89
2.21
3.79
4.35
FedRW (Ours)
2.56
2.67
2.69
1.61
1.63
1.64
3.15
3.17
3.17
4.07
4.26
4.26
8

---

As detailed in Table FedRW consistently outperforms the baseline across all datasets and duplication levels with GPT-2 Large. The improvement evident on the highly structured Sonnets and Haiku datasets, where FedRW achieves relative perplexity reductions of up to 6.44% and 11.42% at 30% duplication, respectively. The strict structures of these datasets likely worsen the negative impact of redundancy, highlighting FedRW's ability to preserve content diversity and reduce overfitting through adaptive reweighting.
Similar trends are observed on less structured datasets. For Short Jokes, FedRW reduces perplexity from 3.79 to 3.15 under 30% duplication, despite its high lexical diversity. Likewise, on Rotten Tomatoes, which is composed of short, opinion-based based reviews often prone to duplication, perplexity decreases from 2.21 to 1.61. These results indicate FedRW's effectiveness even when redundancy arises from stylistic repetition.
Furthermore, FedRW exhibits robustness varying duplication rates. While the baseline's hard filtering yields fixed perplexity (10% to 30% duplication), FedRW maintains stable or slightly improved performance. For instance, perplexity Short Jokes remains constant at 3.17, and on Haiku, it decreases from 2.69 to 2.56. These observations align with prior research suggesting that controlled repetition can enhance generalization by reinforcing key training patterns M4 Instead of discarding duplicates, FedRW adaptively reweights updates to retain informative redundancy, as seen in datasets where increased duplication slightly improves performance. This suggests that effectively managed redundancy can amplify useful linguistic or semantic signals, underscoring FedRW's 's ability adapt to varying levels of data noise.
Table 6: Model perplexity (L) on test set under 30% duplication percentage with DistilGPT2
Dataset
Method
Haiku
Short Jokes
Rotten Tomatoes
IMDB
Poetry
Sonnets
Plays
Raw Data
3.70
2.07
1.78
7.17
2.84
5.87
15.07
Baseline
3.67
2.07
1.77
7.25
3.01
6.08
16.09
FedRW (Ours)
3.65
2.08
1.75
7.00
2.66
5.75
14.50
To evaluate FedRW's generalizability resource-limited scenarios, we evaluate it with DistilGPT2, smaller version of GPT-2 suitable for FL with limited computational resources. Despite its reduced size, which makes it more vulnerable to the negative effects of data duplication, Table shows that FedRW consistently maintains or slightly improves performance across various datasets.
On datasets like Haiku and Short Jokes, perplexity remains similar across the three methods. However, more noticeable variances emerge on Sonnets, Poetry, and Plays, where the baseline sometimes underperforms even the undeduplicated data. This could be due the literary structure and the sparse samples of these datasets. As noted in the baseline, hard deduplication considerably reduces the training samples (e.g., Poetry: 526 to 405; Plays: 542 to 417), potentially increasing training variance, especially for distilled models. By contrast, FedRW's flexible and adaptive approach aims to retain useful instances when handling excessive redundancy. This reweighting strategy provides a more stable training signal to preserve the integrity of sparse datasets, leading to improved generalization.
Table 7: Model perplexity (4) on test set under 30% duplication percentage mainsteam models
Dataset
Model
Method
Haiku
Jokes
Rotten
Poetry
Sonnets
Plays
Baseline
2.47
2.61
1.71
2.54
4.07
8.21
Qwen3-0.6B
FedRW (Ours)
2.36
2.44
1.59
2.21
3.62
7.23
Qwen2.5-0.5B.
Baseline
2.21
2.48
1.58
2.28
4.11
11.77
Instruct
FedRW (Ours)
2.12
2.36
1.55
2.03
3.84
9.92
1lama-3.22-B.
Baseline
2.14
2.34
1.65
2.39
4.11
18.35
Instruct
FedRW (Ours)
2.09
2.21
1.54
1.99
4.00
16.03

---

To further validate FedRW's applicability beyond the GPT-2 family, we evaluate the performance on three representative modern models with diverse architectures: Qwen3-0.6B [45] Qwen2.5-0.5B-
Instruct [4]] and Llama-3.22 3.2-1B-BBn B-Instrucc [47] The results in dablsr]eemonstrraee that data redundancy remains a substantial challenge even for these contemporary architectures. FedRW robustly maintains its advantage in mitigating the impact of redundancy on model performance, particularly under challenging conditions such as data complexity or sparsity. For instance, FedRW achieves an average relative perplexity reduction of approximately 13.43% on the Plays dataset across the three models,
Table 8: Model perplexity (4) on test set under 30% duplication percentage on larger models
Dataset
Model
Method
Haiku
Jokes
Rotten
Poetry
Sonnets
Plays
Twitter
Qwen2.5-3B-
Baseline
1.69
2.09
2.20
2.33
4.14
9.17
3.35
Instruct
FedRW (Ours)
1.55
1.94
2.01
1.85
3.29
7.53
2.46
Qwen2,5-7B-
Baseline
1.68
2.07
1.74
2.09
4.52
8.82
2.24
Instruct
FedRW (Ours)
1.49
1.95
1.61
1.81
3.43
6.54
1.35
With increasing model capacity, memorization of specific patterns due to duplication becomes more pronounced and critical, leading overfitting, degraded generalization, and increased privacy risks 12]. To assess the issue, we conduct experiments on two large-scale models from the Qwen family: Qwen2.5-3B-Insteee Instrrutt and @wen2.5-7BBInstteee perplexity on certain datasets, the results in Table g show that FedRW sustains its performance advantage over the hard deduplication method. Under the extensive near-duplicate contents in Twitter, FedRW achieves a relative reduction of approximately 26.57% in perplexity compared to the baseline.
[46). While larger models may exhibit lower
Table 9: Model Perplexity (4) on test set on the Non-III settings
Method IID
Quantity Skew
Label Skew
Feature Skew
Baseline
1.71
2.02
2.44
3.43
FedRW (Ours)
1.59
1.96
1.66
2.70
To evaluate the efficacy of FedRW under Non-IID data distributions, a major challenge in FI., we conduct experiments on Owen3-0.66 under three scenarios: Quantity Skew, Label Skew, and Feature Skew. For Quantity and Label Skew, we categorized the Rotten Tomatoes dataset by the binary (0/1) labels across 5 clients, with proportions set to [40%. 20%, 20%, 10%, 10%] and label distributions as [(0.5, 0.5), (0.6, 0.4), (0.4, 0.6), (0.9, 0.1), (0.1, 0.9)], respectively. To simulate Feature Skew, we allocate Poetry, Sonnets, and Plays to separate clients, as these datasets differ distinctly in terms of text structure, sentence length, and lexical and syntactic complexity. The results in Table/cooffrm FedRW's robustness to provide stable training process across heterogeneous data distributions.
6 Conclusion
In this work, we introduce FedRW, novel and principled framework designed tackle the widespread challenge of data duplication in federated language model training. At its core is PPMPR, a secure and efficient protocol for data reweighting. PPMPR enables soft deduplication methods without compromising data privacy introducing substantial computational and communication costs. Crucially, our protocol works without a trusted third party, enhancing security and achieving notable improvements in efficiency and scalability.
Our comprehensive experiments across diverse text datasets show that FedRW consistently improves model generalization under redundancy, outperforming the state-of-the-ee art method across varying duplication levels, dataset settings, and model configurations. Beyond simply discarding duplication, FedRW effectively harnesses redundancy to foster more robust representation learning. These compelling results establish FedRW as a practical, privacy-prese prieacy-preservin solution for robust federated training in noisy data scenarios. Moreover, its lightweight modular design allows for seamless integration into broader applications, including multimodal learning pipelines and flexible reweighting strategies, highlighting its potential fundamental building block for future federated LLM systems.
10

---

Acknowledgements
This work is supported in part by the National Key Research and Development Program of China (Grant No. 2020YFA0712300), in part by the National Natural Science Foundation of China (Grant No. 62132005, 62172162), in part by Shanghai Trusted Industry Internet Software Collaborative Innovation Center, and in part by Fundamental Research Funds for the Central Universities. This work is also supported by the Postdoctoral Fellowship Program of CPSF under Grant Number GZB20250407.
References
[1] L. Ouyang. J. Wu. X. Jiang. D. Almeida, C. Wainwright, P. Mishkin, C. Zhang. S. Agarwal, K. Slama, A. Ray, et al. "Training language models to follow instructions with human feedback." Advances in neural information processing systems, vol. 35, pp. 27730-27744, 2022.
[2] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhar- gava, S. Bhosale, et al., "Llama Open foundation and fine-tuned chat models," arXiv preprint arXiv:2307.09288 2023.
[3] M. Hanna, O. Liu, and A. Variengien, "How does gpt-2 compute greater-tha??: Interpreting mathematical abilities in pre-trained language model," Advances in Neural Information Processing Systems, vol. 36, pp. 76033-76060, 2023.
[4] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidr, S. Altman, S. Anadkat, al., "Gpt-4 tochnical report," arXiv preprint wrXiv:2303.0877 2023.
[5] A. Liu, B. Feng. B. Xue, B. Wang, B. Wu, C. Lu. C. Zhao, C. Deng, C. Zhang. C. Ruan, al. "Deepseek-v3 technical report," arXiv preprint arXiv:2412. 19437, 2024.
[6] T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa, "Large language models are zero-shot reasoners," Advances in neural information processing systems, vol. 35, pp. 22199-22213, 2022.
[7] S. Wu, O. Irsoy, S. Lu, V. Dabravolski, M. Dredze, S. Gehrmann, P. Kambadur, D. Rosenberg, and G. Mann, "Bloomberggpt: A large language model for finance," arXiv preprint N.Xv:2333.115644 2023.
[8] G. Wang, Y. Xie, Y. Jiang, A. Mandlekar, C. Xiao, Y. Zhu, Fan, and A. Anandkumar, "Voyager: An open-ended embodied agent with large language models," arXiv preprint arXiv:2305.1629) 2023.
[9] Luo, 7. Pang, Y. Zhang. T. Wang, L. Wang, B. Dang, J. Lao, J. Wang, Chen, Y. Tan, et al., "Skysensegpt: A fine-grained instruction tuning dataset and model for remote sensing vision-language understanding." arXiv preprint 1:Xiv:2406.11000 2024.
[10] C. Raffel, N. Shazeer, A. Roberts, Lec, Narang, M. Matena, Y. Zhou, W. Li, and P. Liu, "Exploring the limits transfer learning with a unified text-to-text transformer," " Journal of machine learning research, vol. 21, no. 140, pp. 1-67, 2020.
[11] K. Lee, D. Ippolito, Nystrom, C. Zhang, Eck, C. training data makes language models better," arXiv preprint arXiv:2107.0649 2021.
and N. Carlini, "Deduplicating
[12] N. Carlini, D. Ippolito. M. Jagielski, K. Lee, F. Tramer, and C. Zhang, "Quantifying memorization across neural language models," in The Eleventh International Conference on Learning Representations, 2022.
[13] F. Shayegani, M. A. A. Mamun, Y. Fu, P. Zaree, Y. Dong, and N. Ahu-Ghazaleh, "Survey of aulnerabilities in large language models revealed by adversarial attacks," " arXiv preprint arXiv:2.3 10.444444 2023.
[14] G. Qi, Y. Chen, X. Mao, B. Hui, X. Li, R. Zhang, and H. Xue, "Model inversion attack via dynamic memory learning"" in Proceedings of the 31st ACM International Conference on Multimedia, pp. 5614 5622, 2023.
[15] J. Mattem, F. Mireshghallah, Z. Jin, B. Schölkopf, M. Sachan, and T. Kirkpatik,,,,, "Membership infer- ence attacks against language models via neighbourhood comparison," arXiv preprint arXiv: 18462, 2023.
[16] G. Penedo, Q. Malartic, D. Hesslow, R. Cojocaru, A. Cappelli, H. Alobcidli, B. Pannier, F, Almazrouci, and J. Launay. "The refinedweb dataset for falcon 0m: outperforming curated corpora with web data, and web data only." arXiv preprint arXiv:2306.01116 2023.
[17] A. Z. Broder, "On the resemblance and containment of documents." in Proceedings. Compression and Complexity of SEQUENCES 1997 (Cat. No. 97TB100171), pp. 21-29, IEEE, 1997.
11

---

[18] T.-Y. Lin, P. Goyal, R. Girshick. K. He. and P. Dollár, "Focal loss for dense object detection," in Proceedings of the IEEE international conference on computer vision. pp. 2980-2988, 2017.
[19] M. Ren, W. Zeng. B. Yang, and R. Urtasun, "Learning to reweight examples for robust deep learning,"" in International conference machine learning. pp. 4334 4343, PMLR, 2018.
[20] S. M. Xie. H. Pham, X. Dong, N. Du, H. Liu, Y. Lu, P. S. Liang, Q. V. Le, T. Ma, and A. W. Yu, "Doremi: Optimizing data mixtures speeds up language model pretraining," Advances in Neural Information Processing Systems, vol. 36, pp. 69798-69818, 2023.
[21] S. M. Xie. S. Santurkar, T. Ma. and P. S. Liang, "Data selection for language models via importance resampling," Advances in Neural Information Procexsing Systems, vol. 36, pp. 34201-34227, 2023.
[22] N. He, W. Xiong, H. Liu, Y. Liao, L. Ding, K. Zhang, G. Tang, X. Han, and W. Yang. "Softdedup: an efficient data reweighting method for speeding up language model pre-training." " arXiv preprint arXiv:2407.06654. 2024.
[23] C. Hou, A. Shrivastava, H. Zhan, R. Conway, T. Le, A. Sagar, Fanti, and D. Lazar, "Pre-text: Training language models on private federated data in the age of Ums,," arXiv preprint arXiv:2406.02958, 2024.
[24] B. McMahan, E, Moore, D. Ramage, S. Hampson, and B. A. Arcas, "Communication-effiiiee learning of deep networks from decentralized data," in Artificial intelligence and statistics, pp. 1273-1282, PMLR, 2017.
[25] A. Abadi, V. A. Dasu, and S. Sarkar, "Privacy-preservingg data deduplication for enhancing federated learning of language models," arXiv preprint arXiv:2407.08152 2024.
[26] S. Kamara, P. Mohassel, M. Raykova, and S. Sadeghian, "Scaling private set intersection to billion-element sets." in Financial Cryptography and Data Security: 18th International Conference, FC 2014, Christ Church, Barbudox, March 3-7, 2014, Revised Selected Papers 18. pp. 195-215, Springer, 2014.
[27] U. Manher and G. Myers, "Suffix arryys: a new method for on-line string searches," siam Journal on Computing, vol. 22, no. pp. 935-948, 1993.
[28] C. Coleman, C. Yeh, S. Mussmann, B. Mirzasoleiman, P. Bailis, P. Liang, J. Leskovec, and M. Zaharia, "Selection via proxy: Efficient data selection for deep learning," arXiv preprint arXiv: I 1829, 2019.
[29] M. Weber, D. Fu, Q. Anthony, Y. Oren, S. Adams, A. Alexandrov, X. Lyu, H. Nguyen, X. Yao, V. Adams, er al., "Redpajama: an open dataset for training large language models," Advances neural information processing systems, vol. 37. pp. 116462-116492, 2024.
[30] X. Zhang, A. Lv, Y. Liu, F. Sung, W. Liu, Shang, X. Chen, and R. Yan, "More is not always hetter? enhancing many-shot in-context learning with differentiated and reweighting objectives," arXiv preprint arXiv:2501.04077 2025.
[31] Z. Lin, Z. Gou, Y. Gong, X. Liu, R. Xu, C. Lin, Yang. Jiao, N. Duan, W. Chen, et al., "Not all tokens are what you need for pretraining," Advances in Neural Information Processing Systems, vol. 37, pp. 29029-29063, 2024.
[32] S. Keelveedhi, M. Bellare, and T. Ristenpart, BaanpLESSs:(SSSrrrr... storage." in 22nd USENIX security symposium (USENIX security 13), pp. 179-194, 2013.
encryption for deduplicated
[33] P. Rindal and P. Schoppmann, "Vole-psi: fast opri and circuit-psi from vector-ole," in Annual international conference on the theory and applicatioon of cryptographic techniques, pp. 901-930, Springer, 2021.
[34] bfbarry, "Haiku dataset." /ta.p//////uu..kgle.... com/ comm/lataasets//bbrrrrr///aaaa dateea/esattrrrrr//aaaaaaaaaaaae 2021.
[35] B. Pang and L. Lee. "Seeing stars: to rating scales," in Proceedings of the ACI,, 2005.
Exploiting class relationships for sentiment cateimmiattaateggiiiaaaain with respect
[36] A. Moudgil, "Short jokes dataset." short- jokes 2017.
[37] HuggingFace and contributors, "Poetry dataset." poetry 2022.
[38] A. Maas, R. E. Daly, P. T. Pham, D. Huang, A. Y. Ng. and C. Potts, "Learning word vectors for sentiment analysis." in Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, pp. 2011.
12

---

[39] Lambent, "Shakespeare sonnets diffused dataset." http:r//hnggggggeeee..///ttttees
suakeperresounettdfffueeee 2023. Accessed: 2024-05-04.
[40] Trelis, "Tiny shakespeare
dataset."
http:///ug..ngaaeeec//ttttttttts
stny-shakespeare 2022.
[41] passionate nlp. "Twitter sentiment analysis dataset." http:://ww...aglle.co//aaaaaatte
2021.
[42] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al, "Iaggugge models are unsupervised multitask learners," OpenAl blog, vol. no. 8. 2019.
[43] HuggingFace and contributors, "Distilgp12." "ebstilgp2.""ttttss:/// htttga////aagggggffafe.e////s/ss 2019.
[44] N. Muennighoff, A. Rush, B. Barak, T. Le Scao, N. Tazi, A. Piktus, S. Pyysalo, T. Wolf, and C. A. Raffel, "Scaling dona-cont language models," Advances in Neural Information Processing Systems, vol. 36, pp. 50358-50376, 2023.
[45] A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al., "Qwen3 technical report," arXiv preprint arXiv:2505.09388 2025.
[46] A. Yang, A. Li. B. Yang. B. Zhang, B. Hui, B. Zheng, B. Yu, al., "Qwen2.5 technical report," arXiv preprint 15.1:
[47] A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan, et al., "The llaaa herd of models," arXiv e-prints, pp. arXiv-2407, 2024.
[48] R. Canetti, "Universally composable security: A new paradigm for cryptographic protocols." in Proceedings 42nd IEEE Symposium Foundations of Computer Science, pp. 136-145, IEEE, 2001.
[49] I. Loshchilov and F. Hutter, "Decoupled weight decay regularization," " arXiv preprint arXiv: 1711::711.001011
2017.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The main claims in the abstract and introduction align with the proposed method (FedRW), its design (PPMPR), and empirical results, including privacy guarantees and performance benefits. See Section
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
The discussion in Section 5.2 acknowledges limitations such as scenarios where baseline performs comparable. Section 6 and Appendix E indicate possible future developments.
Justification:
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
14

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Theoretical assumptions and proofs for PPMPR's s security are provided in Appendix A clearly defining the threat model and formal guarantees.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Sections and 5 and Appendix C detail protocol implementations, datasets, and experimental settings sufficient reproduce the results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
15

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification: All datasets are publicly available with citations provided in Appendix C We are working hard to promote the process of open source.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
publicc/guides//deeSttmt http:///uuuineeeeeeeeeaaaaaaaaaa www.aaammmiioiooooooooooooooiiie
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
//nips //ips.cc/
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
See Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
The paper reports the average performance after repeated experiments for consistency.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
16

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics htttp::/neuuipp: wcc/pubbii/Bthiccciieeeeeeeee
Answer: [Yes]
Justification: The research conducted in the paper complies with NeurIPS Code of Ethics in all aspects.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
The research conducted in the paper is to enhance training of language models, without negative societal impacts.
Justification:
federated
Guidelines:
The answer NA means that there is no societal impact of the work performed.
17

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risk.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The creators or original owners of the assets used in the paper have been appropriately recognized, and the licenses and terms of use have been explicitly mentioned and properly respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
18

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the datasets.paperssiii paperswi thcode. www.com//aaaa datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
19

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMS as any important, original, non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https://neurips: hett::///earrrpsscc//oooerrreeee cc////ooeeeeeecs//22/////.. for what should or should not be described.
20

---

Appendix
Contents
Security
21
Parallel Ochestration Algorithm
24
Experimental Details
24
D Sensitivity Analysis
25
Future Work
26
A
Security
A.1
Security Proof
Modern cryptographic protocols are typically analyzed under the simulation- security paradigm, which formalizes security by comparing a protocol's behavior in the real world to that in ideal world.
In the ideal world, trusted third party honestly executes the desired functionality. All parties submit their inputs to the trusted third party, and the trusted third party returns the correct outputs to the designated parties. In contrast, the real world involves actual protocol execution among potentially adversaries without a trusted third party.
A protocol is said to be secure if for every adversary in the real world, there exists simulator in the real world such that no external environment can tell whether is interacting with real world or an ideal functionality. This paradigm ensures that the protocol leaks no more information than what is inherently revealed by the output of the ideal functionality.
A.2 Universal Composability Model
The Universal Composability (UC) (48] framework provides a rigorous model for analyzing the security of cryptographic protocols under arbitrary adversarial conditions. It ensures that protocol remains secure even when composed with other protocols, making it robust against complex attack scenarios.
In the ideal world, all parties interact through a TTP that computes the desired functionality f. ensuring privacy and correctness. In the real world, parties execute protocol II without TTP. A semi-honest adversary A may observe internal states but not deviate from the protocol.
A protocol II is UC-secure if, for any adversary A in the real world, there exists simulator S that produces view indistinguishable from A's view the ideal world. This ensures that the protocol's behavior in the real world is as secure as the ideal world.
A.3 Threat Model
In this work, we consider a semi-honest adversary model in federated learning (FL), where all participants follow the protocol honestly but may attempt to infer additional information from observations. For the scope of this work, assume no active collusion among parties. While more active or malicious threats- such as inference, backdoor, or reconstruction attacks exist, these are considered orthogonal to the primary objective of this study.
protocol securely computes functionality f {0, 13* x 10 - f0, 13* (0,1)', where (2) For inputs yx, outputs (fi v). f2(x,)) v)) returned respective parties. Extensions to multi-party settings are implied.
A protocol II is secure against semi-honest adversaries if:
21

---

Definition (Security). For any semi-honest adversary A, there exist probabilistic polynomial-time time (PPT) simulators Sim, Simy such that:
(Sim, (z, (r., 1))) = (View) (xx y) {Sim f2 a)) =. (View2 II.A (x,, y) Ja.y"
View(I.A(z
(10)
(11)
Here, (w, f( y)) denotes view based on simulator i's imput E (z. and II's output f), View,
IIA (x, v) represents A's observation party i's view during protocol execution.
E, denotes computational indistinguishability, meaning no PPT algorithm can distinguish the two distributions.
A.4 Formal Definition of Ideal Functionality
We provide the formal definitions of the ideal functionalities employed Section as detailed
in tables 10 to [2
Table 10: Ideal functionality fTwo-Party PSI
Parameters:
Client P1 holds input X1 {at -.. J.h ). and client P2 holds input X₂ {af:... xn) Input X1 {at...., mm from PI. and X₂ fa?...., a from Pz.
Functionality:
Output X1 n N22
Table 11: Ideal functionality fare Client Pl holds input X1 rm.. and client P2 holds input X₂ {af...., ..., J. Input X1 = fat...,,zh, Im from PL. and X₂ = fa?....,. x2) from Pz.
Parameters:
Functionality:
Output C₁ and Cz.
Table 12: Ideal functionality fPPMPR
Parameters:
Each client P3 holds local dataset X; = {-f,... om where E (I, Input X; = (x1.-.., m from Pt.
Functionality:
Output C.
A.5 Security of Protocols
Theorem 1. Tzrc securely implements the ideal functionality f2rc in the semi-honest model.
Proof. As described fzec, we construct simulator to simulate the behavior of the corrupted party.
Case 1: P1 is corrupted.
The simulator Sim receives Pi's input X, and its output from f2Pc. which is G (ree, (xih))]
1. During the PSI phase, P1 acts the sender and receives nothing. As PSI protocol securely implements corresponding ideal functionality, then P₁ learns nothing about X2 beyond what is revealed by the intersection X1 X2. Sin can simulate this as empty view with its own inputs X1.
2. P1 receives the intersection set I and the frequency set F2 from Pz. Sim can construct simulated intersection I' and simulated frequency set F3 based on X, and G. For each of X1;
22

---

If freqx,(2) (2/) > 0, then zy. is added to I', and its corresponding frequency in F22 is set to freqx, (xt))-
If freqx, = 0, then =x is not in I'.
The outputs (2) in the ideal world are indistngguisaabl from the outputs (I, in the real world, as they perfectly match Pi 's output C1.
P1 sends its frequency set F1 to P2. Sim, can generate F1 using X1 and the intersection Z (or T').
The view of Pi consists of its input X1, messages sent (Fi), and messages received F2. The sim- ulated view (X,, FI, is computationally indistinguishable from the real view (1,,
Case 2: P2 is corrupted.
The simulator Simz receives P2's input X2 and its output from f2ec. which is C2 = (x
[freqx,
During the PSI phase, P2 acts as the receiver and receives the intersection set I. Given the security of the PSI protocol, P2 learns nothing about X₁ beyond what is revealed by the intersection X1 n.X2. Simy can construct simulated intersection I'' based on X2 and Cz. For each x E X2:
* If freqx, (x > 0, then 2 is added in I'. If freqx, (ag) = 0, then = is not in T'.
2. P2 sends I (or I') and F2 P1. Simy perfectly simulate this using X2 and I',
3. P2 receives Fi from P1. Sim2 can construct simulated Fi based on C₂ and I'. For each x E T', the corresponding frequency in Fj would be freqx, (z).
The view of P/2 consists of its input X2, messages sent (Z, F2). and messages received (Fi). The sim- ulated view (X2, I', F2, Fi) is computationally indistinguishable from the real view II Fi).
Since the view of both corrupted parties can be simulated given their input and output from fare. I2re securely realizes faec in the semi-honest model.
Theorem 2. pPMMP securely implements the ideal functionality SPPMPR the semi-honest model.
Proof. We construct simulator SiMPrMER for corrupted P& that receives P& 's input Xg and its final output the global frequency vector Cg. from the ideal functionality fPeMPR
P& initializes CN using its local frequencies freqx, (z). This a local computation, and SimprMPR can perform the same step.
2. Pk performs Tup with every other client Pj (for j # k). After each execution, Ps. receives vector C2 and updates Co f- C. + Cs.
For each interaction between Pa and an honest Pj. the security proof for Hrre guarantees that simulator Simarc can generate a view for Pk that indistinguishable from the real view, using only Xx and the output Čj.
Since the final C. is the sum of P& 's local frequencies and pairwise learned frequencies, the overall view of Pk is the collection of views with the I executions of Na SimyrMrR can invoke Simzre for each interaction between Pş and P, to generate a view to simulate this combination.
Since fPemer only outputs the final C. SimppMPR cannot obtain each partial Cr. However, can generate intermediate T% for each interaction such that their sum (plus the initial vector) equals the known final Cp. Given that 1l securely reveals only freqx, (æ) for intersecting samples, the exact distribution does not leak additional information Ps beyond what frrMPr allows.
3. After rounds. Ps outputs the final Cr.
23

---

The view of Pt: consists of its input X2, its initial local frequencies, and the collection of outputs from all pairwise Ugpc executions. Since each 22c is secure against semi-honest adversaries and its view can be simulated, the collection of these simulated views can be combined by SimppprR securely. Therefore, SimppMPR constructs a view for Pk that computationally indistinguishable from its view in real execution. Thus, ПPMPR securely realizes SPPMPr in the semi semi-honest model.
L
B Parallel Ochestration Algorithm
To support the parallel acceleration strategy introduced in Section 4.3
we formally describe the orchestration logic in Algorithm The algorithm organizes client pairs a structured matrix manner, ensuring that each client performs 2PC protocols with all others while maximizing concurrency. Specifically, proceeds in flog, n hierarchical levels, where clients are iteratively grouped into blocks and scheduled to engage in pairwise protocols via index cyclic rotation. The orchestration guarantees correctness while enabling efficient parallelization.
Algorithm Parallel Orchestration for Efficient Execution of PPMPR
1: Input: n clients Pi..... Pr with local datasets X1,...,
2: Output: Global frequency vectors Ci...., Cn for samples of each client
3: Initialize local frequencies: C, t freqx, (.) for all
4: Let m +- [log2 nl
Total number of levels
5: for I = 1 to m do
6: Partition clients into 2m-1 contiguous blocks of equal size
7: for all pairs of blocks (A, B) do
8:
Let + indices in A, 5 indices in B
9:
for k 0 to [b] 1 do
8' + RotL
Left-rotate indices in 5
11:
for i to | do
12:
in parallel: Tarc (Pai, P.)) to update Ca, Cg.
13:
end for end for 15: end for
14:
end for
17:
return (C,,...,c,) C..
C Experimental Details
Datasets. In this section, we summarize the detailed information of each dataset used in the span diverse text domains and reflect wide
experiment. As illustrated in Table these datasets range of structural and lexical properties. The table presents the source, sample size, average sequence length, and brief description for each dataset.
Table 13: Basic information of experimental datasets
Dataset
# Samples
Avg. Sequence Length
Description
Haiku
34
15.281
100
Short-form structured 3-line poems
Short Jokes
231,657
100
Concise User
short jokes
Rotten Tomatoes
351
10.662
200
Movie review snippets expressing sentiment
IMDB 88
99,999
500
Full-length movie reviews with richer narrative structure
Sonnets
460
400
William Shakespeare's 14-line poems
Poetry 37
573
1000
Modern and classic free-form poems by various authors
Plays 40
521
1000
Dramatic scripts from William Shakespeure dialogic structure Tweets labeled with sentiment categories
Twitter 41
74,000
For all datasets, we adopt standard 80/20 train/test split. For movie review datasets, only the review texts are retained, and the sentiment labels are discarded during training. For the Short Jokes dataset, we randomly sample 50,000 entries to ensure tractable training time across 10 clients. In cases where

---

datasets such as IMDB already contain a predefined test set, we merge the original training and test partitions, shuffle the combined set, and then re-split according to the 80/20 ratio.
Environments. We conduct all secure protocol procedures, including 12 and HPPMMRR virtualized server equipped with a 4-core Intel Xeon 2.20GHz CPU and 32GB RAM. For model training. we utilize machine with 20-core Intel Xeon Platinum 8457C CPU, 200GB RAM, and an NVIDIA H20 GPU with 96GB memory. All software is executed under the Linux environment. Each experiment in preprocessing is repeated four times, and report the average performance for consistency.
Hyperparameters. adopt FedAvg as the underlying federated optimization algorithm. For GPT-2 Large and DistilGPT2, we train each client for 1-2 and 1-5 local epochs, respectively, until convergence, with total of 3-5 communication rounds. For Qwen3-0.6B, Qwen2.5-00 and Llama-3.2- 1-- B-- we train each client for 2 local epochs with total of 2-5 communication and Owwe2.55-7B-Instree
we train each client for 1-2 local epochs with a total of 1-2 communication rounds to avoid overfitting. The models are optimized using AdamW [49] with learning rates ranging from 1-5 x 10 A linear warm-up schedule is applied, reserving 10% of training steps for warm-up. To stabilize training, apply f2-norm gradient clipping with threshold of 1.0. The maximum sequence length is set between 50 and 1000, depending on the dataset, and batch sizes range from 2 to 8 with gradient accumulation steps adjusted accordingly to maintain effective batch size.
rounds. For Srrenn.5-3--Insteee
Baseline. We follow the baseline implementation proposed in [25]. which proposes hard dedu- plication approach by pre-filterng duplicated training samples. Specifically, each client performs local deduplication to remove identical samples, which assumes that redundant data uniformly detrimental, and the resulting datasets are used to train the model without further adjustment.
To ensure fair comparison, we utilize the official open-sourced code apply the same preprocessing pipeline and training settings as in FedRW. All datasets, tokenization schemes, model architectures, and evaluation metrics remain consistent across the baseline and our proposed method.
D Sensitivity Analysis
The discussion sensitivity analysis focuses the learning rate to assess FedRW's robustness. The analysis of epochs is omitted as we typically utilize a small number as standard practice to prevent overfitting.
We evaluated the model perplexity DistilGPT2 and Owen2.5-0.5B-Imm Instruct under learning rates of le-3, 5e-4, 3e-4, le-4, 5e-5, and 3e-5. We selected the Plays dataset to investigate FedRW's generalizability, as exhibited significant performance gap in the main results.
Table 14: Model perplexity (L) on Plays test set across various learning rates
Learning Rate
Model
Method
le-3
5e-4
3e-4
le-4
5e-5
3e-5
Baseline
16.12 14.42
16.38
16.13
14.21
15.07
14.18
DistilGPT2
FedRW (Ours)
14.79
14.74
13.17
14.50
12.76
Qwen2.5-0.5B-
Baseline
12.86
11.07
10.63
11.50
11.77
10.67
Instruct
FedRW (Ours)
11.48
9.35
8.15
8.14
9.92
9.81
As shown in Table [4
FedRW robustly maintains its superior performance compared to the baseline and exhibits stable training behavior across the entire range of tested learning rates. This confirms that FedRW's advantage is not overly sensitive to the learning rate selection.
25

---

E Future Work
Advanced Paradigms. FedRW's lightweight, modular design enables seamless integration into broader applications, including multimodal learning pipelines and flexible reweighting strategies. Integration with personalized FL (e-g., diverse model architectures personalization strategies) and dynamic client adaptation (where clients join, leave, or exhibit varying computational capabilities) are also valuable aspects for future research.
Optimizations. Addressing semantic redundancy is significant issue in lcage-scale real-world corpora for LLMs. It is prospective leverage the representation learning capability of transformer- based architectures to extract semantic duplication.
Adversarial Security. FedRW primarily operates under semi-honest threat model, which standard and foundational for practical malicious adversaries would be an interesting research direction. This could involve integrating mechanisms like Differential Privacy on sample frequencies utilizing Zero-Knowledge Proofs to verify client consistency during pre- pre-processing and training. These potential schemes trade off between model accuracy, data privacy, and computational overhead.
protocols. Extending FedRW to resist
26


Reviews:
[{"review_id": "TAllDZih7L", "reviewer": "Reviewer_L3R8", "summary": "This paper proposes FedRW, a novel framework for privacy-preserving soft deduplication in federated LLM training. The PPMPR protocol avoids the need for a trusted third party, scales efficiently through a parallel orchestration strategy, and integrates sample-level loss weighting in training.", "questions": "Check Weaknesses", "limitations": "Yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\nThe technical contributions are clearly defined, and the paper is well-written with a logical structure.\n\nWeaknesses\n\n1.\tIt would be helpful if the authors could include training loss curves or convergence comparisons to illustrate whether the proposed reweighting mechanism slows down or accelerates model convergence relative to deletion-based approaches.\n\n2.\tThere are a few writing-related issues that could improve the clarity of the manuscript. First, using vague terms such as “nuanced” is discouraged in scientific writing, as it does not convey precise meaning. Second, Figures 1 and 2 are visually engaging but lack adequate explanation of key elements. Third, the system formulation in Section 3 is not fully adapted to the federated learning setting, it would benefit from a clearer illustration.\n\n3.\tThe evaluation is limited to synthetic duplication. It would strengthen the paper to include results or discussion on how FedRW performs in real-world scenarios with naturally occurring duplication, such as instruction-tuning data, which often contains repeated patterns.\n\n4.\tThe protocol assumes a semi-honest threat model. However, in practice, malicious clients may attempt to manipulate the protocol by misreporting sample frequencies or injecting misleading data into the intersection process. It would be valuable to discuss whether FedRW could be extended or adapted to remain robust under such adversarial behavior.", "was_revised": true, "final_justification": "Thanks for the clarification. I will remain positive and stick with my original score.", "rebuttal": "We appreciate the reviewer for the valuable and constructive comments. We are encouraged by the positive feedback on **novelty**, **efficiency and effectiveness**, **privacy and security guarantees**, **significance of topic**, and the overall clarity and logical structure of the paper. We hope the following discussions can address the reviewer's remaining questions.\n\n**1. Convergence Comparisons.**\n\n**We appreciate your suggestion and have included the average training loss of FedRW compared with the baseline under 30% duplication in the tables.**\n\n+ GPT-2 Large\n    \n    | Dataset         | Method     | Round 0 | Round 1 | Round 2 | Round 3 |\n    |-----------------|------------|---------|---------|---------|---------|\n    | Haiku           | Baseline   |   **1.842** |   0.839 |   0.806 |   0.811 |\n    |                 | FedRW (ours)|   4.671 |   **0.835** |   **0.725** |   **0.705** |\n    | Rotten Tomatoes | Baseline   |   **1.124** |   0.444 |   0.426 |   0.379 |\n    |                 | FedRW (ours)|   3.038 |   **0.425** |   **0.378** |   **0.375** |\n    | Short Jokes     | Baseline   |   **0.994** |   0.720 |   0.622 |   0.641 |\n    |                 | FedRW (ours)|   1.263 |   **0.679** |   **0.595** |   **0.547** |\n    | Sonnets         | Baseline   |   **2.835** |   1.466 |   1.366 |   1.730 |\n    |                 | FedRW (ours)|   4.329 |   **1.349** |   **1.243** |   **1.168** |\n\n\n+ DistilGPT2\n    \n    | Dataset         | Method       | Round 0 | Round 1 | Round 2 | Round 3 |\n    |-----------------|--------------|---------|---------|---------|---------|\n    | Haiku           | Baseline     |   3.225 |   1.283 |   1.267 |   1.274 |\n    |                 | FedRW (ours) |   **2.950** |   **1.274** |   **1.255** |   **1.252** |\n    | Short Jokes     | Baseline     |   1.511 |   0.751 |     -   |     -   |\n    |                 | FedRW (ours) |   **1.388** |   **0.745** |     -   |     -   |\n    | Rotten Tomatoes | Baseline     |   1.771 |   0.607 |     -   |     -   |\n    |                 | FedRW (ours) |   **1.338** |   **0.437** |     -   |     -   |\n    | IMDB            | Baseline     |   4.234 |   1.964 |     -   |     -   |\n    |                 | FedRW (ours) |   **3.838** |   **1.939** |     -   |     -   |\n    | Poetry          | Baseline     |  10.858 |   2.989 |   1.313 |   1.107 |\n    |                 | FedRW (ours) |   **9.908** |   **1.999** |   **1.122** |   **1.049** |\n    | Sonnets         | Baseline     |  17.589 |   3.443 |   2.019 |   1.870 |\n    |                 | FedRW (ours) |  **15.901** |   **3.048** |   **1.971** |   **1.802** |\n    | Plays           | Baseline     |  23.848 |   3.379 |   2.885 |   2.815 |\n    |                 | FedRW (ours) |  **20.460** |   **3.234** |   **2.854** |   **2.767** |\n\n\nThe datasets, including *Short Jokes*, *Rotten Tomatoes*, and *IMDB*, were trained for fewer rounds to mitigate overfitting. Notably, on GPT-2 Large, FedRW exhibited a higher initial loss compared with the baseline, whereas the DistilGPT-2 model showed a lower one. This consistency is related to the loss reweighting mechanism and the variance of model capacity. In summary, FedRW demonstrates **faster model convergence** and **better final performance**. \n\n\n**2. Writing-related Issues.**\n\n**We sincerely appreciate your thorough reading, and we will revise the issues according to your comments in the final version.**\n\nTo further clarify Figures 1 and 2, we offer the following supplementary explanation:\n\n+ Fig. 1: **(a)** illustrates the inherent challenges of global deduplication in a decentralized setting, where privacy constraints prevent direct data sharing.  \n**(b)** describes the paradigm adopted by the state-of-the-art solution (also used as our baseline), which relies on a trusted third party for deduplication over encrypted data, followed by decryption. The right part of Fig.1 indicates that this pattern may lead to the loss of key training samples.\n+ Fig. 2: The left panel of Fig.2 illustrates the parallel framework of FedRW, where each small square represents a single 2PC protocol interaction. The middle section depicts our reweighting scheme, which preserves the integrity of the dataset. The colors reflect the different weights assigned to samples. The right panel provides a comparison between FedRW and the baseline approach.\n\nAs you pointed out, **Section 3** and **Appendix A.3** illustrate the security and threat model. Due to the space constraints, we were unable to illustrate adaptation to FL in detail. We will include a brief description in the revision as follows:\n\n+ FedRW operates under the **semi-honest adversary model** in federated learning (FL), meaning all participants will follow the protocol honestly but may attempt to infer additional information from observation. For the scope of this work, we assume that the parties do not actively collude with one another. While we acknowledge that other forms of more active or malicious attacks exist in FL – such as inference attacks, backdoor attacks, or reconstruction attacks – these threats are considered orthogonal to the primary objective of this study, which focuses on designing the first privacy-preserving soft deduplication via sample reweighting. These additional and vital security considerations will be explored in our future research to enhance FedRW's robustness.\n\nFollowing your valuable suggestion, we will revise the manuscript with the above information.\n\n\n\n**3. Evaluation in Real-World Scenarios.**\n\n**We appreciate your valuable suggestion, and have conducted experiments on real-world datasets with repeated patterns on two mainstream models, Qwen3-0.6B and Llama-3.2-1B-Instruct.** \n+ _databricks-dolly-15k_ [1] is an open-source dataset of instruction-following records generated by thousands of Databricks employees.\n+ _Twitter Sentiment Analysis_ [2] is an open-source dataset of tweets labeled with sentiment categories, which inherently contains repeated patterns due to the nature of social media.\n\n- Results (Perplexity)\n  \n    - Qwen3-0.6B\n  \n        | Method         | databricks-dolly-15k | Twitter Sentiment Analysis |\n        |----------------|-------|-------------|\n        | Baseline       |   5.11    |     3.30        |\n        | FedRW (Ours)   |   **4.65**    |     **2.98**        |\n    \n    - Llama-3.2-1B-Instruct\n  \n        | Method         | databricks-dolly-15k | Twitter Sentiment Analysis |\n        |----------------|-------|-------------|\n        | Baseline       |   4.79    |      3.74       |\n        | FedRW (Ours)   |   **4.51**    |      **2.49**       |\n\nAs presented in the table, FedRW consistently outperforms the baseline under the same experimental settings, showing the effectiveness and generalization ability of our method.\n\n**References**\n\n[1] databricks, \"databricks-dolly-15k.\" *Hugging Face*.\n\n[2] passionate-nlp, \"Twitter Sentiment Analysis Dataset.\" _Kaggle_.\n\n**4. Extension to Malicious Model.**\n\nThank you for raising the thoughtful concerns about malicious clients. Extending FedRW to resist malicious adversaries would be an interesting and important research direction.\n\nAs mentioned in **2**, we acknowledge that the FedRW primarily operates under a semi-honest threat model, which is **sufficient for the vast majority of real-world scenarios**, given its high efficiency and practicality.\n\nRegarding the **malicious model**, it is feasible to design specific verification based on FedRW’s existing architecture, such as **Zero-Knowledge Proofs** (ZKPs), to verify the consistency of parties’ behaviors during preprocessing and training. However, such approaches would come at a significant computational and communication cost.\n\nThese potential schemes trade off between **model accuracy**, **data privacy**, and **computational overhead**. Given the scope and depth of these problems, we have deferred this research to our future follow-up work. \n\n**We are deeply grateful for the thought-provoking discussions and valuable insights that have enriched our understanding and improved the quality of this work.**"}, {"review_id": "xrvLdR5MeA", "reviewer": "Reviewer_d6ex", "summary": "In FL settings, there is a fundamental dilemma for deduplication: local fails to detect inter-client duplication. Existing method may remove some domain-specific information and introduce additional overheads. This paper propose a reweighting method to softly deduplicate samples without a trusted third party. This framework reweight based on the frequency for a more nuanced control based on the multi-party computation.", "questions": "1. Due to the different data collection fields or geographical locations of different clients, cooperation is meaningful. In this case, it is difficult for two clients to have exactly the same samples. Therefore, when the two repeated samples are not exactly the same, is this method still effective?\n2. I also have doubts about the significance of data deduplication in real FL scenarios. Is there such a problem in real datasets? Have different clients collected a large number of similar datasets?\n3. The experimental settings mention hyperparameters (e.g., learning rates, epochs), but the paper does not discuss sensitivity analyses or how hyperparameter choices impact performance, which could affect generalizability.\n4. This paper also did not conduct experiments with the current mainstream models. Therefore, whether partial repetition will still be harmful when using mainstream models is also an unknown question", "limitations": "See above", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strength:\n\n1. FedRW introduces a pioneering soft deduplication method for FL, avoiding data deletion and preserving dataset integrity, which is a significant advancement over hard deduplication techniques.\n2. By eliminating the need for a trusted third party and using secure 2PC with PSI, FedRW ensures robust privacy guarantees, addressing critical vulnerabilities in decentralized settings.\n\nWeakness:\nSee questions", "was_revised": true, "final_justification": "I am still concerned about the importance of this issue if the model size gets larger, so I will keep my score.", "rebuttal": "We appreciate the reviewer for the valuable and constructive comments. We are encouraged by the positive feedback on **novelty**, **privacy and security guarantees**, **significance of topic**, and the overall clarity of the paper. We hope the following discussions can address the reviewer's remaining questions.\n\n**1. Non-Exact Duplicates**\n\nWe greatly appreciate this insightful question. Indeed, there are scenarios where non-exact duplicates are more commonly encountered. We have conducted perplexity evaluations on the dataset, *Twitter Sentiment Analysis* [1], which contains non-exact repeated patterns due to the nature of social media.\n- Result\n\n    | Method         | Qwen3-0.6B | Llama-3.2-1B-Instruct |\n    |----------------|-------|-------|\n    | Baseline       |   3.30    |       3.74      |\n    | FedRW (Ours)   |   **2.98**    |    **2.49**         |\n\nThese results demonstrate the effectiveness of our method in addressing non-exact repetitive patterns.\n\n\n**2. Significance of Deduplication in Real FL Scenarios**\n\nWe truly appreciate this critical concern. Exact and near-exact duplicates are indeed a common issue in large-scale real-world datasets for LLMs. The presence of duplicates among private client data stems from several prevalent mechanisms:\n\n- **Influence of Shared Public Information**: Private data often contains content derived from or heavily influenced by widely shared public information. Similar to open-source datasets like _Twitter Sentiment Analysis_, which inherently exhibits extensive repetition due to the nature of social media, private data collected from users or organizations also tends to contain duplicated or near-duplicate patterns. Specific common patterns may appear across the private data of different clients. \n\n- **Inherent Data Collection**: Beyond external influences, duplicates can occur within sensitive personal information. For instance, GPS locations in personal travel information can exhibit duplicates due to recurring travel patterns or shared high-traffic areas [2], which can be collected to analyze user behavior patterns.\n\n- **Data Augmentation**: The data augmentation strategies [3] used in private datasets, including those based on generative models or synthetic data, may also produce a large number of similar or identical samples with duplicated patterns.\n \nThese mechanisms indicate that data duplication is a pervasive challenge in real-world, distributed private datasets in FL, necessitating solutions like FedRW.\n\n**References**\n\n[1] passionate-nlp, \"Twitter Sentiment Analysis Dataset.\" _Kaggle_.\n\n[2] Wu, Zhaomin, et al. \"Federated transformer: Multi-party vertical federated learning on practical fuzzily linked data.\" *Advances in Neural Information Processing Systems 37* (2024): 45791-45818.\n\n[3] Hou, Charlie, et al. \"Pre-text: Training language models on private federated data in the age of llms.\" *arXiv preprint arXiv:2406.02958* (2024).\n\n**3.  Evaluations on Mainstream Models**\n\n**We appreciate your suggestion, and have conducted supplementary perplexity evaluations on Qwen3-0.6B, Qwen2.5-0.5B-Instruct, and Llama-3.2-1B-Instruct under 30% duplication.**\n\n+ Qwen3-0.6B \n\n    | Method         | Haiku  | Short Jokes | Rotten Tomatoes | Poetry | Sonnets | Plays |\n    |----------------|--------|-------------|-----------------|--------|---------|-------|\n    | Baseline       |  2.47  |     2.61    |      1.71       |  2.54  |   4.07  |  8.21 |\n    | FedRW (Ours)   |  **2.36**  |     **2.44**    |      **1.59**       |  **2.21**  |   **3.62**  |  **7.23** |\n\n+ Qwen2.5-0.5B-Instruct\n    \n    | Method         | Haiku | Short Jokes | Rotten Tomatoes | Poetry | Sonnets | Plays |\n    |----------------|-------|-------------|-----------------|--------|---------|-------|\n    | Baseline       |   2.21    |     2.48        |       1.58          |    2.28    |     4.11    |   11.77   |\n    | FedRW (Ours)   |    **2.12**   |      **2.36**       |         **1.55**        |    **2.03**   |   **3.84**    |   **9.92**   |\n\n\n+ Llama-3.2-1B-Instruct \n\n    | Method         | Haiku | Short Jokes | Rotten Tomatoes |  Poetry | Sonnets | Plays |\n    |----------------|-------|-------------|-----------------|--------|---------|-------|\n    | Baseline       |   2.14    |     2.34        |     1.65            |         2.39     |    4.11     |  18.35     |\n    | FedRW (Ours)   |    **2.09**   |      **2.21**       |        **1.54**         |          **1.99**    |     **4.00**    |   **16.03**    |\n\n\nAs shown in the tables, FedRW consistently maintains its advantage in mitigating the impact of redundancy on model performance, particularly under challenging conditions such as data sparsity or complexity. \n\n**We agree that the impact of partial repetition may vary with model advancements, while data quality remains a fundamental factor in model performance. Deduplication is still an essential preprocessing step to ensure high-quality data.**\n\n**4. Hyperparameter**\n\n**We appreciate this valuable feedback. Regarding epochs, we typically utilize a small number (Appendix C, Line 516) as standard practice to prevent overfitting. Hence, our primary sensitivity analysis focused on the learning rate to assess FedRW's robustness.**\n\n+ Setup: We evaluated **perplexity** on DistilGPT2 and Qwen2.5-0.5B-Instruct under **learning rates of 1e-3, 5e-4, 3e-4, 1e-4, 5e-5, and 3e-5**. We selected the **_Plays_** dataset, which exhibited a significant performance gap, to investigate FedRW's generalizability. \n\n+ DistilGPT2\n    \n    | Method         | 1e-3 | 5e-4 | 3e-4 | 1e-4| 5e-5 | 3e-5 |\n    |----------------|-------|-------|-------|-------|-------|-------|\n    | Baseline       | 16.12 | 16.38 | 16.13 | 14.21 | 15.07 | 14.18 | \n    | FedRW (Ours)   | **14.42** | **14.79** | **14.74** |**13.17** | **14.50** | **12.76** | \n\n+ Qwen2.5-0.5B-Instruct\n\n    | Method         | 1e-3 | 5e-4 | 3e-4 | 1e-4| 5e-5 | 3e-5 |\n    |----------------|-------|-------|-------|-------|-------|-------|\n    | Baseline       | 12.86 | 11.07 | 10.63 | 11.50 | 11.77 | 10.67| \n    | FedRW (Ours)   | **11.48** | **9.35** | **8.15** |**8.14** | **9.92**| **9.81** | \n\n\nThe experimental results indicate that FedRW consistently maintains its performance advantages and exhibits stable training behavior across this range of learning rates.\n\n\n**We are deeply grateful for your thoughtful consideration, which has enriched our understanding and improved the quality of this work.**"}, {"review_id": "5kcNzUcdDg", "reviewer": "Reviewer_xmuh", "summary": "FedRW has proposed an innovative privacy preserving federated learning framework that replaces trusted third parties with a frequency aware sample reweighting mechanism, securely calculates global sample frequencies using the PPMPR protocol, and combines a layered parallel strategy to reduce preprocessing complexity, achieving a preprocessing acceleration of up to 28.78 times while improving model generalization ability.", "questions": "See Weakness.", "limitations": "Yes, the proposed method, while promising, reveals certain limitations in the Experiments section.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: \n1. Balancing privacy and efficiency\n2. By using logarithmic functions to dynamically adjust sample weights and preserve low-frequency sample information, an average confusion reduction of 11.42% was achieved on datasets such as Haiku and Sonnets, which is particularly better than the performance of hard deletion methods in data scarcity scenarios.\n3. MPC protocol based on semi honest model provides provable security\nWeaknesses:\n1.The innovation of FedRW is mainly reflected in the first soft deduplication implementation and efficiency optimization in federated scenarios (such as pre-processing acceleration), but its protocol, weight mechanism, and parallel strategy heavily rely on existing technologies (PSI, centralized soft deduplication).\n2.This framework provides an efficient and scalable solution for privacy sensitive federated training, but further optimization is needed for semantic redundancy processing and dynamic client adaptation capabilities.", "was_revised": true, "final_justification": "After the rebuttal and discussion process, I am happy to change my decision to accept this paper.", "rebuttal": "We appreciate the reviewer for the valuable and constructive comments. We are encouraged by the positive feedback on **novelty**, **efficiency and effectiveness**, **privacy and security guarantees**, **significance of topic**, and the overall clarity of the paper. We hope the following discussions can address the reviewer's remaining questions.\n\n**1. Rely on Techniques**\n\nWe sincerely appreciate your recognition of the innovation and originality of FedRW. We agree that FedRW draws inspiration from the centralized soft deduplication. **However, the same problem remains unresolved in the FL setting, where simply applying the existing techniques introduces new privacy challenges.** Compared with multiple rounds of key agreement and encryption in SOTA work, we leverage the PSI primitive as a foundational component to implement **a lightweight variant, tailored for the defined 2PC functionality**, while achieving **the stronger, third-party-free privacy assumption**. We also present **an innovative parallel orchestration strategy** to overcome the inherent computational overhead of MPC protocols in FL. The proposed innovations achieve up to 28.78× speedup in preprocessing.\n\nThus, we respectfully argue that the core novelty of our work lies in the innovative integration, adaptation, and optimization of these techniques to address a previously unexplored and challenging problem: enabling efficient and privacy-preserving soft deduplication in FL without relying on a trusted third party. \n  \n\n**2. Future Optimization**\n\nWe appreciate the reviewer's insightful comments for optimization, and we fully agree that these are vital directions for future research. \n\nRegarding the **semantic matching**, this is indeed a significant issue in large-scale real-world datasets for LLMs. We may leverage the representation learning capability of transformer-based architectures to extract semantic duplication.\n\nRegarding the **dynamic client adaptability**, we acknowledge the importance of dynamic scenarios in which clients may join, leave, or exhibit varying computational capabilities. A preliminary idea is to employ Updatable PSI [1] to address the changes in sample weights caused by client arrivals and departures, and this may trade off between functionality and computational overhead. We plan to explore these valuable aspects in our future research.\n\n**References**\n\n[1] Agarwal, Archita, et al. \"Updatable private set intersection from structured encryption.\" *Cryptology ePrint Archive* (2024).\n\n**We sincerely appreciate your thoughtful and well-justified suggestion, as well as the valuable inspiration you have provided regarding future optimization of our work.**"}, {"review_id": "3pCe88zjzR", "reviewer": "Reviewer_y8JQ", "summary": "This paper introduces Federated ReWeighting (FedRW), a privacy-preserving framework designed to address the challenges of data duplication in the federated training of large language models (LLMs). Traditional deduplication methods in federated learning often rely on trusted third parties and involve deleting redundant data, which can lead to the loss of valuable information and introduce privacy risks. FedRW proposes a \"soft deduplication\" approach that reweights data samples based on their global frequency, rather than deleting them, and does so without requiring a trusted third party.", "questions": "See above.", "limitations": "There is no discussion on limitations in the paper.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n- The paper focuses on the data duplication issue in federated learning, an important topic in the field.\n- The paper clearly illustrates the framework in the method part.\n\nWeaknesses\n- The paper does not clearly show the difference between the proposed method and the baseline, EP-MPD.\n- The paper does not explain the FL experiment settings. What FL settings are used currently? Is the proposed method still effective under different FL settings, such as IID, non-IID, or personalized FL?\n- In the experiments, only GPT-2 Large and DistilGPT-2 are used for the method. It would be better to include some more recent LLMs in the experiments.", "was_revised": true, "final_justification": "The rebuttal has addressed my concerns. I updated the score accordingly.", "rebuttal": "We appreciate the reviewer for the valuable and constructive comments. We are encouraged by the positive feedback on the **significance of topic** and the clarity of the paper. We hope the following discussions can address the reviewer's remaining questions.\n\n**1. Clarity of Difference From Baseline and FL Experimental Settings**\n\nWe appreciate this feedback on the clarity of the difference between FedRW and the baseline (EP-MPD), and the FL experimental settings. We want to clarify that we have detailed the key information in our manuscript:\n\n+ **Section 1 (Introduction)**: We introduced EP-MPD, the SOTA work for federated hard deduplication, and illustrated three key challenges unresolved, followed by the introduction of our proposed method, FedRW. We also presented an overview of EP-MPD and its potential limitations in Figure 1.\n+ **Section 2 (Related Work)**: In the subsection on _**Distributed Deduplication**_ in existing work, we discussed the mechanism and limitations of EP-MPD.\n+ **Section 4 (Framework)**: This section offered a detailed explanation of FedRW. In Figure 2, we presented the architecture framework of FedRW and provided an intuitive comparison on protocol runtime, perplexity, and third-party reliance. We consider these presentations to be sufficient in illustrating the essential differences between the two methods. \n+ **Section 5 (Experiments)**: The experimental results on preprocessing and model performance clearly showed the difference between FedRW and EP-MPD.\n+ **Section 5.1 (Experimental Settings)**: This section declared the datasets, client number, data preprocessing and distribution strategies, evaluation metrics, and overall experimental setup. For completeness, Appendix C includes a detailed description of the datasets, the partitioning standard, the experimental environment, and the hyperparameter configurations.\n\nWe sincerely hope that the above clarifications effectively resolve your concerns. \n\n\n**2. Effectiveness Across FL Settings**\n\nWe appreciate your concerns for more detailed effectiveness under various FL settings. Regarding the three specific FL settings:\n\n+ Current FL settings: In this study, we followed the baseline setup for **unsupervised fine-tuning**. As described in Lines 203-204, \"To simulate redundancy, duplicates are synthetically added into the training set at different rates and distributed uniformly across 10 clients.\" The data follows an **IID** distribution.\n\n+ Non-IID: **We appreciate the suggestion to explore the Non-IID setting, and we have conducted experiments on Qwen3-0.6B to evaluate FedRW's effectiveness.**\n  1. Setup 1: We categorized the *Rotten Tomatoes* dataset by *label* and set up 5 clients. To simulate **Quantity Skew**, we distribute the data according to the proportion of the total train set as **[40%, 20%, 20%, 10%, 10%]**. For **Label Skew**, we set the distributions for the binary (0/1) labels as: **[(0.5, 0.5), (0.6, 0.4), (0.4, 0.6), (0.9, 0.1), (0.1, 0.9)]**. \n  \n  2. Result 1 (under 25% duplication)\n   \n        | Method         | IID | Quantity Skew | Label Skew |\n        |----------------|-------|-------|-------|\n        | Baseline       |1.71| 2.02 | 2.44 | 1.73 |\n        | FedRW (Ours)   |**1.59**| **1.96** | **1.66** |\n  \n  3. Setup 2: To simulate **Feature Skew**, we allocated datasets *Poetry*, *Sonnets*, and *Plays* to separate clients. **These datasets differ distinctly in terms of text structure, sentence length, and lexical and syntactic complexity.**\n   \n  4. Result 2\n   \n        | Method         | Feature Skew |\n        |----------------|-------|\n        | Baseline       | 3.43 |\n        | FedRW (Ours)   | **2.70** |\n\n    The experimental results indicate that FedRW consistently maintains its performance advantages in the **Non-IID** settings.\n\n\n+ Personalized FL:\nWe appreciate the suggestion to explore personalized FL. While the current FedRW framework primarily focuses on enhancing a global LLM, we acknowledge that language models benefit significantly from personalization to adapt to individual user preferences. A deeper exploration may involve diverse model architectures or personalization strategies, beyond the scope of this foundational privacy-preserving soft deduplication framework. We consider adapting FedRW to integrate with this pattern a valuable and exciting avenue for future research. \n\n\n**3. Evaluations on Mainstream Models**\n\n**We appreciate your valuable suggestion, and have conducted additional perplexity evaluations on Qwen3-0.6B, Qwen2.5-0.5B-Instruct, and Llama-3.2-1B-Instruct under 30% duplication.**\n\n+ Qwen3-0.6B \n\n    | Method         | Haiku  | Short Jokes | Rotten Tomatoes | Poetry | Sonnets | Plays |\n    |----------------|--------|-------------|-----------------|--------|---------|-------|\n    | Baseline       |  2.47  |     2.61    |      1.71       |  2.54  |   4.07  |  8.21 |\n    | FedRW (Ours)   |  **2.36**  |     **2.44**    |      **1.59**       |  **2.21**  |   **3.62**  |  **7.23** |\n\n+ Qwen2.5-0.5B-Instruct\n    \n    | Method         | Haiku | Short Jokes | Rotten Tomatoes | Poetry | Sonnets | Plays |\n    |----------------|-------|-------------|-----------------|--------|---------|-------|\n    | Baseline       |   2.21    |     2.48        |       1.58          |    2.28    |     4.11    |   11.77   |\n    | FedRW (Ours)   |    **2.12**   |      **2.36**       |         **1.55**        |    **2.03**   |   **3.84**    |   **9.92**   |\n\n\n+ Llama-3.2-1B-Instruct \n\n    | Method         | Haiku | Short Jokes | Rotten Tomatoes |  Poetry | Sonnets | Plays |\n    |----------------|-------|-------------|-----------------|--------|---------|-------|\n    | Baseline       |   2.14    |     2.34        |     1.65            |         2.39     |    4.11     |  18.35     |\n    | FedRW (Ours)   |    **2.09**   |      **2.21**       |        **1.54**         |          **1.99**    |     **4.00**    |   **16.03**    |\n\nAs shown in the tables, FedRW continues to show its advantage in addressing the impact of duplication on model performance, particularly under challenging conditions such as data sparsity or complexity. \n\n**4. Discussion on Limitations**\n\nWe appreciate your concern regarding the lack of a detailed discussion on limitations. While we have discussed FedRW’s boundaries and potential extensions in Section 6 (Lines 284–286), as well as certain limitations within the Experiments section, we commit to providing a clearer and more comprehensive discussion of these aspects in the revised manuscript.\n\n**We sincerely hope our replies have clearly addressed your concerns. We are deeply grateful for the valuable feedback and the broader perspectives you offered, which have helped us better position the significance of this work within the broader research landscape.**"}]

Author rebuttal:

Meta-review: There is a fundamental dilemma for deduplication in FL: local devices fail to detect inter-client duplication. Existing method may remove some domain-specific information and introduce additional overheads. This paper propose a reweighting method to softly deduplicate samples without a trusted third party.

Strenghts: FedRW introduces a new soft deduplication method for FL. By eliminating the need for a trusted third party and using secure 2PC with PSI, FedRW ensures robust privacy guarantees. 

Weaknesses: Whether deduplication is a real problem in FL is not fully addressed. 

The main reason for my decision is that the reviewers seem to agree on the merits of this paper: proposing a novel solution to deduplication in FL that does not require a trusted 3rd party.
Decision: Accept (poster)