Federated Transformer: Multi-Party Vertical Federated Learning on Practical Fuzzily Linked Data

Authors: zhaomin  wu
junyi  hou
yiqun  diao
bingsheng  he
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 16:30:27

Abstract: Federated Learning (FL) is an evolving paradigm that enables multiple parties to collaboratively train models without sharing raw data. Among its variants, Vertical Federated Learning (VFL) is particularly relevant in real-world, cross-organizational collaborations, where distinct features of a shared instance group are contributed by different parties. In these scenarios, parties are often linked using fuzzy identifiers, leading to a common practice termed as _multi-party fuzzy VFL_. Existing models generally address either multi-party VFL or fuzzy VFL between two parties. Extending these models to practical multi-party fuzzy VFL typically results in significant performance degradation and increased costs for maintaining privacy. To overcome these limitations, we introduce the _Federated Transformer (FeT)_, a novel framework that supports multi-party VFL with fuzzy identifiers. FeT innovatively encodes these identifiers into data representations and employs a transformer architecture distributed across different parties, incorporating three new techniques to enhance performance. Furthermore, we have developed a multi-party privacy framework for VFL that integrates differential privacy with secure multi-party computation, effectively protecting local representations while minimizing associated utility costs. Our experiments demonstrate that the FeT surpasses the baseline models by up to 46\% in terms of accuracy when scaled to 50 parties. Additionally, in two-party fuzzy VFL settings, FeT also shows improved performance and privacy over cutting-edge VFL models.

Main paper: 
 introduction: 
 Federated Learning (FL) is a learning paradigm that enables multiple parties to collaboratively train a model while preserving the privacy of their local data [27] . Among its various forms, Vertical Federated Learning (VFL) [53] is particularly prevalent form in real-world applications as highlighted in a recent technical report [48] . In VFL, participants possess different features of the same set of instances, where common features, such as names or addresses, serve as identifiers (a.k.a. keys) to link datasets across these parties.Real-world applications often necessitate multi-party fuzzy VFL, characterized by two key attributes. First, it supports collaboration among multiple parties, commonly observed in collaborations across hospitals [33] , sensors [52] , and financial institutions [38] . Second, it accommodates scenarios where these parties are linked using fuzzy identifiers, such as addresses. Such scenarios are prevalent in applications, as illustrated in an analysis [50] of the German Record Linkage Center [11] . For instance, multiple vehicle rental companies that are fuzzily linked by source and destination addresses in the same city can collaborate to predict travel times.To illustrate the significance of multi-party fuzzy VFL, consider the application of travel cost prediction in a city through collaboration among taxi, car, bike, and bus companies, as shown in Figure 1 . Since personal travel information is private and cannot be shared, VFL is essential. Additionally, route identifiers -starting and ending GPS locations -can only be linked using fuzzy methods. However, linking closely related source and destination points with multi-party fuzzy VFL can significantly enhance prediction accuracy.Existing VFL approaches generally address either the multi-party aspect or the fuzzy identifier issue. Several methods [33, 52, 38, 21, 26] facilitate multi-party VFL using Private Set Intersection (PSI) [10] to link datasets. These methods often presume the existence of precise, universal keys, which are not feasible in common VFL scenarios involving fuzzy identifiers. Conversely, other studies [17, 34, 50] propose two-party fuzzy VFL models that utilize cross-party key similarities for training. However, when extended to multi-party fuzzy VFL, these similarity-based approaches encounter significant challenges in performance and privacy. While some methods achieve reasonable performance, they often compromise privacy or incur prohibitive costs.Despite the potential of multi-party fuzzy VFL, several significant challenges must be addressed for effective implementation. First, as the number of parties with fuzzy identifiers increases, maintaining performance becomes increasingly challenging. The addition of parties leads to a quadratic growth in the number of key pairs, an increase in incorrect linkages between fuzzy identifiers, and larger model sizes. These factors collectively heighten the risk of overfitting and adversely impact model performance. Second, the rising costs of preserving privacy intensify as more parties with correlated data participate, leading to either significant computational costs [33, 52, 38] or accuracy loss [47] . Third, in a collaboration of multiple parties, a communication bottleneck arises for the party with labels, termed the primary party. This party must communicate with all other parties without labels, termed secondary parties, in each training round, placing substantial demands on the primary party's bandwidth. These challenges significantly hinder the practical deployment of VFL.To address these issues, we introduce the Federated Transformer (FeT) to enhance performance and reduce privacy costs in multi-party fuzzy VFL. First, to tackle performance issues, we encode key similarities into data representations aligned by positional encoding averaging, which eliminates the need for quadratic calculations of key pairs. Additionally, we have designed a trainable dynamic masking module that automatically filters out incorrectly linked pairs, enhancing model accuracy by up to 13% in 50-party fuzzy VFL on the MNIST dataset. Second, to mitigate the escalating costs of privacy protection, we introduce SplitAvg, a hybrid approach that merges encryption-based and noise-based methods, maintaining a consistent noise level regardless of the number of participating parties. Third, to alleviate communication overhead on the primary party, we implement a party dropout strategy, which randomly excludes certain secondary parties during each training round. This effectively reduces communication costs by approximately 80% and improves model generalization. Our codes are available on GitHub 1 . In summary, our contributions are as follows:• We design Federated Transformer (FeT), a novel model achieving promising performance under multi-party fuzzy VFL. • We introduce SplitAvg to enhance the privacy of FeT by protecting local representations in multi-party fuzzy VFL, with a theoretical proof of its differential privacy. • Experimental results demonstrate that FeT significantly outperforms baseline models by up to 46% in terms of accuracy in 50-parties VFL. Moreover, while providing enhanced privacy, FeT consistently surpasses state-of-the-art models even in traditional two-party fuzzy VFL scenarios. 
preliminaries: 
 In this section, we provide the foundational concepts necessary for understanding our approach to differential privacy. Differential Privacy (DP) offers a rigorous mathematical framework for preserving individual privacy. It quantifies privacy in terms of the probability of producing the same output from two similar datasets that differ by exactly one record. Definition 1. Consider a randomized function M : R d → O and two neighboring databases D 0 , D 1 ∼ R d that differ by exactly one record. For every possible output set O ⊆ O, M satisfies (ε, δ)-differential privacy ifPr[M(D 0 ) ∈ O] ≤ e ε Pr[M(D 1 ) ∈ O] + δ,where ε ≥ 0 and δ ≥ 0.A single query that adheres to differential privacy is termed a mechanism. For example, the Gaussian mechanism [4] is commonly used to achieve DP by adding Gaussian noise to the output of the function.Theorem 1 (Gaussian Mechanism [4] ). For a function f : X → R d characterized by a global L 2 sensitivity ∆ 2 , which signifies that the maximum difference in the L 2 -norm of the outputs of f on any two neighboring databases is ∆ 2 , and for any ε ≥ 0 and δ ∈ [0, 1], the Analytic Gaussian Mechanism is defined as M(x) = f (x) + Z, where Z ∼ N (0, σ 2 I). This mechanism satisfies (ε, δ)-differential privacy if Φ ∆2 2σ -εσ ∆2 -e ε Φ -∆2 2σ -εσ ∆2 ≤ δ, where Φ(t) = 1 √ 2π t -∞ e -x 2 /2 dx is the cumulative distribution function (CDF) of the standard univariate Gaussian distribution.When multiple queries are made on the same database, independent Gaussian noise is added to each query to maintain differential privacy. The privacy loss of the composition of Gaussian mechanisms is formulated in Theorem 2. Theorem 2 (Moments Accountant [1] ). There exist constants c 1 and c 2 so that given the sampling probability q = L N and the number of steps T , for any ε < c 1 q 2 T , DPSGD [1] is (ε, δ)-differentially private for any δ > 0 if we choose σ > c 2 q √ T log(1/δ) ε .Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: See Section 8.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.• The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
related work: 
 Performance. Traditional VFL approaches [29, 7] are typically limited to two-party scenarios. In contrast, existing multi-party VFL methods [12, 51, 33, 52, 38, 21, 26] often rely on the assumption of precise identifiers that ensure perfect alignment across all parties. These methods generally employ the SplitNN framework [45] , where each party maintains a portion of the model, and the models are collaboratively trained on well-aligned data samples through the transfer of representations and gradients, commonly known as split learning. However, the requirement for perfect data alignment is impractical in many real-world scenarios [50, 3] , where identifiers are often imprecise.To address this limitation, semi-supervised VFL [22, 30, 19, 20] has emerged, attempting to improve model performance by leveraging unlinked records through semi-supervised or self-supervised learning. However, these methods still assume that datasets from each party can be precisely linked using exact identifiers, a premise that is often untenable in real-world settings [50, 3] . Given that the quality of linkage significantly impacts VFL accuracy [34] , exploring effective linkage methods remains a pivotal issue in VFL.On the other hand, FedSim [50] , based on real linkage projects at the German Record Linkage Center (GRLC) [3] , acknowledges that the keys of each party are usually not precisely linkable and that records may have one-to-many relationships, leading to fuzzy linkage scenarios, as seen with keys like GPS addresses. FedSim enhances training performance by performing soft linkage and conducting training based on transmitted key similarities. Nonetheless, it faces limitations in scalability beyond two parties and introduces new privacy concerns by directly transferring similarities.In summary, while existing studies face significant performance challenges when handling fuzzy keys in multi-party settings, our proposed FeT demonstrates a scalable design that addresses these challenges and shows promising performance improvements in both multi-party fuzzy VFL and two-party settings compared to FedSim.Privacy. The privacy concerns associated with VFL are multifaceted. First, the primary party may infer data representations from secondary parties [31] . Second, the secondary party may derive gradients from the primary party [41, 54] . Third, external attackers could conduct membership inference attacks [39] on the deployed model [51] . This paper primarily addresses the second concern: safeguarding representations, while acknowledging other concerns as open challenges.To address the privacy of representations in VFL, various methods have been proposed, falling into two primary categories: encryption-based methods and noise-based methods. Encryption-based methods [26, 14, 33, 52, 38, 21, 36] utilize computationally intensive cryptographic techniques to encrypt intermediate results. However, these methods often incur significant communication overhead when scaled to multiple parties. Conversely, noise-based methods [47, 46] protect data by perturbing [47] or manipulating [46] local representations. These methods typically do not provide theoretical privacy guarantees or require substantial amounts of noise when scaling to multiple parties in VFL, which can degrade performance. Unlike existing studies that rely solely on either approach, this paper explores a combined strategy incorporating both encryption-based and noise-based methods, ensuring the model scales effectively to multiple parties without the need for excessive noise.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes] Justification: See Section 6 and Appendix A.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
problem statement: 
 In this section, we formalize the concept of multi-party fuzzy VFL. We consider a supervised learning task where one party holding labels, termed the primary party P , collaborates with k parties that do not hold labels, referred to as the secondary parties. The primary party P possesses n data records denoted as x P := {x i } n i=1 along with corresponding labels y := {y i } n i=1 . Each secondary party S k maintains its own dataset x S k . All parties share common features, referred to as identifiers, expressed asx i = [k i , d i ],where [•] signifies concatenation. These identifiers k i may exhibit inaccuracies and fuzziness, despite residing within the same range.min θ 1 n n i=1 L(f (θ; x P i , x S1 , . . . , x S k ); y i ) + Ω(θ)In this formulation, L(•) denotes the loss function, θ represents the model parameters, and Ω(θ) refers to the regularization term. The symbol n indicates the number of samples in the primary party P .Threat Model. This study focuses on defending feature reconstruction attacks [25, 31] , which target local representations shared with the primary party. FeT operates under the assumption that all participating parties are honest-but-curious, meaning they adhere to the protocol but may attempt to infer additional information about other parties. Furthermore, we assume that the parties do not collude with one another. While other forms of attacks, such as label inference attacks [13] and backdoor attacks aimed at compromising labels and gradients, exist, they are considered orthogonal to the objectives of this study. These additional threats will be explored in our future research.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: See Section 7.1 and Appendix B. Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. Justification: We follow the code of ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
approach: 
 In this section, we address the performance and communication challenges inherent in multi-party fuzzy VFL. To tackle these issues, we introduce a transformer-based architecture named the Federated Transformer (FeT). This model encodes keys into data representations, thereby reducing reliance on key similarities. To accurately exclude incorrectly linked data records, we propose a trainable dynamic masking module that generates masks based on keys. Furthermore, to combat overfitting caused by the large model and to alleviate communication bottlenecks faced by the primary party, we introduce a party dropout strategy that randomly invalidates some parties during training. Additionally, we identify a positional encoding misalignment issue across parties in the FeT and propose positional encoding averaging to ensure consistent alignment, thereby enhancing model performance.The architecture of the FeT is illustrated in Figure 2 . In FeT, each secondary party has an encoder, while the primary party has both an encoder and a subsequent decoder. The internal structure of both the encoder and decoder closely adheres to the conventional transformer model [43] . We utilize multi-dimensional positional encoding [28] to integrate key information into feature vectors. Outputs from the encoders at the secondary parties are aggregated and then fed into the decoder. Details regarding the privacy mechanisms employed during this aggregation phase are discussed in Section 6, while the details of the training process are elaborated in Section 5.2. We then elaborate on three techniques designed to improve performance and reduce communication costs.Dynamic Masking. The size of the neighborhood varies significantly depending on the party and key values. Consequently, including a large number of neighbors K for all parties can hinder the model's ability to extract meaningful information and result in overfitting. To address this, we introduce a dynamic "key padding mask" in the transformer, generated from the identifier values using a trainable MLP. This approach allows the model to effectively disregard distant data records, thereby eliminating the influence of irrelevant data when K is large. This concept resembles the weight gate in FedSim, but it diverges by using identifiers as inputs instead of similarities, enhancing privacy by preventing the transmission of similarity data across parties.The learned dynamic masking is visualized in Figure 3 . We derive two key insights from the visualization: (1) Dynamic masking effectively focuses on a localized area around the primary party's identifiers. Data records with distant identifiers on secondary parties (in cooler colors) receive small negative mask values, reducing their significance in the attention layers -without accessing the primary party's original identifiers. (2) The focus area varies in scale and direction across samples: for example, the left figure concentrates on a small bottom area, the middle figure on a small top area, and the right figure on a broad area in all directions.Party Dropout. Extending the Federated Transformer (FeT) to support multiple parties can be challenging for several reasons. First, the communication bandwidth required by the primary party becomes a significant bottleneck within the SplitAvg framework, increasing linearly with the number of parties. Second, the inclusion of many parties can result in an excessive number of parameters, which may lead to overfitting. To mitigate these issues, we introduce the concept of party dropout.Inspired by traditional dropout [40] , we randomly set a portion r d of the parties' representations to zero during training. This method serves as a form of regularization, thus helping to reduce overfitting, while also significantly cutting down on communication overhead. In our experiments, we demonstrate that increasing the party dropout rate to 0.8 leads to minimal accuracy loss and can even improve accuracy. Consequently, the communication overhead on the primary party can be reduced by up to 80%, enhancing scalability when dealing with large numbers of parties.Like traditional dropout, it is crucial to maintain consistent scaling of the representations during both training and testing phases. This consistency is naturally achieved within the SplitAvg framework.During the averaging process, if a ratio r d k of parties is set to zero, we adjust by dividing only by the number of non-zero parties, (1 -r d )k. This method ensures that the scale of the averaged representations remains consistent across training and testing, regardless of the value of r d .Positional Encoding Averaging. In positional encoding (PE), it is generally expected that the distances between encoded representations are positively correlated with the distances between identifiers. In FeT, each party employs its own encoder and PE layer, each tasked with encoding its local identifiers into representations. This configuration leads to significant PE misalignment issues, as illustrated in Figure 4 . Although the identifiers and their corresponding encoded representations maintain a positive correlation within the PE layer of each party, there is almost no correlation between the identifiers and encoded representations across different parties. This lack of correlation causes integration issues and affects accuracy. However, directly sharing a PE layer among all parties is not viable due to privacy concerns. To address this, we propose positional encoding averaging.Every T pe epoch, the positional encoding layers are averaged and broadcast to all parties under a secure multi-party computation (MPC) scheme, akin to FedAvg [23] in horizontal federated learning [27] . While the privacy of the transmitted model can be a concern, this issue is an orthogonal open problem in horizontal federated learning.The FeT training process begins with employing Privacy-Preserving Record Linkage (PPRL) [44] to evaluate identifier similarities between the primary party P and each secondary party. Secondary parties each contribute a random subset for linkage (line 5). For each P 's record, K nearest neighbors within these subsets from secondary parties are determined (line 6). The training leverages data embeddings of dimensions B × L × H, where B is batch size, L is the sequence length, and H is the hidden layer size, following the transformer architecture. In FeT's context, L = 1 for the primary and L = K for secondary parties, linking each primary record with K neighboring records from the secondaries. Identifiers are transformed into vectors using multi-dimensional positional encoding [28] and combined with data representations for processing via self-attention blocks (lines 7, 10). Secondary parties' representations are averaged under the MPC protocol. The primary party then employs attention blocks for forward propagation to compute the final prediction (line 13). Backpropagation sends gradient updates from the primary to secondary parties to refine their local models (lines 14-16 xh ← randomly choose q ratio from x h6 xh i ← link K records with x P i from xh 7 Rh i ← f (θ h l ; xh i ) Rh i ← Rh i / max 1, ∥k Rh i ∥2 C // Norm clipping end R P i ← f (θ P l ; x P i ) RP i ← R P i / max 1, k∥R P i ∥2 C // Norm clipping H i ← MPCAvg R S1 i , . . . , R S k i , N (0, C 2 σ 2 ) ŷi ← f (θ P a ; H i ) ∇ θ P a ← ∂ℓ(yi,ŷi) ∂θ P a , θ P a ← θ P a -η t ∇ θ P a for party h ∈ {P, S 1 , . . . , S k } do ∇ θ h l ← ∂ℓ(yi,ŷi) ∂θ h a , θ h l ← η t ∇ θ h l end end end 
privacy: 
 In this section, we address the challenges of privacy in multi-party fuzzy VFL. First, the risk of transferring raw similarities has been mitigated by the design of the FeT itself. Second, to address the increasing costs when more parties join, we introduce a multi-party privacy-preserving VFL framework, SplitAvg, which is compatible with FeT. The architecture of SplitAvg is illustrated in Figure 5 . SplitAvg integrates differential privacy (DP), secure multi-party computation (MPC) [6] , and norm clipping to enhance the privacy of representations. Additionally, to further improve the utility of FeT under DP, we employ privacy amplification techniques that reduce the noise scales by incorporating random sampling.This section outlines three techniques applied to the SplitAvg to improve privacy: representation norm clipping, privacy amplification, and MPC with distributed Gaussian noise. These strategies collectively protect the privacy of each secondary party's representations through differential privacy and ensure that privacy risks do not escalate with an increase in the number of parties due to MPC.Representation Norm Clipping. The magnitude of the ℓ 2 -norm is pivotal in determining the sensitivity of differential privacy. To limit the maximum change of the ℓ 2 -norm, norm clipping is essential. Specifically, for a representation R, we ensure that ∥R∥ 2 ≤ C, where C is a predefined positive real number. This is achieved by scaling R by a factor of C, formally, R = R/ max(1, ∥R∥ 2 /C). Through this process, any representation R with ∥R∥ 2 exceeding C is scaled to C, while values of ∥R∥ 2 below C remain unaffected.Secure Multi-Party Computation with Distributed Gaussian Noise. To address the challenges of applying differential privacy in multi-party VFL, we propose a method that integrates noise addition into the process of aggregating representations, facilitated through MPC. In this setup, each secondary party first independently conducts representation norm clipping to limit the scale of their data. Subsequently, these clipped representations, along with Gaussian noise N (0, σ/k 2 ) independently generated by each of the k secondary parties, are aggregated through averaging under MPC. For the primary party, this aggregation is equivalent to adding independent Gaussian noise N (0, σ 2 ) to the averaged result. The adoption of MPC in our framework ensures that the secondary parties do not need to individually add N (0, σ 2 ) noise to their representations. Instead, as the primary party only has access to the averaged result under MPC, each secondary party can add a smaller amount of noise. This method effectively improves utility with a small efficiency cost due to MPC. Privacy Amplification by Secondary Subsampling. This technique is specifically designed for FeT configurations. According to the principle of privacy amplification [5] , applying a function to a randomly sampled subset of data enhances privacy compared to applying the same function to the entire dataset. By initiating linkage from a randomly sampled subset rather than the full dataset, the privacy parameters effectively shift from (ε, δ) to (qε, qδ), where q < 1 represents the sampling rate. In FeT, the primary party typically selects subsets of candidate data records for training from each secondary party, targeting those with neighboring identifiers. By pre-sampling these subsets at a rate of q before conducting a k-nearest neighbors (kNN) search, we avoid processing the entire dataset, which in turn reduces the noise required to maintain the same privacy level.Our analysis of differential privacy focuses on a hypothetical global dataset linked using all secondary parties, denoted as x S1 , . . . , x S k . Since these datasets are correlated, removing one data record from this global dataset will result in changes to the representations in all secondary parties. Consequently, privacy loss accumulates across secondary parties without the use of MPC. However, with MPC, a single aggregated noise, formed by distributing smaller noise contributions among parties, can be added, effectively reducing the overall required noise. The privacy guarantee for these representations is formally articulated in Theorem 3, with the proof provided in Appendix A. Theorem 3. For certain constants c 1 and c 2 , given a sampling rate q, the total number of epochs T , and the number of batches B in each epoch, each representation R S k achieves (ε, δ)-differential privacy for all ε < c 1 q 2 T , with any δ > 0, by selecting the standard deviation σ of the Gaussian noise mechanism as follows:EQUATION 
experiments: 
 This section presents the experimental setup and results. We begin by outlining the experimental settings in Section 7.1, followed by an assessment of performance across varying numbers of parties and neighbors in Section 7.2. We then analyze the privacy of FeT in Section 7.3. Additionally, an ablation study is conducted in Appendix C to evaluate the contribution of each component to performance, including dynamic masking, party dropout, positional encoding, key fuzziness, and SplitAvg. The performance of FeT with exact key matching is assessed in Appendix D, while the computational and memory efficiency of MPC and training is evaluated in Appendix E. Privacy evaluation on two-party real datasets is included in Appendix F. Furthermore, FeT's performance under imbalanced feature splits across parties, based on VertiBench [49] , is presented in Appendix G.Datasets. Our experiments utilize five datasets, including three real-world datasets: house [35, 2] , bike [8, 42] , and hdb [18, 37] , along with two high-dimensional datasets: gisette [16] and MNIST [24] . Detailed descriptions of these datasets can be found in Appendix 7.1. To simulate multi-party fuzzy VFL, we partition the features equally and randomly among the parties. The primary party's feature dimensions are reduced to 4 using principal component analysis (PCA) to serve as universal keys. To create fuzzy linked scenarios, we add independent Gaussian noise with a scale of 0.05 to the keys of each party.Baselines. We include three baselines in our experiments: (1) Solo: training only on the primary party;(2) Top1Sim: linking each data record in the primary party only with its most similar neighbor in the secondary parties; (3) FedSim [50] : training on the top K neighboring data records. Effect of Number of Neighbors K. In this experiment, we assess the impact of the number of neighbors, K, on FeT's performance by varying K from 1 to 100. The results are displayed in Figure 6 . The FedSim baseline is trained using the optimal K value (i.e., 50 for hdb and 100 for house and bike). The figure reveals two key insights: First, FeT's performance improves as K increases, demonstrating its ability to filter useful information even as the number of unrelated data records grows. Second, FeT consistently outperforms all baselines at larger values of K, highlighting its superiority in fuzzy VFL scenarios. In this experiment, we assess FeT's performance in fuzzy VFL with various numbers of parties. Due to the absence of real multi-party VFL data, we employ synthetic data for our evaluations. We partition the features equally and randomly among the parties, reducing the primary party's feature dimensions to 4 using PCA as the universal keys. To simulate fuzzy linked scenarios, we add independent Gaussian noise with a scale of 0.05 to the keys of each party.Figure 7 shows that FeT generally outperforms the baselines, particularly with a larger number of parties. This advantage is attributed to Solo's lack of informative features and Top1Sim's noiseaffected linkage. FedSim performs poorly as the top-1 linked secondary parties are unaware of the primary parties' keys, leading to misalignment in subsequent soft linkage and training steps. On the gisette dataset with k = 10, FeT and other models slightly underperform compared to Solo, likely due to overfitting given gisette's small size. In this subsection, we analyze how the performance of FeT varies with different noise scales (σ) and sampling rates on secondary parties, demonstrating the impact of privacy constraints on its accuracy. The results are depicted in Figure 8 . We observe three key points: First, a moderate sampling rate has a negligible effect on model performance and may even slightly improve performance (e.g., on the MNIST dataset) by reducing overfitting. Second, despite increasing noise levels and enhanced privacy guarantees, FeT consistently outperforms baseline models. Third, the ε -σ privacy-noise curves illustrate that solely adding Gaussian noise without MPC, even with advanced analysis theorems such as Rényi Differential Privacy (RDP) [32] , would require much larger noise scales compared to our approach that integrates MPC.  
conclusion: 
 In this study, we introduce the Federated Transformer (FeT), specifically designed to support multiparty VFL while effectively addressing critical challenges related to performance, privacy, and communication. Furthermore, we provide theoretical proof of FeT's differential privacy, ensuring that data representations remain protected from secondary parties. Notably, our experiments demonstrate that FeT surpasses baseline models, even under stringent privacy guarantees and within the traditional two-party setting, establishing its efficacy and robustness in complex federated environments.Broader Impact. The architecture of FeT, even without privacy mechanisms, has potential applications in multimodal learning. Multimodal tasks often require the alignment of data records across different modalities, which can be quite challenging. For instance, aligning 24Hz video streams with 48kHz audio tracks is complex, as each video frame may correspond to a range of audio samples. FeT has shown its capability to effectively learn from such fuzzily aligned data. Furthermore, the transformer model has proven effective across various data types, including images, text, and tabular data, highlighting FeT's suitability for multimodal learning applications. A Proof Theorem 3. For certain constants c 1 and c 2 , given a sampling rate q, the total number of epochs T , and the number of batches B in each epoch, each representation R S k achieves (ε, δ)-differential privacy for all ε < c 1 q 2 T , with any δ > 0, by selecting the standard deviation σ of the Gaussian noise mechanism as follows: 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: See Section 8.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risk.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We have cited the used datasets in Section 7.1 and list the licenses in Appendix I.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: EQUATIONProof. The proof leverages the moments accountant bound [1] , which is applicable to a sequence of Gaussian mechanisms applied to subsampled data. We begin by establishing that the output's ℓ 2 -norm for each function is constrained by a constant C. This constraint ensures that each randomized function adheres to differential privacy under the Gaussian mechanism. By determining the cumulative count of Gaussian mechanisms applied, we can directly invoke Theorem 2 to reach our conclusion.To clarify the process, we apply norm clipping to each party as specified in Section 6, scaling each party at a rate of C/k. This scaling guarantees that, for every Rh i , the condition ∥ Rh i ∥ 2 ≤ C/k is satisfied. Using the triangle inequality within normed vector spaces, we derive:EQUATIONSince the ℓ 2 -norm of H i is bounded by C, adding Gaussian noise to H i satisfies the conditions for differential privacy. Throughout the training, B • T independent noises are introduced, resulting in a sequence of Gaussian mechanisms targeting a randomly selected subset at a ratio q. Consequently, Equation 1 is derived by directly applying Theorem 2.Datasets. In this section, we include the detailed information of each dataset used in the experiment. These real-world datasets are obtained in the same manner as those utilized in FedSim, with each dataset comprising two parties. Details of the real datasets are presented in Table 2 . The synthetic dataset, gisette [16] , consists of 6,000 instances and 5,000 features and serves as a binary classification task with balanced labels. The MNIST dataset [24] consists of 60,000 instances and 28x28 features for a 10-class digit classification task. Hyperparameters. Each algorithm was run until convergence, with a maximum of 50 to 100 epochs. The learning rate and weight decay were consistently set at 10 -3 and 10 -5 , respectively. For the Solo model, a multi-layer perceptron (MLP) with two hidden layers of 400 units each was employed. In contrast, the Top1Sim utilized a single-layer MLP with a hidden size of 200 for both local and aggregation models. For FedSim, the number of K neighbors was selected from the set {50, 100}. For FeT, the number of blocks is set to 6 for both local model and aggregation model.Environments. We evaluate FeT on a server equipped with dual Intel Xeon Gold 6346 CPUs, eight A100 GPUs with CUDA version 12.2, and 1008GB RAM, running Python 3.10.13 with PyTorch 2.1.1+cu121 on Linux kernel 6.5.0. Efficiency experiments were conducted on a machine powered by a 64-core Intel(R) Xeon(R) Gold 6226R CPU @ 2.90GHz and 376 GB of RAM. Each experiment was run five times, and we report the average and standard deviation.In this section, we evaluate the performance improvement of each proposed component of the FeT.Our findings indicate that dynamic masking is crucial for enhancing performance, while both party dropout and positional encoding averaging contribute modestly to these improvements. Detailed analyses are provided below.We evaluate the performance of FeT with and without dynamic masking, as shown in Table 3 .The evaluation includes two-party datasets (house, bike, hdb) and two 50-party synthetic datasets (MNIST and gisette). The results indicate that dynamic masking leads to an improvement of up to 13 percentage points, particularly noticeable in datasets with a large number of parties. This suggests that dynamic masking significantly enhances model performance, especially in multi-party settings.Next, we evaluate the effect of the dropout rate under specific hyperparameter settings: the number of parties k = 50, the number of neighbors K = 100, and key noise set to 0.05. The impact of the party dropout rate is demonstrated in Figure 9 and Table 4 . Our observations reveal that a moderate party dropout rate of 0.6 enhances FeT's generalization by reducing the model size. Notably, FeT maintains stable performance even as the dropout rate increases to 0.8. This indicates that party dropout not only improves generalization but also significantly reduces communication overhead across parties.Based on these findings, we set the dropout rate to 0.6 by default in multi-party experiments. We now assess the effect of the frequency of positional encoding (PE) averaging, as depicted in Figure 10 and Table 5 . We find that PE averaging yields improvements, particularly with a large number of parties, such as 50 on MNIST, where alignment of encodings becomes crucial. Based on our observations, we set the frequency to 1 in most experiments. Additionally, we assess the impact of positional encoding on the performance of FeT, as detailed in Table 3 . These evaluation of MNIST and gisette are conducted with the number of neighbors K = 100 and key noise 0.05. The results indicate that positional encoding is important for enhancing the performance of FeT.We evaluate the impact of identifier fuzziness on FeT's performance by introducing Gaussian noise of varying scales to the keys. The results are presented in Figure 11 . From the figure, we derive two key observations: (1) Both FeT and baseline models show improved performance in more balanced scenarios. (2) FeT consistently outperforms the baselines across different levels of heterogeneity, demonstrating its robustness to varying degrees of noise. These findings highlight the resilience of FeT in the presence of noise, which is critical for practical applications.To evaluate the comparative performance of the SplitAvg (without noise) and SplitNN, we conducted training for both models using identical hyperparameters on the same VFL dataset, gisette. The outcomes are illustrated in Figure 12 . Analysis of the figure yields two primary observations. Firstly, SplitNN and SplitAvg exhibit very similar loss and accuracy curves during training, indicating that both models behave very similarly. Secondly, upon expanding the number of participating parties to 128, we observe that the performance curves of both models remain closely aligned, albeit with the split-sum model exhibiting a marginally lower accuracy. This minor discrepancy is attributed to the increased model parameters in SplitNN, which can typically be compensated for by increasing the number of parameters in SplitAvg.While FeT primarily focuses on scenarios involving fuzzy linkage, we also evaluate its robustness in exact linkage contexts. To achieve this, we synthesize exact linkage data by generating pure random keys within the range of [-1, 1] without introducing any noise. Each party is randomly divided into five or ten groups, with each group containing an equal number of features. Importantly, each party retains the exact keys, ensuring a controlled environment for our evaluation.The results of our experiments are summarized in Table 6 . From the table, we observe that Top1Sim achieves the highest accuracy, as it is inherently well-suited for exact linkage scenarios. In contrast, Parameter Efficiency. In our analysis, we assess the computational efficiency of standard addition compared to multi-party computation (MPC) addition, as shown in Table 7 . Under the arithmetic GMW protocol [15] , and given that the size of the aggregated vector varies by dataset, we use a typical size for our experiments. Specifically, we conduct MPC addition to aggregate 10,000-dimensional vectors from multiple parties. Each experiment is performed five times, with the average timing reported. Although MPC generally incurs higher computational requirements, the results in Table 7 indicate that aggregating high-dimensional vectors via MPC incurs only a one-second overhead, even as the number of parties increases to 100. This minimal time cost is relatively small, especially when compared to other factors such as communication costs. Therefore, our findings suggest that MPC remains a feasible and efficient approach for representation aggregation in the context of VFL. In this section, we explore how the performance of FeT varies with different noise scales σ and secondary sampling rate, illustrating the influence of privacy constraints on its accuracy. The outcomes are depicted in Figure 13 . From this figure, we observe two key points. First, for large secondary datasets like bike, a moderate sampling rate has a negligible effect on model performance.Conversely, for smaller secondary datasets like hdb, performance is quite sensitive to sampling rates. Second, as the noise scale increases for secondary parties, the performance of FeT does not degrade sharply; instead, it gradually converges to a state where only primary features are utilized due to our dynamic masking design. In this scenario, FeT also outperforms MLP-based Solo primarily due to the transformer's key encoding, which has proven to be more effective than incorporating all keys into the training process, as evidenced in spatial-temporal prediction tasks [9] . Next, we explore the relationship between σ and ε as outlined in Theorem 3, setting hyperparameters to reflect typical training conditions. The number of epochs is chosen based on common convergence epochs: 10 for bike, and 50 for house and hdb. We adopt a batch size of 8k and set δ to 1/N , with N representing the size of party S 1 . This correlation between ϵ and σ is depicted in Figure 14 . The figure illustrates that reasonable noise levels can yield robust privacy guarantees. For instance, within a noise scale conducive to maintaining competitive performance, FeT achieves ε = 3 for hdb and ε = 5 for house, indicating effective privacy preservation under practical noise conditions.The preceding experiments were conducted using a balanced feature split for VFL. Building on this foundation, we extended our evaluation of FeT to include datasets with varying levels of imbalance, motivated by the recent benchmarks presented in VertiBench [49] . The MNIST datasets are divided by features according to the methodology described in VertiBench [49] , utilizing imbalance parameters α ∈ {0.1, 0.5, 1.0, 5.0, 10.0, 50.0}, where a higher α value denotes greater balance across parties. The findings, illustrated in Figure 15 , lead to two key observations: firstly, both FeT and the baseline algorithms exhibit improved performance in more balanced scenarios. Secondly, despite the varying The design of FeT includes three primary limitations that warrant careful consideration. First, FeT operates under the assumption that common features exist across all parties. While this assumption is valid in many scenarios, it may not hold in more complex situations where the parties lack a shared set of features. This limitation necessitates further investigation into alternative frameworks or adaptations that can accommodate such cases, particularly in heterogeneous environments.Second, although FeT facilitates the application of scalable differential privacy across multiple parties, the stringent privacy safeguards can lead to significant accuracy reductions when operating with low values of ε. This trade-off between privacy and utility is particularly concerning in performancesensitive applications, where quantifying the extent of accuracy loss is essential for informing users about the potential impacts on their results. Future work should explore methods to balance privacy and accuracy more effectively.Third, similar to other fuzzy VFL methods [50] , FeT assumes a correlation between identifiers and data representations. This assumption may not hold in cases where identifiers are randomly generated, which could lead to overfitting and minor performance deficits compared to Top1 approaches. Experiments on such datasets (Appendix D) indicate that while FeT performs well in many scenarios, its effectiveness may vary significantly depending on the nature of the data and the key generation process. Therefore, further empirical studies are needed to assess FeT's robustness across diverse datasets and identifier generation strategies.The licenses of the datasets used in this work are presented in Table 9 . We utilize the code from FedSim [50] as our baseline, which is licensed under the Apache V2 license2 . Our own code will also be open-sourced under the Apache V2 license. Justification: We have stated in abstract and introduction that the paper proposes a novel FeT framework to address the scalability and privacy issues in Vertical Federated Learning (VFL).Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.https://github.com/Xtra-Computing/FeThttps://www.apache.org/licenses/LICENSE-2.0 


Decision: This paper tackles the problem of multi-party VFL for fuzzily linked data (i.e., where the matching of samples across VFL clients is imperfect).
While multi-party or fuzzy VFL have been researched in prior work, the combination of the two has not.
The paper argues the multi-party problem introduces unique challenges such as more complex fuzzy linkage and higher privacy loss and seeks to mitigate these issues through several strategies including positional encoding averaging, dynamic masking of incorrect pairs, and SplitAvg that combines encryption and noise-based approaches to privacy.

The reviewers were generally positive about the paper after better understanding the problem and motivation. Clarification regarding the practical applicability and context for multi-party fuzzily linked data was noted by multiple reviewers and should be included in the final paper. Additionally, the authors should change the title to include "multi-party" as they suggested and also incorporate this into the paper more carefully to avoid overclaiming their contributions.

While there were some concerns regarding computational issues or experimental conditions, the real-world VFL datasets and author response reasonably addressed these concerns. I would recommend incorporating these discussions into the final paper.

In summary, this paper provides an interesting and effective solution to the multi-party fuzzy linked problem using a distributed transformer architecture that has practical real-world relevance for VFL scenarios.

Reviews: review: paper_summary: paper_summary: This paper proposes the Federated Transformer (FeT) framework, which aims to address performance and privacy challenges in multi-party fuzzy vertical federated learning (VFL). FeT leverages the Transformer architecture to encode fuzzy identifiers and distribute training across different parties. The authors introduce three innovative techniques—position encoding averaging, dynamic mask module, and party dropout strategy—to enhance model performance while minimizing computational and communication overhead. Additionally, FeT incorporates a scalable privacy framework that combines differential privacy and secure multi-party computation, effectively safeguarding local data representations and ensuring manageable privacy maintenance costs. Experimental results demonstrate that FeT outperforms baseline models, offering superior performance and enhanced privacy protection. Overall, FeT overcomes the limitations of existing models in multi-party fuzzy VFL, showcasing exceptional performance and practicality.
strength_weakness: strength_weakness: This manuscript proposes a scalable vertical federated learning for practical fuzzy linked data, aiming to solve the performance and privacy issues of real-world VFL in its application. However, there are still some major issues, as follows:

1. In the Introduction section, the definition of fuzzy identifiers and multi-party fuzzy VFL and the relationship between the identifiers, fuzzy identifiers and multi-party fuzzy VFL is not clearly introduced, which may lead to confusing concepts for readers. In addition, the important role of fuzzy identifiers in linking real-world data sets should be emphasized to prove the practical value of studying multi-party fuzzy VFL.

2. In the training of FeT, PPRL was first used to evaluate the identifier similarity between the primary participant P and each secondary participant. However, the author did not mention this method in the previous content. Based on this, a very important question is: in the comparative experiment, the author did not seem to compare the performance gap, privacy gap, and computing resource consumption gap between the training using only the PPRL method and the training based on the transformer structure in this article?

3. In FeT, the combination of dynamic mask, participant discarding mechanism, and hybrid privacy protection mechanism will inevitably affect the training process's time, computing resource consumption, and privacy protection ability. Therefore, it is also important to compare the training efficiency and computing resource consumption, as well as the privacy protection ability of FeT methods with current mainstream methods. This represents whether FeT can surpass other methods in real-world applications. The setting of the experiment may be as follows: 1. Training time of FeT and mainstream methods on the same data set and under the same conditions. 2. CPU/GPU usage between FeT and mainstream methods in the training process to measure memory consumption and network communication overhead, especially in the case of multi-party participation. 3. Evaluate the privacy protection effect of FeT and mainstream methods through quantified privacy protection indicators (such as ε value of differential privacy).

Minor weaknesses:
1. In lines 27 and 28, the authors describe collaboration between hospitals, financial institutions, and sensors. It is equivalent to juxtaposing the three, which is unreasonable.
2. In lines 58 and 59, the authors mention that experiments have shown that model accuracy has been improved by up to 13 percentage points on a 50-square fuzzy VFL on the MNIST dataset. It is not given here which method is compared with, please add.
3. In line 139, the author defines a common feature shared by all parties as an identifier, expressed as \(x^i = [k^i, d^i] \), where \([\cdot]\) signifies. \([,] \) and\([\cdot]\) is not the same
questions: questions: same as the weakness
limitations: limitations: NA
paper_summary: paper_summary: This paper introduces the Federated Transformer (FeT) framework, which is designed to support multi-party VFL.
It enhances the efficiency of model training among multiple parties using fuzzy identifiers while ensuring data privacy.
Experiment results show that FeT performed well when it scaled to 50 parties and the authors also provide theoretical proof of the differential privacy.
strength_weakness: strength_weakness: **Weaknesses of the Paper**

**Model Design**

1) The introduction of a trainable dynamic masking module aims to improve the exclusion of incorrectly linked data records. However, if this module is not well-optimized, it could introduce additional noise or errors which degrades the model performance.

2) The proposal of positional encoding averaging may not available in all VFL scenarios, which may cause inconsistencies in data processing which may affect the model training.

**Privacy Trade-offs**

The paper says that stringent privacy safeguards may cause accuracy reductions, particularly with low values of $\epsilon$. It would be better if the authors could provide a more detailed analysis of the trade-offs between privacy and performance.

**Theoretical Proof**

While the paper provides some theoretical proofs, it may not offer a complete set of proofs for all claims made. The theoretical section did not adequately discuss the limitations of the proposed methods or the implications of violating the underlying assumptions. A more thorough exploration of these aspects would enhance the robustness of the theoretical framework.

**Experiment**

The experiments are conducted on a small number of datasets, which may not represent the full range of scenarios in VFL. Furthermore, The experiments explore little the trade-off between privacy guarantees and model utility, especially at low values of ε in differential privacy.
questions: questions: **Questions**

1\. **Scenario Application**:

How can FeT be adapted to work with unstructured data types such as text or images? Is there any preliminary results or insights on applying FeT to these types of data?

2\. **Privacy-Performance Trade-off**:
Can you provide a more detailed analysis of the trade-off between privacy and performance, especially at different levels of $\epsilon$?

3\. **Model Performance on datasets with different features**:
How does the performance of the Federated Transformer (FeT) vary when applied to datasets with significantly different feature distributions across multiple parties?
limitations: limitations: The authors have addressed the limitations of their work by identifying three primary limitations related to the assumptions of common features across parties, the potential accuracy reductions due to stringent privacy safeguards, and the correlation between keys and data representations.

Here are some other potential limitations that may be considered:

**Large scale cases**: The model's performance may have the risk of degrading with a large number of parties or larger datasets, which may limit its application in certain large federated learning scenarios.

**Privacy Trade-offs**: While the model is designed to enhance privacy, there may still be some vulnerabilities to be exploited, especially when the underlying assumptions about data sharing are violated.
paper_summary: paper_summary: This paper investigates vertical federated learning (VFL) linked with fuzzy identifiers and develops a new framework called Federated Transformer (FeT). The proposed framework leverages the transformer architecture to encode the identification information and distributes the subnets of the transformer across parties.
strength_weakness: strength_weakness: 1.	The paper is not well motivated. It would be great if the paper could provide concrete use cases of fuzzy VFL, that involves fuzzy data/linkages while requiring the privacy needs of distributed machine learning. The German Record Linkage Center might not be an appropriate example for federated learning.
2.	Several concepts in fuzzy VFL need further clarification. For example, what strategies are used in existing fuzzy linkage? How are the fuzzy identifiers presented? How is privacy preserved in fuzzy identifiers?
3.	The design rationale behind dynamic masking is unclear. Why is dynamic masking necessary? How does it function? 
4.	The novelty of party dropout needs further elaboration. Dropout is widely used in federated learning. How does the proposed approach differ from existing designs? 
5.	It is unclear why the proposed framework employs two different privacy mechanisms: differential privacy and MPC. What unique challenge is each privacy mechanism designed to address?
questions: questions: 1.	Please provide more concrete use cases of fuzzy VFL.
2.	Please clarify the key concept in fuzzy VFL.
3.	The design rationale behind the key components in the proposed framework is unclear. Please see my comments above and articulate them in more detail.
limitations: limitations: N/A
paper_summary: paper_summary: The paper introduces the Federated Transformer (FeT), a novel framework for Vertical Federated Learning (VFL) that addresses the challenges of fuzzy identifiers in multi-party scenarios. It incorporates three key innovations to enhance performance, privacy, and reduce communication overhead: positional encoding averaging, dynamic masking, and a party dropout strategy. The approach significantly outperforms baseline models in scalability and privacy preservation.
strength_weakness: strength_weakness: 1.The performance improvements reported are impressive; however, how dependent are these improvements on the initial conditions of data alignment and the distribution of fuzzy identifiers? Could the authors provide insights on the robustness of the Federated Transformer (FeT) under less ideal conditions?
2. While the paper discusses scalability extensively, there is less focus on the computational resources required. Could the authors comment on the computational overhead and the practicality of deploying FeT in real-world scenarios with potentially limited resources?
3.The FeT introduces several complex mechanisms such as dynamic masking and positional encoding averaging. How do these additions affect the training time and complexity of the model? Is there a significant trade-off between performance and efficiency?
4.The experiments are conducted primarily on synthetic datasets. How well does FeT generalize to other real-world datasets, particularly those with higher levels of noise and less structured data?
questions: questions: Please see Weakness and Limitations.
limitations: limitations: 1.I think you can give us a better case diagram to help us better understand your new task.
2.I think there is an update error in line 16 of your algorithm.
3.In the Party Dropout section of the article, around line 188, you said that the communication overhead on the primary party can be reduced by up to 80%, but I did not find the corresponding experiment or proof in the article. I think this part needs to be better explained.
4.The experimental settings focus on synthetic and controlled environments. Real-world applications might introduce variables not accounted for in this study, potentially affecting the generalizability of the results.
5.The complexity of the model and the required computational resources are not thoroughly discussed, which could be a limitation for practical deployment in resource-constrained environments.
6.While the model addresses fuzzy identifiers, the performance heavily relies on the quality of the linkage, which may not always be feasible or accurate in real-world scenarios.

Comments: title: Summary of Rebuttal at the Three-Day Deadline for Author-Reviewer Discussion
comment: Thanks for your responses to address my concerns.
Thanks for your response. According to your response and other reviewer's comments, I change the rating as "Borderline Reject".
Thank you for your responses. Most of my concerns have been addressed. I will raise my score.
We understand the time and effort required to review our rebuttal. To assist you during the discussion phase, we have provided a brief summary of the major concerns and our responses for your convenience. With only **three days** remaining, we sincerely appreciate your consideration of our response and kindly request that you let us know if you have any further comments.

### Pros
- Reviewers `jPVu`, `sPbx`, and `so53` recognize the novelty of our framework design.
- Reviewers `jPVu`, `sPbx`, and `P7Jg` acknowledge our significant performance improvements over existing approaches.
- Reviewer `sPbx` appreciates the broader impact of this paper on multimodal learning.
- Reviewer `jPVu` highlights the effectiveness of our proposed privacy mechanism—SplitAvg—in reducing noise while preserving the same level of privacy.

### Major Concerns and Our Response

**Efficiency of FeT**: We compared FeT's computational and memory efficiency with FedSim, the state-of-the-art fuzzy VFL model, as shown in Table 11 of "rebuttal.pdf". We find that FeT delivers **better performance and improved memory efficiency with a similar number of parameters** compared to FedSim, despite a slightly lower training speed.

**FeT's Reliance on Linkage Quality**: FeT is **significantly less reliant** on initial linkage quality than existing VFL models, as demonstrated by three pieces of evidence in "rebuttal.pdf":
- **Ablation Study on K** (Figure 14): FeT consistently outperforms baselines even when many unrelated data records are included (K > 50), indicating resilience to low linkage quality.
- **Performance with Different Fuzziness** (Figure 12): FeT outperforms baselines under conditions of moderate to high noise in identifiers, showcasing strong resilience to fuzziness.
- **Dynamic Masking Visualization** (Figure 11): The visualization demonstrates that dynamic masking plays a crucial role in handling low-quality linkage, effectively focusing on a few close and relevant records among 4,900 fuzzily linked ones.

**Performance of FeT in "Less Ideal" Cases**: FeT performs better in **general fuzzy VFL scenarios** but slightly underperforms Top1Sim [4,5] in **ideal cases with precisely matched identifiers**, as evidenced by Figure 12 in "rebuttal.pdf" and Appendix D.

**Real Application of Multi-party Fuzzy VFL**: In Figure 15 of "rebuttal.pdf," we present a real-world application involving travel cost prediction in a city, achieved through collaboration among taxi, car, bike, and bus companies.

**Design of Dynamic Masking**: We used an ablation study (Table 3 of Appendix C.1) to show that **dynamic masking is necessary**; without it, accuracy may drop by up to 13%. Additionally, a visualization in Figure 11 of "rebuttal.pdf" shows that **dynamic masking learns to focus on a sample-specific localized area around the primary party's identifiers**.

**VFL with Limited Resources**:
1. **FeT's Limited GPU Memory Performance**: Table 11 of "rebuttal.pdf" shows that FeT performs well even with GPU memory under 1 GB, making it viable for resource-constrained VFL scenarios.
2. **Cross-Silo VFL Prevalence**: Cross-device VFL with limited resources is rare. VFL typically occurs in cross-silo settings [3], involving dozens to hundreds of parties with adequate computational power. Discussions with industry collaborators providing commercial VFL services confirm that it is uncommon for a single user to have features spread across thousands of distinct parties.

**Prevalence of Multi-party Fuzzy VFL in the Real World**: Traditional VFL assumes that each data record represents a **user**, making the assumption of a universal user ID natural. However, in real-world applications, these records often represent **various real-world objects**—such as a room (e.g., `house` and `hdb` datasets) or a travel route (e.g., `taxi` dataset).

Reviewers have also provided many useful comments on topics such as privacy, definitions, heterogeneous features, etc. We have addressed all these concerns in detail in our response to each reviewer. We greatly appreciate the reviewers' time and efforts in helping us improve this manuscript and are eager to know if you have any further comments regarding our response. Given the three-day constraint, your early reply would be greatly appreciated.