Title: Federated Transformer: Multi-Party Vertical Federated Learning on Practical Fuzzily Linked Data
Venue: NeurIPS 2024
Accepted: yes
Keywords: vertical federated learning

federated learning

transformer

record linkage

entity alignment

differential privacy

fuzzy alignment
Abstract: Federated Learning (FL) is an evolving paradigm that enables multiple parties to collaboratively train models without sharing raw data. Among its variants, Vertical Federated Learning (VFL) is particularly relevant in real-world, cross-organizational collaborations, where distinct features of a shared instance group are contributed by different parties. In these scenarios, parties are often linked using fuzzy identifiers, leading to a common practice termed as _multi-party fuzzy VFL_. Existing models generally address either multi-party VFL or fuzzy VFL between two parties. Extending these models to practical multi-party fuzzy VFL typically results in significant performance degradation and increased costs for maintaining privacy. To overcome these limitations, we introduce the _Federated Transformer (FeT)_, a novel framework that supports multi-party VFL with fuzzy identifiers. FeT innovatively encodes these identifiers into data representations and employs a transformer architecture distributed across different parties, incorporating three new techniques to enhance performance. Furthermore, we have developed a multi-party privacy framework for VFL that integrates differential privacy with secure multi-party computation, effectively protecting local representations while minimizing associated utility costs. Our experiments demonstrate that the FeT surpasses the baseline models by up to 46\% in terms of accuracy when scaled to 50 parties. Additionally, in two-party fuzzy VFL settings, FeT also shows improved performance and privacy over cutting-edge VFL models.

Federated Transformer: Multi-Party Vertical Federated Learning on Practical Fuzzily Linked Data

Abstract

Federated Learning (FL) is an evolving paradigm that enables multiple parties to collaboratively train models without sharing raw data. Among its variants, Vertical Federated Learning (VFL) is particularly relevant in real-world, crossorganizational collaborations, where distinct features of a shared instance group are contributed by different parties. In these scenarios, parties are often linked using fuzzy identifiers, leading to a common practice termed as multi-party fuzzy VFL . Existing models generally address either multi-party VFL or fuzzy VFL between two parties. Extending these models to practical multi-party fuzzy VFL typically results in significant performance degradation and increased costs for maintaining privacy. To overcome these limitations, we introduce the Federated Transformer (FeT) , a novel framework that supports multi-party VFL with fuzzy identifiers. FeT innovatively encodes these identifiers into data representations and employs a transformer architecture distributed across different parties, incorporating three new techniques to enhance performance. Furthermore, we have developed a multi-party privacy framework for VFL that integrates differential privacy with secure multi-party computation, effectively protecting local representations while minimizing associated utility costs. Our experiments demonstrate that the FeT surpasses the baseline models by up to 46% in terms of accuracy when scaled to 50 parties. Additionally, in two-party fuzzy VFL settings, FeT also shows improved performance and privacy over cutting-edge VFL models.

1 Introduction

Federated Learning (FL) is a learning paradigm that enables multiple parties to collaboratively train a model while preserving the privacy of their local data [27]. Among its various forms, Vertical Federated Learning (VFL) [53] is particularly prevalent form in real-world applications as highlighted in a recent technical report [48]. In VFL, participants possess different features of the same set of instances, where common features, such as names or addresses, serve as identifiers (a.k.a. keys ) to link datasets across these parties.

Real-world applications often necessitate multi-party fuzzy VFL , characterized by two key attributes. First, it supports collaboration among multiple parties , commonly observed in collaborations across hospitals [33], sensors [52], and financial institutions [38]. Second, it accommodates scenarios where these parties are linked using fuzzy identifiers, such as addresses. Such scenarios are prevalent in applications, as illustrated in an analysis [50] of the German Record Linkage Center [11]. For instance, multiple vehicle rental companies that are fuzzily linked by source and destination addresses in the same city can collaborate to predict travel times.

To illustrate the significance of multi-party fuzzy VFL, consider the application of travel cost prediction in a city through collaboration among taxi, car, bike, and bus companies, as shown in Figure 1. Since personal travel information is private and cannot be shared, VFL is essential.

Bike

Car

Bus

Taxi

Route identifiers

A

(start lon., startlat.,

Task-Travel CostPrediction:

end lon.endlat.)

Whatisthecostoftraveling

UniqueFeatures

Rental

GasCost

#Passengers

Price

fromAtoB?

Availability

Additionally, route identifiers - starting and ending GPS locations - can only be linked using fuzzy methods. However, linking closely related source and destination points with multi-party fuzzy VFL can significantly enhance prediction accuracy.

Existing VFL approaches generally address either the multi-party aspect or the fuzzy identifier issue. Several methods [33, 52, 38, 21, 26] facilitate multi-party VFL using Private Set Intersection (PSI) [10] to link datasets. These methods often presume the existence of precise, universal keys, which are not feasible in common VFL scenarios involving fuzzy identifiers. Conversely, other studies [17, 34, 50] propose two-party fuzzy VFL models that utilize cross-party key similarities for training. However, when extended to multi-party fuzzy VFL, these similarity-based approaches encounter significant challenges in performance and privacy. While some methods achieve reasonable performance, they often compromise privacy or incur prohibitive costs.

Despite the potential of multi-party fuzzy VFL, several significant challenges must be addressed for effective implementation. First, as the number of parties with fuzzy identifiers increases, maintaining performance becomes increasingly challenging. The addition of parties leads to a quadratic growth in the number of key pairs, an increase in incorrect linkages between fuzzy identifiers, and larger model sizes. These factors collectively heighten the risk of overfitting and adversely impact model performance. Second, the rising costs of preserving privacy intensify as more parties with correlated data participate, leading to either significant computational costs [33, 52, 38] or accuracy loss [47]. Third, in a collaboration of multiple parties, a communication bottleneck arises for the party with labels, termed the primary party . This party must communicate with all other parties without labels, termed secondary parties , in each training round, placing substantial demands on the primary party's bandwidth. These challenges significantly hinder the practical deployment of VFL.

To address these issues, we introduce the Federated Transformer (FeT) to enhance performance and reduce privacy costs in multi-party fuzzy VFL. First, to tackle performance issues, we encode key similarities into data representations aligned by positional encoding averaging , which eliminates the need for quadratic calculations of key pairs. Additionally, we have designed a trainable dynamic masking module that automatically filters out incorrectly linked pairs, enhancing model accuracy by up to 13% in 50-party fuzzy VFL on the MNIST dataset. Second, to mitigate the escalating costs of privacy protection, we introduce SplitAvg, a hybrid approach that merges encryption-based and noise-based methods, maintaining a consistent noise level regardless of the number of participating parties. Third, to alleviate communication overhead on the primary party, we implement a party dropout strategy, which randomly excludes certain secondary parties during each training round. This effectively reduces communication costs by approximately 80% and improves model generalization. Our codes are available on GitHub 1 . In summary, our contributions are as follows:

We design Federated Transformer (FeT) , a novel model achieving promising performance under multi-party fuzzy VFL.

We introduce SplitAvg to enhance the privacy of FeT by protecting local representations in multi-party fuzzy VFL, with a theoretical proof of its differential privacy.

Experimental results demonstrate that FeT significantly outperforms baseline models by up to 46% in terms of accuracy in 50-parties VFL. Moreover, while providing enhanced privacy, FeT consistently surpasses state-of-the-art models even in traditional two-party fuzzy VFL scenarios.

2 Preliminaries

In this section, we provide the foundational concepts necessary for understanding our approach to differential privacy. Differential Privacy (DP) offers a rigorous mathematical framework for preserving individual privacy. It quantifies privacy in terms of the probability of producing the same output from two similar datasets that differ by exactly one record.

Definition 1. Consider a randomized function M : R d → O and two neighboring databases D 0 , D 1 ∼ R d that differ by exactly one record. For every possible output set O ⊆ O , M satisfies ( ε, δ ) -differential privacy if

A single query that adheres to differential privacy is termed a mechanism . For example, the Gaussian mechanism [4] is commonly used to achieve DP by adding Gaussian noise to the output of the function.

Theorem 1 (Gaussian Mechanism [4]) . For a function f : X → R d characterized by a global L 2 sensitivity ∆ 2 , which signifies that the maximum difference in the L 2 -norm of the outputs of f on any two neighboring databases is ∆ 2 , and for any ε ≥ 0 and δ ∈ [0 , 1] , the Analytic Gaussian Mechanism is defined as M ( x ) = f ( x ) + Z , where Z ∼ N (0 , σ 2 I ) . This mechanism satisfies ( ε, δ ) -differential privacy if Φ ( ∆ 2 2 σ -εσ ∆ 2 ) -e ε Φ ( -∆ 2 2 σ -εσ ∆ 2 ) ≤ δ, where Φ( t ) = 1 √ 2 π ∫ t -∞ e -x 2 / 2 dx is the cumulative distribution function (CDF) of the standard univariate Gaussian distribution.

When multiple queries are made on the same database, independent Gaussian noise is added to each query to maintain differential privacy. The privacy loss of the composition of Gaussian mechanisms is formulated in Theorem 2.

Theorem 2 (Moments Accountant [1]) . There exist constants c 1 and c 2 so that given the sampling probability q = L N and the number of steps T , for any ε < c 1 q 2 T , DPSGD [1] is ( ε, δ ) -differentially private for any δ > 0 if we choose σ > c 2 q √ T log(1 /δ ) ε .

3 Related Work

Performance. Traditional VFL approaches [29, 7] are typically limited to two-party scenarios. In contrast, existing multi-party VFL methods [12, 51, 33, 52, 38, 21, 26] often rely on the assumption of precise identifiers that ensure perfect alignment across all parties. These methods generally employ the SplitNN framework [45], where each party maintains a portion of the model, and the models are collaboratively trained on well-aligned data samples through the transfer of representations and gradients, commonly known as split learning. However, the requirement for perfect data alignment is impractical in many real-world scenarios [50, 3], where identifiers are often imprecise.

To address this limitation, semi-supervised VFL [22, 30, 19, 20] has emerged, attempting to improve model performance by leveraging unlinked records through semi-supervised or self-supervised learning. However, these methods still assume that datasets from each party can be precisely linked using exact identifiers, a premise that is often untenable in real-world settings [50, 3]. Given that the quality of linkage significantly impacts VFL accuracy [34], exploring effective linkage methods remains a pivotal issue in VFL.

On the other hand, FedSim [50], based on real linkage projects at the German Record Linkage Center (GRLC) [3], acknowledges that the keys of each party are usually not precisely linkable and that records may have one-to-many relationships, leading to fuzzy linkage scenarios, as seen with keys like GPS addresses. FedSim enhances training performance by performing soft linkage and conducting training based on transmitted key similarities. Nonetheless, it faces limitations in scalability beyond two parties and introduces new privacy concerns by directly transferring similarities.

In summary, while existing studies face significant performance challenges when handling fuzzy keys in multi-party settings, our proposed FeT demonstrates a scalable design that addresses these challenges and shows promising performance improvements in both multi-party fuzzy VFL and two-party settings compared to FedSim.

Privacy. The privacy concerns associated with VFL are multifaceted. First, the primary party may infer data representations from secondary parties [31]. Second, the secondary party may derive gradients from the primary party [41, 54]. Third, external attackers could conduct membership inference attacks [39] on the deployed model [51]. This paper primarily addresses the second concern: safeguarding representations, while acknowledging other concerns as open challenges.

To address the privacy of representations in VFL, various methods have been proposed, falling into two primary categories: encryption-based methods and noise-based methods. Encryption-based methods [26, 14, 33, 52, 38, 21, 36] utilize computationally intensive cryptographic techniques to encrypt intermediate results. However, these methods often incur significant communication overhead when scaled to multiple parties. Conversely, noise-based methods [47, 46] protect data by perturbing [47] or manipulating [46] local representations. These methods typically do not provide theoretical privacy guarantees or require substantial amounts of noise when scaling to multiple parties in VFL, which can degrade performance. Unlike existing studies that rely solely on either approach, this paper explores a combined strategy incorporating both encryption-based and noise-based methods, ensuring the model scales effectively to multiple parties without the need for excessive noise.

4 Problem Statement

In this section, we formalize the concept of multi-party fuzzy VFL. We consider a supervised learning task where one party holding labels, termed the primary party P , collaborates with k parties that do not hold labels, referred to as the secondary parties . The primary party P possesses n data records denoted as x P := { x i } n i =1 along with corresponding labels y := { y i } n i =1 . Each secondary party S k maintains its own dataset x S k . All parties share common features, referred to as identifiers, expressed as x i = [ k i , d i ] , where [ · ] signifies concatenation. These identifiers k i may exhibit inaccuracies and fuzziness, despite residing within the same range.

In this formulation, L ( · ) denotes the loss function, θ represents the model parameters, and Ω( θ ) refers to the regularization term. The symbol n indicates the number of samples in the primary party P .

Threat Model. This study focuses on defending feature reconstruction attacks [25, 31], which target local representations shared with the primary party. FeT operates under the assumption that all participating parties are honest-but-curious , meaning they adhere to the protocol but may attempt to infer additional information about other parties. Furthermore, we assume that the parties do not collude with one another. While other forms of attacks, such as label inference attacks [13] and backdoor attacks aimed at compromising labels and gradients, exist, they are considered orthogonal to the objectives of this study. These additional threats will be explored in our future research.

5 Approach

In this section, we address the performance and communication challenges inherent in multi-party fuzzy VFL. To tackle these issues, we introduce a transformer-based architecture named the Federated Transformer (FeT). This model encodes keys into data representations, thereby reducing reliance on key similarities. To accurately exclude incorrectly linked data records, we propose a trainable dynamic masking module that generates masks based on keys. Furthermore, to combat overfitting caused by the large model and to alleviate communication bottlenecks faced by the primary party, we introduce a party dropout strategy that randomly invalidates some parties during training. Additionally, we identify a positional encoding misalignment issue across parties in the FeT and propose positional encoding averaging to ensure consistent alignment, thereby enhancing model performance.

5.1 Model Structure

The architecture of the FeT is illustrated in Figure 2. In FeT, each secondary party has an encoder, while the primary party has both an encoder and a subsequent decoder. The internal structure of both the encoder and decoder closely adheres to the conventional transformer model [43]. We utilize

Sum

FeedForward

Neural Network Layer

Data matrix

LayerNorm

Dense Layer

mask

Multi-head

Attention Block

attention

Dux

LayerNorm

Self-Attention Layer

Q

Dynamic

Securemulti-partysummation

QKV

masking

Attention Block

Self-Attention Layer

xnsa

mask

Self-Attention Layer

mask

Self-Attention Layer

Feed

Q

K

Feed

Q

Feed

Q

Forward

Forward

Forward

Multi-dimensional

Positional Encoding

PE

Encoding

PE

Encoding

PE

Encoding

key

Data xP

key

Data xS1

key

Data xSk

Primary Party P

Secondary Party S1

Secondary Party Sk

multi-dimensional positional encoding [28] to integrate key information into feature vectors. Outputs from the encoders at the secondary parties are aggregated and then fed into the decoder. Details regarding the privacy mechanisms employed during this aggregation phase are discussed in Section 6, while the details of the training process are elaborated in Section 5.2. We then elaborate on three techniques designed to improve performance and reduce communication costs.

Dynamic Masking. The size of the neighborhood varies significantly depending on the party and key values. Consequently, including a large number of neighbors K for all parties can hinder the model's ability to extract meaningful information and result in overfitting. To address this, we introduce a dynamic 'key padding mask' in the transformer, generated from the identifier values using a trainable MLP. This approach allows the model to effectively disregard distant data records, thereby eliminating the influence of irrelevant data when K is large. This concept resembles the weight gate in FedSim, but it diverges by using identifiers as inputs instead of similarities, enhancing privacy by preventing the transmission of similarity data across parties.

The learned dynamic masking is visualized in Figure 3. We derive two key insights from the visualization: (1) Dynamic masking effectively focuses on a localized area around the primary party's identifiers. Data records with distant identifiers on secondary parties (in cooler colors) receive small negative mask values, reducing their significance in the attention layers - without accessing the primary party's original identifiers. (2) The focus area varies in scale and direction across samples: for example, the left figure concentrates on a small bottom area, the middle figure on a small top area, and the right figure on a broad area in all directions.

Party Dropout. Extending the Federated Transformer (FeT) to support multiple parties can be challenging for several reasons. First, the communication bandwidth required by the primary party becomes a significant bottleneck within the SplitAvg framework, increasing linearly with the number of parties. Second, the inclusion of many parties can result in an excessive number of parameters, which may lead to overfitting. To mitigate these issues, we introduce the concept of party dropout . Inspired by traditional dropout [40], we randomly set a portion r d of the parties' representations to zero during training. This method serves as a form of regularization, thus helping to reduce overfitting, while also significantly cutting down on communication overhead. In our experiments, we demonstrate that increasing the party dropout rate to 0.8 leads to minimal accuracy loss and can

-1.00

-1.25

-1.50

-1.75

-2.00

-2.50

-3.00

-2.25

-2.75

★

Keys - Secondary

Key - Primary

30

20

-10

0

-10

Dynamic

Mask

-0.50

-1.00

1.00

0.00

1.50

0.50

★

Keys - Secondary

Key - Primary

25

20

15

10

5

0

-5

-10

namic

Value

Dyn

0.50

0.40

0.20

0.90

0.80

0.70

0.60

0.30

Keys - Secondary

Key - Primary

4

2

0

6

2

4

: Mask Value

Dynamic

-2.50

-2.00

-1.50

-1.00

1.50

2.00

2.50

3.00

0.20

0.30

0.400.50

0.60

0.70

0.80

0.90

(a) Keys: P 0 , encodings: P 1 ∼ P 3

(b) Keys: P 0 , encodings: P 0

(c) Keys: P 0 , encodings: P 3

PearsonCorr.=0.5054

PearsonCorr.=-0.0210

PearsonCorr.=0.0203

2.0

2.0

2.0

inputdistance

inputdistance

inputdistance

1.5

1.5

1.5

1.0

Po-Po

1.0

1.0

E

P

0.5

0.0

Po-P1

Po-P3

E

P

0.5

0.0

0.5

0.0

0.00

0.25

0.50

0.75

1.00

0.00

0.05

0.10

1.00

1.02

1.04

PE outputdistance

PEoutputdistance

PEoutputdistance

even improve accuracy. Consequently, the communication overhead on the primary party can be reduced by up to 80%, enhancing scalability when dealing with large numbers of parties.

Like traditional dropout, it is crucial to maintain consistent scaling of the representations during both training and testing phases. This consistency is naturally achieved within the SplitAvg framework. During the averaging process, if a ratio r d k of parties is set to zero, we adjust by dividing only by the number of non-zero parties, (1 -r d ) k . This method ensures that the scale of the averaged representations remains consistent across training and testing, regardless of the value of r d .

Positional Encoding Averaging. In positional encoding (PE), it is generally expected that the distances between encoded representations are positively correlated with the distances between identifiers. In FeT, each party employs its own encoder and PE layer, each tasked with encoding its local identifiers into representations. This configuration leads to significant PE misalignment issues, as illustrated in Figure 4. Although the identifiers and their corresponding encoded representations maintain a positive correlation within the PE layer of each party, there is almost no correlation between the identifiers and encoded representations across different parties. This lack of correlation causes integration issues and affects accuracy. However, directly sharing a PE layer among all parties is not viable due to privacy concerns. To address this, we propose positional encoding averaging.

Every T pe epoch, the positional encoding layers are averaged and broadcast to all parties under a secure multi-party computation (MPC) scheme, akin to FedAvg [23] in horizontal federated learning [27]. While the privacy of the transmitted model can be a concern, this issue is an orthogonal open problem in horizontal federated learning.

5.2 Training

The FeT training process begins with employing Privacy-Preserving Record Linkage (PPRL) [44] to evaluate identifier similarities between the primary party P and each secondary party. Secondary parties each contribute a random subset for linkage (line 5). For each P 's record, K nearest neighbors within these subsets from secondary parties are determined (line 6). The training leverages data embeddings of dimensions B × L × H , where B is batch size, L is the sequence length, and H is the hidden layer size, following the transformer architecture. In FeT's context, L = 1 for the primary and L = K for secondary parties, linking each primary record with K neighboring records from the secondaries. Identifiers are transformed into vectors using multi-dimensional positional

encoding [28] and combined with data representations for processing via self-attention blocks (lines 7, 10). Secondary parties' representations are averaged under the MPC protocol. The primary party then employs attention blocks for forward propagation to compute the final prediction (line 13). Backpropagation sends gradient updates from the primary to secondary parties to refine their local models (lines 14-16). The privacy mechanism including norm clipping (lines 8, 11) and distributed Gaussian noise (line 12) are further discussed in Section 6.

6 Privacy

In this section, we address the challenges of privacy in multi-party fuzzy VFL. First, the risk of transferring raw similarities has been mitigated by the design of the FeT itself. Second, to address the increasing costs when more parties join, we introduce a multi-party privacy-preserving VFL framework, SplitAvg , which is compatible with FeT. The architecture of SplitAvg is illustrated in Figure 5. SplitAvg integrates differential privacy (DP), secure multi-party computation (MPC) [6], and norm clipping to enhance the privacy of representations. Additionally, to further improve the utility of FeT under DP, we employ privacy amplification techniques that reduce the noise scales by incorporating random sampling.

6.1 Differentially Private Split Neural Network - SplitAvg

This section outlines three techniques applied to the SplitAvg to improve privacy: representation norm clipping, privacy amplification, and MPC with distributed Gaussian noise. These strategies collectively protect the privacy of each secondary party's representations through differential privacy and ensure that privacy risks do not escalate with an increase in the number of parties due to MPC.

Representation Norm Clipping. The magnitude of the ℓ 2 -norm is pivotal in determining the sensitivity of differential privacy. To limit the maximum change of the ℓ 2 -norm, norm clipping is essential. Specifically, for a representation R , we ensure that ∥ R ∥ 2 ≤ C , where C is a predefined positive

real number. This is achieved by scaling R by a factor of C , formally, ˆ R = R / max(1 , ∥ R ∥ 2 /C ) . Through this process, any representation R with ∥ R ∥ 2 exceeding C is scaled to C , while values of ∥ R ∥ 2 below C remain unaffected.

Secure Multi-Party Computation with Distributed Gaussian Noise. To address the challenges of applying differential privacy in multi-party VFL, we propose a method that integrates noise addition into the process of aggregating representations, facilitated through MPC. In this setup, each secondary party first independently conducts representation norm clipping to limit the scale of their data. Subsequently, these clipped representations, along with Gaussian noise N (0 , σ/k 2 ) independently generated by each of the k secondary parties, are aggregated through averaging under MPC. For the primary party, this aggregation is equivalent to adding independent Gaussian noise N (0 , σ 2 ) to the averaged result. The adoption of MPC in our framework ensures that the secondary parties do not need to individually add N (0 , σ 2 ) noise to their representations. Instead, as the primary party only has access to the averaged result under MPC, each secondary party can add a smaller amount of noise. This method effectively improves utility with a small efficiency cost due to MPC.

Privacy Amplification by Secondary Subsampling. This technique is specifically designed for FeT configurations. According to the principle of privacy amplification [5], applying a function to a randomly sampled subset of data enhances privacy compared to applying the same function to the entire dataset. By initiating linkage from a randomly sampled subset rather than the full dataset, the privacy parameters effectively shift from ( ε, δ ) to ( qε, qδ ) , where q < 1 represents the sampling rate. In FeT, the primary party typically selects subsets of candidate data records for training from each secondary party, targeting those with neighboring identifiers. By pre-sampling these subsets at a rate of q before conducting a k-nearest neighbors (kNN) search, we avoid processing the entire dataset, which in turn reduces the noise required to maintain the same privacy level.

6.2 Privacy Guarantee

Our analysis of differential privacy focuses on a hypothetical global dataset linked using all secondary parties, denoted as x S 1 , . . . , x S k . Since these datasets are correlated, removing one data record from this global dataset will result in changes to the representations in all secondary parties. Consequently, privacy loss accumulates across secondary parties without the use of MPC. However, with MPC, a single aggregated noise, formed by distributing smaller noise contributions among parties, can be added, effectively reducing the overall required noise. The privacy guarantee for these representations is formally articulated in Theorem 3, with the proof provided in Appendix A.

Theorem 3. For certain constants c 1 and c 2 , given a sampling rate q , the total number of epochs T , and the number of batches B in each epoch, each representation R S k achieves ( ε, δ ) -differential privacy for all ε < c 1 q 2 T , with any δ > 0 , by selecting the standard deviation σ of the Gaussian noise mechanism as follows:

FeT Modules

Decoder

SplitAvgPrivacyModules

Secure Multi-Party Averaging

N(0,o² /k)

N(0,0² /k)

Encoder

Encoder

Encoder

Linkage

Linkage

Sampling

Sampling

7 Experiments

This section presents the experimental setup and results. We begin by outlining the experimental settings in Section 7.1, followed by an assessment of performance across varying numbers of parties and neighbors in Section 7.2. We then analyze the privacy of FeT in Section 7.3. Additionally, an ablation study is conducted in Appendix C to evaluate the contribution of each component to performance, including dynamic masking, party dropout, positional encoding, key fuzziness, and

SplitAvg. The performance of FeT with exact key matching is assessed in Appendix D, while the computational and memory efficiency of MPC and training is evaluated in Appendix E. Privacy evaluation on two-party real datasets is included in Appendix F. Furthermore, FeT's performance under imbalanced feature splits across parties, based on VertiBench [49], is presented in Appendix G.

7.1 Experimental Settings

Datasets. Our experiments utilize five datasets, including three real-world datasets: house [35, 2], bike [8, 42], and hdb [18, 37], along with two high-dimensional datasets: gisette [16] and MNIST [24]. Detailed descriptions of these datasets can be found in Appendix B. To simulate multiparty fuzzy VFL, we partition the features equally and randomly among the parties. The primary party's feature dimensions are reduced to 4 using principal component analysis (PCA) to serve as universal keys. To create fuzzy linked scenarios, we add independent Gaussian noise with a scale of 0.05 to the keys of each party.

Baselines. We include three baselines in our experiments: (1) Solo: training only on the primary party; (2) Top1Sim: linking each data record in the primary party only with its most similar neighbor in the secondary parties; (3) FedSim [50]: training on the top K neighboring data records.

7.2 Performance

Two-party fuzzy VFL. In this experiment, we evaluate the performance of FeT in two-party settings without privacy mechanisms. The detailed results are presented in Table 1. Our experiments demonstrate that FeT consistently outperforms the leading two-party fuzzy VFL methods. Notably, this performance improvement is achieved while enhancing privacy protections, as FeT does not involve transferring similarity data.

Effect of Number of Neighbors K . In this experiment, we assess the impact of the number of neighbors, K , on FeT's performance by varying K from 1 to 100. The results are displayed in Figure 6. The FedSim baseline is trained using the optimal K value (i.e., 50 for hdb and 100 for house and bike ). The figure reveals two key insights: First, FeT's performance improves as K increases, demonstrating its ability to filter useful information even as the number of unrelated data records grows. Second, FeT consistently outperforms all baselines at larger values of K , highlighting its superiority in fuzzy VFL scenarios.

house

taxi

hdb

32

Top1Sim

Top1Sim

250

Top1Sim

RMSE

50

RMSE

RMSE

E

30

240

FedSim (BestK)

FedSim.(BestK)

28

FedSim (BestK)

40

0

50

100

0

50

100

50

100

NeighborsK

NeighborsK

Neighbors K

Effect of Number of Parties In this experiment, we assess FeT's performance in fuzzy VFL with various numbers of parties. Due to the absence of real multi-party VFL data, we employ synthetic data for our evaluations. We partition the features equally and randomly among the parties, reducing

the primary party's feature dimensions to 4 using PCA as the universal keys. To simulate fuzzy linked scenarios, we add independent Gaussian noise with a scale of 0.05 to the keys of each party.

Figure 7 shows that FeT generally outperforms the baselines, particularly with a larger number of parties. This advantage is attributed to Solo's lack of informative features and Top1Sim's noiseaffected linkage. FedSim performs poorly as the top-1 linked secondary parties are unaware of the primary parties' keys, leading to misalignment in subsequent soft linkage and training steps. On the gisette dataset with k = 10 , FeT and other models slightly underperform compared to Solo, likely due to overfitting given gisette 's small size.

10

20

30

40

50

Number of Parties

0.4

0.6

0.8

Test Accuracy

MNIST (noise 0.05)

Solo

FeT

FedSim

Top1

10

20

30

40

50

Number of Parties

0.88

0.90

0.92

0.94

0.96

Test Accuracy

Solo

FedSim

FeT

Top1

7.3 Privacy

In this subsection, we analyze how the performance of FeT varies with different noise scales ( σ ) and sampling rates on secondary parties, demonstrating the impact of privacy constraints on its accuracy. The results are depicted in Figure 8. We observe three key points: First, a moderate sampling rate has a negligible effect on model performance and may even slightly improve performance (e.g., on the MNIST dataset) by reducing overfitting. Second, despite increasing noise levels and enhanced privacy guarantees, FeT consistently outperforms baseline models. Third, the ε -σ privacy-noise curves illustrate that solely adding Gaussian noise without MPC, even with advanced analysis theorems such as Rényi Differential Privacy (RDP) [32], would require much larger noise scales compared to our approach that integrates MPC.

0

2

4

6

8

0.93

0.94

0.95

0.96

0.97

gisette

dpSample=0.2

dpSample=0.5

dpSample=1.0

1

2

3

4

5

6

7

8

10 0

10 1

gisette

FeT

RDP

0

2

4

6

8

0.88

0.89

0.90

0.91

0.92

0.93

MNIST

dpSample=0.2

dpSample=0.5

dpSample=1.0

1

2

3

4

5

6

7

8

10 1

MNIST

FeT

RDP

8 Conclusion

In this study, we introduce the Federated Transformer (FeT), specifically designed to support multiparty VFL while effectively addressing critical challenges related to performance, privacy, and communication. Furthermore, we provide theoretical proof of FeT's differential privacy, ensuring that data representations remain protected from secondary parties. Notably, our experiments demonstrate that FeT surpasses baseline models, even under stringent privacy guarantees and within the traditional two-party setting, establishing its efficacy and robustness in complex federated environments.

Broader Impact. The architecture of FeT, even without privacy mechanisms, has potential applications in multimodal learning. Multimodal tasks often require the alignment of data records across different modalities, which can be quite challenging. For instance, aligning 24Hz video streams with 48kHz audio tracks is complex, as each video frame may correspond to a range of audio samples. FeT has shown its capability to effectively learn from such fuzzily aligned data. Furthermore, the transformer model has proven effective across various data types, including images, text, and tabular data, highlighting FeT's suitability for multimodal learning applications.

Acknowledgments

This research/project is supported by the National Research Foundation Singapore and DSO National Laboratories under the AI Singapore Programme (AISG Award No: AISG2-RP-2020-018), the National Research Foundation Singapore and Infocomm Media Development Authority under its Trust Tech Funding Initiative. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation Singapore, DSO National Laboratories, and Infocomm Media Development Authority. This research is also sponsored by Webank Scholars Program. We thank the AMD Heterogeneous Accelerated Compute Clusters (HACC) program (formerly known as the XACC program - Xilinx Adaptive Compute Cluster program) for the generous hardware donation.

References

Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. 2016. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security . 308-318.

Airbnb. 2019. Airbnb prices dataset. http://insideairbnb.com/get-the-data.html .

Manfred Antoni and Rainer Schnell. 2019. The past, present and future of the German Record Linkage Center (GRLC). Jahrbücher für Nationalökonomie und Statistik 239, 2 (2019), 319331.

Borja Balle and Yu-Xiang Wang. 2018. Improving the gaussian mechanism for differential privacy: Analytical calibration and optimal denoising. In International Conference on Machine Learning . PMLR, 394-403.

Amos Beimel, Hai Brenner, Shiva Prasad Kasiviswanathan, and Kobbi Nissim. 2014. Bounds on the sample complexity for private learning and private data release. Machine learning 94 (2014), 401-437.

Keith Bonawitz, Vladimir Ivanov, Ben Kreuter, Antonio Marcedone, H Brendan McMahan, Sarvar Patel, Daniel Ramage, Aaron Segal, and Karn Seth. 2017. Practical secure aggregation for privacy-preserving machine learning. In proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security . 1175-1191.

Timothy J Castiglia, Anirban Das, Shiqiang Wang, and Stacy Patterson. 2022. CompressedVFL: Communication-Efficient Learning with Vertically Partitioned Data. In Proceedings of the 39th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 162) , Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (Eds.). PMLR, 2738-2766. https://proceedings.mlr.press/ v162/castiglia22a.html

CitiBike. 2016. Citi bike system data. https://www.citibikenyc.com/system-data .

Yuren Cong, Wentong Liao, Hanno Ackermann, Bodo Rosenhahn, and Michael Ying Yang. 2021. Spatial-temporal transformer for dynamic scene graph generation. In Proceedings of the IEEE/CVF international conference on computer vision . 16372-16382.

Changyu Dong, Liqun Chen, and Zikai Wen. 2013. When private set intersection meets big data: an efficient and scalable protocol. In Proceedings of the 2013 ACM SIGSAC conference on Computer & communications security . 789-800.

Johanna Eberle and Michael Weinhardt. 2016. Record linkage of the linked employer-employee survey of the socio-economic panel study (SOEP-LEE) and the establishment history panel (BHP). German Record Linkage Center Working Paper Series, No. WP-GRLC-2016-01 (2016).

Siwei Feng, Han Yu, and Yuebing Zhu. 2024. MMVFL: A Simple Vertical Federated Learning Framework for Multi-Class Multi-Participant Scenarios. Sensors 24, 2 (2024). https: //doi.org/10.3390/s24020619

Chong Fu, Xuhong Zhang, Shouling Ji, Jinyin Chen, Jingzheng Wu, Shanqing Guo, Jun Zhou, Alex X Liu, and Ting Wang. 2022. Label inference attacks against vertical federated learning. In 31st USENIX Security Symposium (USENIX Security 22) . USENIX Association, Boston, MA, 1397-1414. https://www.usenix.org/conference/usenixsecurity22/ presentation/fu-chong

Fangcheng Fu, Huanran Xue, Yong Cheng, Yangyu Tao, and Bin Cui. 2022. BlindFL: Vertical Federated Machine Learning without Peeking into Your Data. In Proceedings of the 2022 International Conference on Management of Data (Philadelphia, PA, USA) (SIGMOD '22) . Association for Computing Machinery, New York, NY, USA, 1316-1330. https://doi. org/10.1145/3514221.3526127

O. Goldreich, S. Micali, and A. Wigderson. 1987. How to play ANY mental game. In Proceedings of the Nineteenth Annual ACM Symposium on Theory of Computing (New York, New York, USA) (STOC '87) . Association for Computing Machinery, New York, NY, USA, 218-229. https://doi.org/10.1145/28395.28420

I. Guyon, S. Gunn, A. Ben-Hur, and G. Dror. 2004. Gisette Dataset. https://www.csie. ntu.edu.tw/~cjlin/libsvmtools/datasets/binary/gisette_scale.bz2

Stephen Hardy, Wilko Henecka, Hamish Ivey-Law, Richard Nock, Giorgio Patrini, Guillaume Smith, and Brian Thorne. 2017. Private federated learning on vertically partitioned data via entity resolution and additively homomorphic encryption. arXiv preprint arXiv:1711.10677 (2017).

Singapore HDB. 2018. Resale flat prices in Singapore. https://data.gov.sg/dataset/ resale-flat-prices .

Yuanqin He, Yan Kang, Xinyuan Zhao, Jiahuan Luo, Lixin Fan, Yuxing Han, and Qiang Yang. 2022. A hybrid self-supervised learning framework for vertical federated learning. arXiv preprint arXiv:2208.08934 (2022).

Chung-ju Huang, Leye Wang, and Xiao Han. 2023. Vertical Federated Knowledge Transfer via Representation Distillation for Healthcare Collaboration Networks. In Proceedings of the ACM Web Conference 2023 . ACM, 4188-4199.

Yimin Huang, Wanwan Wang, Xingying Zhao, Yukun Wang, Xinyu Feng, Hao He, and Ming Yao. 2023. EFMVFL: an efficient and flexible multi-party vertical federated learning without a third party. ACM Transactions on Knowledge Discovery from Data 18, 3 (2023), 1-20.

Yan Kang, Yang Liu, and Xinle Liang. 2022. FedCVT: Semi-supervised vertical federated learning with cross-view training. ACM Transactions on Intelligent Systems and Technology (TIST) 13, 4 (2022), 1-16.

Jakub Koneˇ cn` y, H Brendan McMahan, Felix X Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. 2016. Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv:1610.05492 (2016).

Yann LeCun and Corinna Cortes. 2005. The mnist database of handwritten digits. https: //api.semanticscholar.org/CorpusID:60282629

Jingtao Li, Adnan Siraj Rakin, Xing Chen, Zhezhi He, Deliang Fan, and Chaitali Chakrabarti. 2022. Ressfl: A resistance transfer framework for defending model inversion attack in split federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 10194-10202.

Jialin Li, Tongjiang Yan, and Pengcheng Ren. 2023. VFL-R: a novel framework for multi-party in vertical federated learning. Applied Intelligence 53, 10 (2023), 12399-12415.

Qinbin Li, Zeyi Wen, Zhaomin Wu, Sixu Hu, Naibo Wang, Yuan Li, Xu Liu, and Bingsheng He. 2021. A survey on federated learning systems: Vision, hype and reality for data privacy and protection. IEEE Transactions on Knowledge and Data Engineering (2021).

Yang Li, Si Si, Gang Li, Cho-Jui Hsieh, and Samy Bengio. 2021. Learnable fourier features for multi-dimensional spatial positional encoding. Advances in Neural Information Processing Systems 34 (2021), 15816-15829.

Yang Liu, Xinwei Zhang, Yan Kang, Liping Li, Tianjian Chen, Mingyi Hong, and Qiang Yang. 2022. FedBCD: A communication-efficient collaborative learning framework for distributed features. IEEE Transactions on Signal Processing 70 (2022), 4277-4290.

Yang Liu, Xiong Zhang, and Libin Wang. 2020. Asymmetrical vertical federated learning. arXiv (2020).

Xinjian Luo, Yuncheng Wu, Xiaokui Xiao, and Beng Chin Ooi. 2021. Feature inference attack on model predictions in vertical federated learning. In 2021 IEEE 37th International Conference on Data Engineering (ICDE) . IEEE, 181-192.

Ilya Mironov. 2017. Rényi differential privacy. In 2017 IEEE 30th computer security foundations symposium (CSF) . IEEE, 263-275.

Vaikkunth Mugunthan, Pawan Goyal, and Lalana Kagal. 2021. Multi-vfl: A vertical federated learning system for multiple data and label owners. arXiv preprint arXiv:2106.05468 (2021).

Richard Nock, Stephen Hardy, Wilko Henecka, Hamish Ivey-Law, Jakub Nabaglo, Giorgio Patrini, Guillaume Smith, and Brian Thorne. 2021. The Impact of Record Linkage on Learning from Feature Partitioned Data. In Proceedings of the 38th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 139) , Marina Meila and Tong Zhang (Eds.). PMLR, 8216-8226. https://proceedings.mlr.press/v139/nock21a.html

Qichen Qiu. 2017. Kaggle dataset: housing price in Beijing. https://www.kaggle.com/ ruiqurm/lianjia .

Xinchi Qiu, Heng Pan, Wanru Zhao, Chenyang Ma, Pedro PB Gusmao, and Nicholas D Lane. 2023. vFedSec: Efficient Secure Aggregation for Vertical Federated Learning via Secure Layer. arXiv preprint arXiv:2305.16794 (2023).

Salary.sg. 2020. Secondary school rankings in Singapore. https://www.salary.sg/2020/ secondary-schools-ranking-2020-psle-cut-off/ .

Haoran Shi, Yali Jiang, Han Yu, Yonghui Xu, and Lizhen Cui. 2022. MVFLS: multi-participant vertical federated learning based on secret sharing. The Federate Learning (2022), 1-9.

Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. 2017. Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP) . IEEE, 3-18.

Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research 15, 1 (2014), 1929-1958.

Jiankai Sun, Xin Yang, Yuanshun Yao, and Chong Wang. 2022. Label leakage and protection from forward embedding in vertical federated learning. arXiv preprint arXiv:2203.01451 (2022).

New York TLC. 2016. TLC trip record data. https://www1.nyc.gov/site/tlc/about/ tlc-trip-record-data.page .

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017).

Dinusha Vatsalan, Ziad Sehili, Peter Christen, and Erhard Rahm. 2017. Privacy-preserving record linkage for big data: Current approaches and research challenges. In Handbook of Big Data Technologies . Springer.

Praneeth Vepakomma, Otkrist Gupta, Tristan Swedish, and Ramesh Raskar. 2018. Split learning for health: Distributed deep learning without sharing raw patient data. arXiv preprint arXiv:1812.00564 (2018).

Praneeth Vepakomma, Abhishek Singh, Otkrist Gupta, and Ramesh Raskar. 2020. NoPeek: Information leakage reduction to share activations in distributed deep learning. ICDM Workshop (2020).

Chang Wang, Jian Liang, Mingkai Huang, Bing Bai, Kun Bai, and Hao Li. 2020. Hybrid differentially private federated learning on vertically partitioned data. arXiv preprint arXiv:2009.02763 (2020).

Webank. 2023. The application and development report of open-source privacy computing framework (FATE) in financial industry. https://www.163.com/dy/article/ HR7M3K7P055219FH.html

Zhaomin Wu, Junyi Hou, and Bingsheng He. 2024. VertiBench: Advancing Feature Distribution Diversity in Vertical Federated Learning Benchmarks. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net. https://openreview.net/forum?id=glwwbaeKm2

Zhaomin Wu, Qinbin Li, and Bingsheng He. 2022. A Coupled Design of Exploiting Record Similarity for Practical Vertical Federated Learning. Advances in Neural Information Processing Systems 35 (2022), 21087-21100.

Zhaomin Wu, Qinbin Li, and Bingsheng He. 2022. Practical vertical federated learning with unsupervised representation learning. IEEE Transactions on Big Data (2022).

Yang Yan, Guozheng Yang, Yan Gao, Cheng Zang, Jiajun Chen, and Qiang Wang. 2022. Multiparticipant vertical federated learning based time series prediction. In Proceedings of the 8th International Conference on Computing and Artificial Intelligence . 165-171.

Liu Yang, Di Chai, Junxue Zhang, Yilun Jin, Leye Wang, Hao Liu, Han Tian, Qian Xu, and Kai Chen. 2023. A Survey on Vertical Federated Learning: From a Layered Perspective. arXiv preprint arXiv:2304.01829 (2023).

Tianyuan Zou, Yang Liu, Yan Kang, Wenhan Liu, Yuanqin He, Zhihao Yi, Qiang Yang, and Ya-Qin Zhang. 2022. Defending batch-level label inference and replacement attacks in vertical federated learning. IEEE Transactions on Big Data (2022).

Appendix

Table of Contents

A Proof

Theorem 3. For certain constants c 1 and c 2 , given a sampling rate q , the total number of epochs T , and the number of batches B in each epoch, each representation R S k achieves ( ε, δ ) -differential privacy for all ε < c 1 q 2 T , with any δ > 0 , by selecting the standard deviation σ of the Gaussian noise mechanism as follows:

Proof. The proof leverages the moments accountant bound [1], which is applicable to a sequence of Gaussian mechanisms applied to subsampled data. We begin by establishing that the output's ℓ 2 -norm for each function is constrained by a constant C . This constraint ensures that each randomized function adheres to differential privacy under the Gaussian mechanism. By determining the cumulative count of Gaussian mechanisms applied, we can directly invoke Theorem 2 to reach our conclusion.

To clarify the process, we apply norm clipping to each party as specified in Section 6, scaling each party at a rate of C/k . This scaling guarantees that, for every ˆ R h i , the condition ∥ ˆ R h i ∥ 2 ≤ C/k is satisfied. Using the triangle inequality within normed vector spaces, we derive:

Since the ℓ 2 -norm of H i is bounded by C , adding Gaussian noise to H i satisfies the conditions for differential privacy. Throughout the training, B · T independent noises are introduced, resulting in a sequence of Gaussian mechanisms targeting a randomly selected subset at a ratio q . Consequently, Equation 1 is derived by directly applying Theorem 2.

B Experimental Details

Datasets. In this section, we include the detailed information of each dataset used in the experiment. These real-world datasets are obtained in the same manner as those utilized in FedSim, with each dataset comprising two parties. Details of the real datasets are presented in Table 2. The synthetic dataset, gisette [16], consists of 6,000 instances and 5,000 features and serves as a binary classification task with balanced labels. The MNIST dataset [24] consists of 60,000 instances and 28x28 features for a 10-class digit classification task.

Metrics. For regression tasks, Root Mean Square Error (RMSE) is utilized, while accuracy is applied to classification tasks. Early stopping is performed based on the validation set, with the corresponding test scores reported.

Hyperparameters. Each algorithm was run until convergence, with a maximum of 50 to 100 epochs. The learning rate and weight decay were consistently set at 10 -3 and 10 -5 , respectively. For the Solo model, a multi-layer perceptron (MLP) with two hidden layers of 400 units each was employed. In contrast, the Top1Sim utilized a single-layer MLP with a hidden size of 200 for both local and aggregation models. For FedSim, the number of K neighbors was selected from the set { 50 , 100 } . For FeT, the number of blocks is set to 6 for both local model and aggregation model.

Environments. We evaluate FeT on a server equipped with dual Intel Xeon Gold 6346 CPUs, eight A100 GPUs with CUDA version 12.2, and 1008GB RAM, running Python 3.10.13 with PyTorch 2.1.1+cu121 on Linux kernel 6.5.0. Efficiency experiments were conducted on a machine powered by a 64-core Intel(R) Xeon(R) Gold 6226R CPU @ 2.90GHz and 376 GB of RAM. Each experiment was run five times, and we report the average and standard deviation.

C Ablation Study

In this section, we evaluate the performance improvement of each proposed component of the FeT. Our findings indicate that dynamic masking is crucial for enhancing performance, while both party dropout and positional encoding averaging contribute modestly to these improvements . Detailed analyses are provided below.

C.1 Dynamic Masking

We evaluate the performance of FeT with and without dynamic masking, as shown in Table 3. The evaluation includes two-party datasets ( house , bike , hdb ) and two 50-party synthetic datasets ( MNIST and gisette ). The results indicate that dynamic masking leads to an improvement of up to 13 percentage points, particularly noticeable in datasets with a large number of parties. This suggests that dynamic masking significantly enhances model performance, especially in multi-party settings.

C.2 Party Dropout

Next, we evaluate the effect of the dropout rate under specific hyperparameter settings: the number of parties k = 50 , the number of neighbors K = 100 , and key noise set to 0 . 05 . The impact of the party dropout rate is demonstrated in Figure 9 and Table 4. Our observations reveal that a moderate party dropout rate of 0.6 enhances FeT's generalization by reducing the model size. Notably, FeT maintains stable performance even as the dropout rate increases to 0.8. This indicates that party dropout not

only improves generalization but also significantly reduces communication overhead across parties. Based on these findings, we set the dropout rate to 0.6 by default in multi-party experiments.

mnist

gisette

0.85

0.93

0.80

0.92

ccuracy

0.75

0.91

0.70

0.90

0.65

0.89-

0.60

0

0.2

0.4

0.6

0.8

1.0

0

0.2

0.4

0.6

0.8

1.0

Party Dropout Rate

Party Dropout Rate

C.3 Positional Encoding

We now assess the effect of the frequency of positional encoding (PE) averaging, as depicted in Figure 10 and Table 5. We find that PE averaging yields improvements, particularly with a large number of parties, such as 50 on MNIST , where alignment of encodings becomes crucial. Based on our observations, we set the frequency to 1 in most experiments.

Additionally, we assess the impact of positional encoding on the performance of FeT, as detailed in Table 3. These evaluation of MNIST and gisette are conducted with the number of neighbors K = 100 and key noise 0 . 05 . The results indicate that positional encoding is important for enhancing the performance of FeT.

C.4 Fuzziness of Keys

We evaluate the impact of identifier fuzziness on FeT's performance by introducing Gaussian noise of varying scales to the keys. The results are presented in Figure 11. From the figure, we derive two key observations: (1) Both FeT and baseline models show improved performance in more balanced scenarios. (2) FeT consistently outperforms the baselines across different levels of heterogeneity, demonstrating its robustness to varying degrees of noise. These findings highlight the resilience of FeT in the presence of noise, which is critical for practical applications.

C.5 SplitAvg vs. SplitNN

To evaluate the comparative performance of the SplitAvg (without noise) and SplitNN, we conducted training for both models using identical hyperparameters on the same VFL dataset, gisette . The

mnist

gisette

0.950

0.96

0.925

ccuracy

0.900

0.94

B

0.875

Party 2

Party 2

Party 5

Party5

0.850

Party20

0.92

Party20

Party50

Party 50

0.825

0

2

4

6

8

10

0

2

4

6

8

10

AveragePEFrequency

AveragePEFrequency

outcomes are illustrated in Figure 12. Analysis of the figure yields two primary observations. Firstly, SplitNN and SplitAvg exhibit very similar loss and accuracy curves during training, indicating that both models behave very similarly. Secondly, upon expanding the number of participating parties to 128, we observe that the performance curves of both models remain closely aligned, albeit with the split-sum model exhibiting a marginally lower accuracy. This minor discrepancy is attributed to the increased model parameters in SplitNN, which can typically be compensated for by increasing the number of parameters in SplitAvg.

D Exact Linkage

While FeT primarily focuses on scenarios involving fuzzy linkage, we also evaluate its robustness in exact linkage contexts. To achieve this, we synthesize exact linkage data by generating pure random keys within the range of [ -1 , 1] without introducing any noise. Each party is randomly divided into five or ten groups, with each group containing an equal number of features. Importantly, each party retains the exact keys, ensuring a controlled environment for our evaluation.

The results of our experiments are summarized in Table 6. From the table, we observe that Top1Sim achieves the highest accuracy, as it is inherently well-suited for exact linkage scenarios. In contrast,

gisette(50parties)

mnist(50parties)

0.90

0.950

FeT

FeT

Solo

0.85

Solo

2

0.925

Top1Sim

Top1Sim

0.80

0.900

cura

D

C

A

0.875

A

0.75

0.70

0.850

0.65

0.02

0.04

0.06

0.08

0.10

0.02

0.04

0.06

0.08

0.10

NoiseScaleonldentifiers

NoiseScaleonIdentifiers

0.95

0.90

sum

0.3

LoSs

0.4

0.2

concat

sum

ccuracy

0.96

0.92

0.94

0.90

sum

SS07

0.3

0.2

concat

sum

0.85

0

10

20

30

40

concat

50

0.1

0

10

20

30

40

50

0.88

0

10

20

30

40

concat

50

0.1

0

10

20

OE

40

50

Epoch

Epoch

Epoch

Epoch

the accuracy of FeT shows a slight decrease, which may be attributed to overfitting; however, its performance remains competitive and does not suffer significantly in this context.

E Efficiency

Parameter Efficiency. In our analysis, we assess the computational efficiency of standard addition compared to multi-party computation (MPC) addition, as shown in Table 7. Under the arithmetic GMWprotocol [15], and given that the size of the aggregated vector varies by dataset, we use a typical size for our experiments. Specifically, we conduct MPC addition to aggregate 10,000-dimensional vectors from multiple parties. Each experiment is performed five times, with the average timing reported. Although MPC generally incurs higher computational requirements, the results in Table 7 indicate that aggregating high-dimensional vectors via MPC incurs only a one-second overhead, even as the number of parties increases to 100. This minimal time cost is relatively small, especially when compared to other factors such as communication costs. Therefore, our findings suggest that MPC remains a feasible and efficient approach for representation aggregation in the context of VFL.

Training Computational and Memory Efficiency. We evaluate the computational and memory efficiency of FeT during training on an RTX3090 GPU with a batch size of 128. The results, shown in Table 8, lead to three key observations: (1) FeT has a comparable number of parameters to FedSim; (2) FeT demonstrates improved memory efficiency compared to FedSim, although this improvement comes with a trade-off in training speed; and (3) the additional components, such as dynamic masking (DM) and positional encoding (PE), introduce only a minor overhead in terms of both parameters and computational cost.

F Privacy on Two-Party Real Datasets

In this section, we explore how the performance of FeT varies with different noise scales σ and secondary sampling rate, illustrating the influence of privacy constraints on its accuracy. The outcomes are depicted in Figure 13. From this figure, we observe two key points. First, for large secondary datasets like bike , a moderate sampling rate has a negligible effect on model performance. Conversely, for smaller secondary datasets like hdb , performance is quite sensitive to sampling rates. Second, as the noise scale increases for secondary parties, the performance of FeT does not degrade sharply; instead, it gradually converges to a state where only primary features are utilized due to our dynamic masking design. In this scenario, FeT also outperforms MLP-based Solo primarily due to the transformer's key encoding, which has proven to be more effective than incorporating all keys into the training process, as evidenced in spatial-temporal prediction tasks [9].

0

2

4

6

8

45

50

55

60

65

70

75

RMSE

Solo

house

0

2

4

6

8

232.5

235.0

237.5

240.0

242.5

245.0

247.5

250.0

RMSE

Solo

taxi

0

2

4

6

8

27

28

29

30

31

32

33

34

RMSE

Solo

hdb

dpSample=0.1

dpSample=0.5

dpSample=1.0

Solo

Next, we explore the relationship between σ and ε as outlined in Theorem 3, setting hyperparameters to reflect typical training conditions. The number of epochs is chosen based on common convergence epochs: 10 for bike , and 50 for house and hdb . We adopt a batch size of 8k and set δ to 1 /N , with N representing the size of party S 1 . This correlation between ϵ and σ is depicted in Figure 14. The figure illustrates that reasonable noise levels can yield robust privacy guarantees. For instance, within a noise scale conducive to maintaining competitive performance, FeT achieves ε = 3 for hdb and ε = 5 for house , indicating effective privacy preservation under practical noise conditions.

G Performance on Imbalanced Split

The preceding experiments were conducted using a balanced feature split for VFL. Building on this foundation, we extended our evaluation of FeT to include datasets with varying levels of imbalance, motivated by the recent benchmarks presented in VertiBench [49]. The MNIST datasets are divided by features according to the methodology described in VertiBench [49], utilizing imbalance parameters α ∈ { 0 . 1 , 0 . 5 , 1 . 0 , 5 . 0 , 10 . 0 , 50 . 0 } , where a higher α value denotes greater balance across parties. The findings, illustrated in Figure 15, lead to two key observations: firstly, both FeT and the baseline algorithms exhibit improved performance in more balanced scenarios. Secondly, despite the varying

1

2

3

4

5

6

7

8

10 1

10 2

house

FeT

RDP

1

2

3

4

5

6

7

8

10 1

10 2

taxi

FeT

RDP

1

2

3

4

5

6

7

8

10 1

hdb

FeT

RDP

levels of data imbalance, FeT consistently shows competitive or superior performance relative to the baselines.

mnist (10 Parties)

mnist (20 Parties)

90

90

(%)

(%)

80

85

FeT

FeT

Solo

80

Solo

70

Top1Sim

Top1Sim

10-1

100

101

10-1

100

101

Balanceα

Balanceα

H Limitations

The design of FeT includes three primary limitations that warrant careful consideration. First, FeT operates under the assumption that common features exist across all parties. While this assumption is valid in many scenarios, it may not hold in more complex situations where the parties lack a shared set of features. This limitation necessitates further investigation into alternative frameworks or adaptations that can accommodate such cases, particularly in heterogeneous environments.

Second, although FeT facilitates the application of scalable differential privacy across multiple parties, the stringent privacy safeguards can lead to significant accuracy reductions when operating with low values of ε . This trade-off between privacy and utility is particularly concerning in performancesensitive applications, where quantifying the extent of accuracy loss is essential for informing users about the potential impacts on their results. Future work should explore methods to balance privacy and accuracy more effectively.

Third, similar to other fuzzy VFL methods [50], FeT assumes a correlation between identifiers and data representations. This assumption may not hold in cases where identifiers are randomly generated, which could lead to overfitting and minor performance deficits compared to Top1 approaches. Experiments on such datasets (Appendix D) indicate that while FeT performs well in many scenarios, its effectiveness may vary significantly depending on the nature of the data and the key generation process. Therefore, further empirical studies are needed to assess FeT's robustness across diverse datasets and identifier generation strategies.

I License

The licenses of the datasets used in this work are presented in Table 9. We utilize the code from FedSim [50] as our baseline, which is licensed under the Apache V2 license 2 . Our own code will also be open-sourced under the Apache V2 license.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We have stated in abstract and introduction that the paper proposes a novel FeT framework to address the scalability and privacy issues in Vertical Federated Learning (VFL).

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: See Section 8.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: See Section 6 and Appendix A.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: See Section 7.1 and Appendix B.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The codes are available at a GitHub repository https://github.com/ Xtra-Computing/FeT .

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: See Section 7.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We repeat the experiments with five different random seeds.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: See the paragraph on environments in Section 7.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We follow the code of ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: See Section 8.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risk.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have cited the used datasets in Section 7.1 and list the licenses in Appendix I.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces the Federated Transformer (FeT), a novel framework for Vertical Federated Learning (VFL) that addresses the challenges of fuzzy identifiers in multi-party scenarios. It incorporates three key innovations to enhance performance, privacy, and reduce communication overhead: positional encoding averaging, dynamic masking, and a party dropout strategy. The approach significantly outperforms baseline models in scalability and privacy preservation.\n\nSTRENGTHS:\n1. The integration of a transformer architecture for managing fuzzy identifiers in multi-party VFL is innovative. The dynamic masking and party dropout strategies are creative solutions that enhance scalability and reduce communication costs.\n2. The proposed model is well-articulated, with rigorous experimental validation that demonstrates substantial improvements over existing methods. The integration of differential privacy and secure multi-party computation strengthens the privacy aspect.\n3.The paper is well-organized with clear explanations of the problems, proposed solutions, and results. It is accessible to readers with a background in federated learning.\n4.The results show significant improvements in both performance and privacy, making it a valuable contribution to the field of federated learning, especially in applications involving sensitive data across multiple parties.\n\nWEAKNESSES:\n1.The performance improvements reported are impressive; however, how dependent are these improvements on the initial conditions of data alignment and the distribution of fuzzy identifiers? Could the authors provide insights on the robustness of the Federated Transformer (FeT) under less ideal conditions?\n2. While the paper discusses scalability extensively, there is less focus on the computational resources required. Could the authors comment on the computational overhead and the practicality of deploying FeT in real-world scenarios with potentially limited resources?\n3.The FeT introduces several complex mechanisms such as dynamic masking and positional encoding averaging. How do these additions affect the training time and complexity of the model? Is there a significant trade-off between performance and efficiency?\n4.The experiments are conducted primarily on synthetic datasets. How well does FeT generalize to other real-world datasets, particularly those with higher levels of noise and less structured data?\n\nQUESTIONS:\nPlease see Weakness and Limitations.\n\nLIMITATIONS:\n1.I think you can give us a better case diagram to help us better understand your new task.\n2.I think there is an update error in line 16 of your algorithm.\n3.In the Party Dropout section of the article, around line 188, you said that the communication overhead on the primary party can be reduced by up to 80%, but I did not find the corresponding experiment or proof in the article. I think this part needs to be better explained.\n4.The experimental settings focus on synthetic and controlled environments. Real-world applications might introduce variables not accounted for in this study, potentially affecting the generalizability of the results.\n5.The complexity of the model and the required computational resources are not thoroughly discussed, which could be a limitation for practical deployment in resource-constrained environments.\n6.While the model addresses fuzzy identifiers, the performance heavily relies on the quality of the linkage, which may not always be feasible or accurate in real-world scenarios.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates vertical federated learning (VFL) linked with fuzzy identifiers and develops a new framework called Federated Transformer (FeT). The proposed framework leverages the transformer architecture to encode the identification information and distributes the subnets of the transformer across parties.\n\nSTRENGTHS:\n1.\tThe paper tackles the important topic of achieving multi-party fuzzy VFL with both promising performance and robust privacy.\n2.\tThe design of using a transformer to encode fuzzy identifiers is quite novel.\n3.\tThe code is publicly available.\n\nWEAKNESSES:\n1.\tThe paper is not well motivated. It would be great if the paper could provide concrete use cases of fuzzy VFL, that involves fuzzy data/linkages while requiring the privacy needs of distributed machine learning. The German Record Linkage Center might not be an appropriate example for federated learning.\n2.\tSeveral concepts in fuzzy VFL need further clarification. For example, what strategies are used in existing fuzzy linkage? How are the fuzzy identifiers presented? How is privacy preserved in fuzzy identifiers?\n3.\tThe design rationale behind dynamic masking is unclear. Why is dynamic masking necessary? How does it function? \n4.\tThe novelty of party dropout needs further elaboration. Dropout is widely used in federated learning. How does the proposed approach differ from existing designs? \n5.\tIt is unclear why the proposed framework employs two different privacy mechanisms: differential privacy and MPC. What unique challenge is each privacy mechanism designed to address?\n\nQUESTIONS:\n1.\tPlease provide more concrete use cases of fuzzy VFL.\n2.\tPlease clarify the key concept in fuzzy VFL.\n3.\tThe design rationale behind the key components in the proposed framework is unclear. Please see my comments above and articulate them in more detail.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the Federated Transformer (FeT) framework, which is designed to support multi-party VFL.\nIt enhances the efficiency of model training among multiple parties using fuzzy identifiers while ensuring data privacy.\nExperiment results show that FeT performed well when it scaled to 50 parties and the authors also provide theoretical proof of the differential privacy.\n\nSTRENGTHS:\n**Strengths of the Paper**\n\n**Originality**\n\nThe biggest novelty of the work is the introduction of the Federated Transformer (FeT) framework. It presents a new aspect of handling vertical federated learning (VFL) with fuzzy identifiers.\n\n- **FeT Structure**: The work introduces the architecture of FeT in detail. The model can be split into two parties: the primary party and the secondary party. It also includes several innovative techniques including Dynamic Masking, Party Dropout, and Positional Encoding Averaging.\n\n**Experiment**\n\nThe authors compare the performance of the Federated Transformer (FeT) against multiple baseline methods which is a robust benchmarking. The experimental results demonstrate that FeT significantly outperforms baseline models, achieving improvements of up to 46 percentage points when scaled to 50 parties, which is a significant performance gain.\n\n**Significance and Future Impact**\nFeT is suitable for multimodal learning which aligns many federated learning scenarios in real life while protecting individual privacy.\n\nWEAKNESSES:\n**Weaknesses of the Paper**\n\n**Model Design**\n\n1) The introduction of a trainable dynamic masking module aims to improve the exclusion of incorrectly linked data records. However, if this module is not well-optimized, it could introduce additional noise or errors which degrades the model performance.\n\n2) The proposal of positional encoding averaging may not available in all VFL scenarios, which may cause inconsistencies in data processing which may affect the model training.\n\n**Privacy Trade-offs**\n\nThe paper says that stringent privacy safeguards may cause accuracy reductions, particularly with low values of $\\epsilon$. It would be better if the authors could provide a more detailed analysis of the trade-offs between privacy and performance.\n\n**Theoretical Proof**\n\nWhile the paper provides some theoretical proofs, it may not offer a complete set of proofs for all claims made. The theoretical section did not adequately discuss the limitations of the proposed methods or the implications of violating the underlying assumptions. A more thorough exploration of these aspects would enhance the robustness of the theoretical framework.\n\n**Experiment**\n\nThe experiments are conducted on a small number of datasets, which may not represent the full range of scenarios in VFL. Furthermore, The experiments explore little the trade-off between privacy guarantees and model utility, especially at low values of ε in differential privacy.\n\nQUESTIONS:\n**Questions**\n\n1\\. **Scenario Application**:\n\nHow can FeT be adapted to work with unstructured data types such as text or images? Is there any preliminary results or insights on applying FeT to these types of data?\n\n2\\. **Privacy-Performance Trade-off**:\nCan you provide a more detailed analysis of the trade-off between privacy and performance, especially at different levels of $\\epsilon$?\n\n3\\. **Model Performance on datasets with different features**:\nHow does the performance of the Federated Transformer (FeT) vary when applied to datasets with significantly different feature distributions across multiple parties?\n\nLIMITATIONS:\nThe authors have addressed the limitations of their work by identifying three primary limitations related to the assumptions of common features across parties, the potential accuracy reductions due to stringent privacy safeguards, and the correlation between keys and data representations.\n\nHere are some other potential limitations that may be considered:\n\n**Large scale cases**: The model's performance may have the risk of degrading with a large number of parties or larger datasets, which may limit its application in certain large federated learning scenarios.\n\n**Privacy Trade-offs**: While the model is designed to enhance privacy, there may still be some vulnerabilities to be exploited, especially when the underlying assumptions about data sharing are violated.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes the Federated Transformer (FeT) framework, which aims to address performance and privacy challenges in multi-party fuzzy vertical federated learning (VFL). FeT leverages the Transformer architecture to encode fuzzy identifiers and distribute training across different parties. The authors introduce three innovative techniques—position encoding averaging, dynamic mask module, and party dropout strategy—to enhance model performance while minimizing computational and communication overhead. Additionally, FeT incorporates a scalable privacy framework that combines differential privacy and secure multi-party computation, effectively safeguarding local data representations and ensuring manageable privacy maintenance costs. Experimental results demonstrate that FeT outperforms baseline models, offering superior performance and enhanced privacy protection. Overall, FeT overcomes the limitations of existing models in multi-party fuzzy VFL, showcasing exceptional performance and practicality.\n\nSTRENGTHS:\n1. This manuscript proposes the FeT framework. The framework combines federated learning with the Transformer architecture. At the same time, the dynamic mask technique, participant discarding mechanism, and location coding average techniques are applied to improve the model's performance and privacy protection in multi-party fuzzy vertical federation learning.\n\n2. The FeT framework introduces SplitAvg, a hybrid privacy protection mechanism, which effectively reduces the introduction of noise and improves data utility. In addition, the privacy amplification technology enhances the privacy protection effect. This ensures data privacy as well as the efficiency and robustness of the model.\n\nWEAKNESSES:\nThis manuscript proposes a scalable vertical federated learning for practical fuzzy linked data, aiming to solve the performance and privacy issues of real-world VFL in its application. However, there are still some major issues, as follows:\n\n1. In the Introduction section, the definition of fuzzy identifiers and multi-party fuzzy VFL and the relationship between the identifiers, fuzzy identifiers and multi-party fuzzy VFL is not clearly introduced, which may lead to confusing concepts for readers. In addition, the important role of fuzzy identifiers in linking real-world data sets should be emphasized to prove the practical value of studying multi-party fuzzy VFL.\n\n2. In the training of FeT, PPRL was first used to evaluate the identifier similarity between the primary participant P and each secondary participant. However, the author did not mention this method in the previous content. Based on this, a very important question is: in the comparative experiment, the author did not seem to compare the performance gap, privacy gap, and computing resource consumption gap between the training using only the PPRL method and the training based on the transformer structure in this article?\n\n3. In FeT, the combination of dynamic mask, participant discarding mechanism, and hybrid privacy protection mechanism will inevitably affect the training process's time, computing resource consumption, and privacy protection ability. Therefore, it is also important to compare the training efficiency and computing resource consumption, as well as the privacy protection ability of FeT methods with current mainstream methods. This represents whether FeT can surpass other methods in real-world applications. The setting of the experiment may be as follows: 1. Training time of FeT and mainstream methods on the same data set and under the same conditions. 2. CPU/GPU usage between FeT and mainstream methods in the training process to measure memory consumption and network communication overhead, especially in the case of multi-party participation. 3. Evaluate the privacy protection effect of FeT and mainstream methods through quantified privacy protection indicators (such as ε value of differential privacy).\n\nMinor weaknesses:\n1. In lines 27 and 28, the authors describe collaboration between hospitals, financial institutions, and sensors. It is equivalent to juxtaposing the three, which is unreasonable.\n2. In lines 58 and 59, the authors mention that experiments have shown that model accuracy has been improved by up to 13 percentage points on a 50-square fuzzy VFL on the MNIST dataset. It is not given here which method is compared with, please add.\n3. In line 139, the author defines a common feature shared by all parties as an identifier, expressed as \\(x^i = [k^i, d^i] \\), where \\([\\cdot]\\) signifies. \\([,] \\) and\\([\\cdot]\\) is not the same\n\nQUESTIONS:\nsame as the weakness\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-16"}