Federated Transformer: Multi-Party Vertical Federated Learning on Practical Fuzzily Linked Data
Authors: Zhaomin Wu
Junyi Hou
Yiqun Diao
Bingsheng He
Venue: NeurIPS 2024 poster (2024)
Keywords: vertical federated learning
federated learning
transformer
record linkage
entity alignment
differential privacy
fuzzy alignment
Primary area: deep_learning_architectures
TL;DR: We introduce the Federated Transformer (FeT), a novel framework that supports multi-party Vertical Federated Learning (VFL) with fuzzy identifiers, which surpasses existing models in performance and privacy.

Abstract: Federated Learning (FL) is an evolving paradigm that enables multiple parties to collaboratively train models without sharing raw data. Among its variants, Vertical Federated Learning (VFL) is particularly relevant in real-world, cross-organizational collaborations, where distinct features of a shared instance group are contributed by different parties. In these scenarios, parties are often linked using fuzzy identifiers, leading to a common practice termed as _multi-party fuzzy VFL_. Existing models generally address either multi-party VFL or fuzzy VFL between two parties. Extending these models to practical multi-party fuzzy VFL typically results in significant performance degradation and increased costs for maintaining privacy. To overcome these limitations, we introduce the _Federated Transformer (FeT)_, a novel framework that supports multi-party VFL with fuzzy identifiers. FeT innovatively encodes these identifiers into data representations and employs a transformer architecture distributed across different parties, incorporating three new techniques to enhance performance. Furthermore, we have developed a multi-party privacy framework for VFL that integrates differential privacy with secure multi-party computation, effectively protecting local representations while minimizing associated utility costs. Our experiments demonstrate that the FeT surpasses the baseline models by up to 46\% in terms of accuracy when scaled to 50 parties. Additionally, in two-party fuzzy VFL settings, FeT also shows improved performance and privacy over cutting-edge VFL models.

Federated Transformer: Multi-Party Vertical Federated Learning on Practical Fuzzily Linked Data
Zhaomin Wu, Junyi Hou, Yiqun Diao, Bingsheng He
National University of Singapore, Singapore
zhaomin@nus edu. sg, (junyi.A. h,, yiqun, hebs) nus edu..gedusgg
Abstract
Federated Learning (FL.) is evolving paradigm that enables multiple parties to collaboratively train models without sharing raw data. Among its variants, Vertical Federated Learning (VFL) particularly relevant real-world, cross- organizational collaborations, where distinct features of shared instance group are often linked using contributed by different parties. these scenarios, parties multi-parry party fuzzy VFL. fuzzy identifiers, leading to common practice termed as multi Existing models generally address either aulli-partyy VFL or fuzzy VFL between two parties. Extending these models to practical multi- party fuzzy VFL typically results in significant performance degradation and increased costs for maintaining privacy. To overcome these limitations, we introduce the Federated Transformer (FeT), novel framework that supports multi-party VFL with fuzzy identifiers. FeT innovatively encodes these identifiers into data representations and employs a transformer architecture distributed across different parties, incorporating three new techniques to enhance performance. Furthermore, we have developed multi-party privacy framework for VFL that integrates differential privacy with secure multi- party computation, effectively protecting local representations while minimizing associated utility costs. Our experiments demonstrate that the FeT surpasses the baseline models by up 46% in terms of accuracy when scaled to 50 parties. Additionally, two-party fuzzy VFL. settings, FeT also shows improved performance and privacy cutting-edge VFL models.
Introduction
Federated Learning (FI.) is learning paradigm that enables multiple parties collaboratively train a model while preserving the privacy of their local data [27]- Among its various forms, Vertical Federated Learning (VFL.) 53 particularly prevalent form in real-world applications as highlighted
in recent technical report [48) In VFL. participants possess different features the same set of instances, where common features, such as names or addresses, serve as identifiers (a.k.a. keys) to link datasets across these parties.
Real-world applications often necessitate multi- cralfi-partyffeee fuzzy VFL, characterized by two key attributes. First, it supports collaboration among multiple parties, commonly observed in collaborations across hospitals [33], sensors
[52]. and financial institutions [38. Second, it accommodates scenarios where these parties are linked using fuzzy identifiers, such as addresses. Such scenarios are prevalent in applications, illustrated in analysis 500) of the German Record Linkage Center [TI]. For instance, multiple vehicle rental companies that are fuzzily linked by source and destination addresses in the same city collaborate predict travel times.
To illustrate the significance multi-party fuzzy VFL, consider the application of travel cost prediction in city through collaboration among taxi, car, bike, and bus companies, shown in Figure Since personal travel information is private and cannot be shared, VFL is essential.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

B
Bike
Car
Bus
Taxi
A 08
Route identifiers
(start lon., start lat., end lon. end lat.)
Task Travel Cost Prediction:
What is the cost traveling from A to B?
Unique Features
Rental Availability
Gas Cost
#Passengers
Price
Figure 1: Real application of multi parttipprty fuzzy VFL: travel cost prediction in a city
Additionally, route identifiers starting and ending GPS locations can only be linked using fuzzy methods. However, linking closely related source and destination points with multi-party fuzzy VFL significantly enhance prediction accuracy.
Existing VFL approaches generally address either the multi-party aspect or the fuzzy identifier 38
26 facilitate multi-party VFL using Private Set Intersection (PSI) [10] to link datasets. These methods often presume the existence of precise, universal keys, which are not feasible in common VFL scenarios involving fuzzy identifiers. Conversely, other propose two-party fuzzy VFL models that utilize cross-party key similarities for training. However, when extended multi-party fuzzy VFL, these similarity- -based approaches encounter significant challenges in performance and privacy. While some methods achieve reasonable performance, they often compromise privacy incur prohibitive costs.
issue. Several methods [33, 52
21
studies 17 34 50
Despite the potential of multi-party fuzzy VFL. several significant challenges must be addressed for effective implementation. First, as the number of parties with fuzzy identifiers increases, maintaining performance becomes increasingly challenging. The addition of parties leads to a quadratic growth in the number of key pairs, increase in incorrect linkages between fuzzy identifiers, and larger model sizes. These factors collectively heighten the risk overfitting and adversely impact model performance. Second, the rising costs of preserving privacy intensify as more parties with correlated data participate, leading to either significant computational costs (33 52 38] or accuracy loss [47]. Third, in collaboration of multiple parties, communication bottleneck arises for the party with labels, termed the primary party. This party must communicate with all other parties without labels, termed secondary parties, in each training round, placing substantial demands on the primary party's bandwidth. These challenges significantly hinder the practical deployment of VFL..
To address these issues, we introduce the Federated Transformer (Fel') to enhance performance and reduce privacy costs in multi-pan multi-party fuzzy VFL. First, tackle performance issues, we encode key similarities into data representations aligned by positional encoding averaging, which eliminates the need for quadratic calculations of key pairs. Additionally, we have designed a trainable dynamic masking module that automatically filters out incorrectly linked pairs, enhancing model accuracy by up to 13% in 50-party fuzzy VFL on the MNIST dataset. Second, to mitigate the escalating costs of privacy protection, we introduce SplitAvg. hybrid approach that merges encryption based and noise-based methods, maintaining a consistent noise level regardless of the number of participating implement party
parties. Third, to alleviate communication overhead on the primary party, dropout strategy, which randomly excludes certain secondary parties during each training round. This effectively reduces communication costs by approximately 80% and improves model generalization. Our codes are available on GitHut!"] summary, our contributions are as follows:
We design Federated Transformer (FeT), novel model achieving promising performance under multi-party fuzzy VFL.
We introduce SplitAvg to enhance the privacy of FeT by protecting local representations in multi-part fuzzy VFL, with theoretical proof of its differential privacy.
Experimental results demonstrate that FeT significantly outperforms baseline models by up to 46% in terms of accuracy in 50-parties VFL. Moreover, while providing enhanced privacy, FeT consistently surpasses state-of-the-an models even in traditional two-party fuzzy VFL scenarios.
https://github.comm com/xtrrccoo Caw.rraaa-ommuuntgeeeeeeee

---

2 Preliminaries
In this section, we provide the foundational concepts necessary for understanding our approach to differential privacy. Differential Privacy (DP) offers a rigorous mathematical framework for preserving individual privacy. quantifies privacy in terms of the probability of producing the same output from two similar datasets that differ by exactly one record.
Definition 1. Consider a randomized function M : R + O and two neighboring databases Du, D2 Be that differ by exactly record. For every possible oupput set O C O, M satisfies (e, 6) differrenta privacy if
Pr(M(DD) E Pr(M((i) E0
where e 2 0 and 8 2 O.
single query that adheres to differential privacy is termed a mechanism. For example, the Gaussian mechanism [4] is commonly used to achieve DP by adding Gaussian noise to the output of the function.
Theorem 1 (Gaussian Mechanism (4). For a function f X - Rd characterized by a global L2 sensitivity 22, which signifies that the maximum difference in the Le-norm of the outputs of f on any two neighboring databases is 22. and for any 2 0 and ó E [0, 1}. the Analytic Gaussian
Mechanism is defined as _Mxx) = f(x) (1x=+
where Z ~ ~~(00, I)). This mechanism satisfies (E, 6)-
differential privacy T (2
20 5z ) e A =o S 6, where $ J= -²² e-= /2
2x A.
is the cumulative distribution function (CDF) of the standard univariate Gaussian distribution.
When multiple queries are made on the same database, independent Gaussian noise is added to each query to maintain differential privacy. The privacy loss of the composition of Gaussian mechanisms formulated Theorem 2
Theorem (Moments Accountant M.. There exist constants e and C2 so that given the sampling probability q = N and the number of steps T, for any < e,g²I, DPSGD [1] is (E, diffffereenl
9v Tlolog(1/6)
private for any > 0 if we choose a > C2
Related Work
Performance. Traditional VFL approaches (29] 7 typically limited two-party scenarios. In contrast, existing multi-party VFL methods [12 51 33
38 21 (66 often rely on the assumption of precise identifiers that ensure perfect alignment across all parties. These methods generally employ the SplitNN framework (45), where each party maintains a portion of the model, and the models collaboratively trained well-aligned data samples through the transfer of representations and gradients, commonly known as split learning. However, the requirement for perfect data alignment is impractical in many real-world scenarios [50 3 where identifiers are often imprecise.
52
To address this limitation, semi-supervised VFL 22
30 19 20 has emerged, attempting to improve model performance by leveraging unlinked records through semi-supervised or self self-supervised learning. However, these methods still assume that datasets from each party can be precisely linked using exact identifiers, a premise that is often untenable in real-world settings [50 3). Given that the quality of linkage significantly impacts VFL accuracy (34] exploring effective linkage methods remains pivotal issue in VFL.
On the other hand, FedSim (50), based on real linkage projects at the German Record Linkage Center (GRLC) (3). acknowledges that the keys of each party are usually not precisely linkable and that records may have one-to-many relationships, leading to fuzzy linkage scenarios, as seen with keys like GPS addresses. FedSim enhances training performance by performing soft linkage and conducting training based on transmitted key similarities. Nonetheless, it faces limitations in scalability beyond two parties and introduces new privacy concerns by directly transferring similarities.
In summary, while existing studies face significant performance challenges when handling fuzzy keys in multi-party settings, our proposed FeT demonstrates scalable design that addresses these challenges and shows promising performance improvements in both multi-party fuzzy VFI. and two-party settings compared to FedSim.
3

---

Privacy. The privacy concerns associated with VFL are multifaceted. First, the primary party may infer data representations from secondary parties [3I. Second, the secondary party may derive gradients from the primary party 41 (4). Third, external attackers could conduct membership inference attacks [39] on the deployed model [51]. This paper primarily addresses the second concern: safeguarding representations, while acknowledging other concerns as open challenges.
To address the privacy of representations in VFL, various methods have been proposed, falling into two primary categories: encryption-based methods and noise-based methods. Encryption-bal
utilize computationally intensive cryptographic techniques to encrypt intermediate results. However, these methods often incur significant communication overhead when scaled to multiple parties. Conversely, noise-based methods [47] [66 protect data by perturbing 47 or manipulating [4]] local representations. These methods typically do not provide theoretical privacy guarantees or require substantial amounts of noise when scaling to multiple parties in VFI., which can degrade performance. Unlike existing studies that rely solely on either approach, this paper explores a combined strategy incorporating both encryption-based and noise-based methods, ensuring the model scales effectively to multiple parties without the need for excessive noise.
methods 26
52 38
36
4 Problem Statement
In this section, we formalize the concept of multi-party fuzzy VFL. We consider supervised learning task where party holding labels, termed the primary party P, collaborates with k: parties that do not hold labels, referred to as the secondary parties. The primary party P possesses n data records denoted as x { } along with corresponding labels y == {ys ) Each secondary party Sy maintains its own dataset S² All parties share common features, referred to identifiers, expressed as x' = [k', d'), where [.] signifies concatenation. These identifiers k' may exhibit inaccuracies and fuzziness, despite residing within the same range.
min C(((02p ...., x + 22(0)
In this formulation, ((-) denotes the loss function, 0 represents the model parameters, and (2()) refers to the regularization term. The symbol n indicates the number of samples in the primary party P.
which target local representations shared with the primary party. FeT operates under the assumption that all participating parties are butrestsbutccurious, meaning they adhere to the protocol but may attempt infer additional information about other parties. Furthermore, we assume that the parties do not collude with one another. While other forms of attacks, such as label inference attacks (13) and backdoor attacks aimed at compromising labels and gradients, exist, they are considered orthogonal to the objectives of this study. These additional threats will be explored in our future research.
Threat Model. This study focuses on defending feature reconstruction attacks 25
31
5 Approach
In this section, we address the performance and communication challenges inherent in fuzzy VFL. To tackle these issues, we introduce transformer-basee architecture named the Federated Transformer (FeT). This model encodes keys into data representations, thereby reducing reliance on key similarities. To accurately exclude incorrectly linked data records, we propose trainable dynamic masking module that generates masks based on keys. Furthermore, to combat overfitting caused by the large model and to alleviate communication bottlenecks faced by the primary party, introduce a party dropout strategy that randomly invalidates some partics during training. Additionally, we identify positional encoding misalignment issue across parties in the FeT and propose positional encoding averaging to ensure consistent alignment, thereby enhancing model performance.
5.1 Model Structure
The architecture of the FeT is illustrated in Figure 2 In FeT, each secondary party has encoder, while the primary party has both an encoder and subsequent decoder. The internal structure of both the encoder and decoder closely adheres to the conventional transformer model [43]. We utilize

---

Sum
Feed Forward
Neural Network Layer
Data matrix
Layerforrm
mask
Attention Block
Multi-head
Q
K
Rgg
Layerhlarm
Self Attention Layer
Q
Dynamic masking
Secure multi-porty summation
Black
Self-Attention Layer
Niee
Self-Attention Layer
flac
mask
Seff-Attention Layer
Tlaa
Feed
Feed
Q
Feed Forward
Forward
Multi dimensional
Positional Encoding
Encoding
Encoding
PE
Encodiin
key
Data x
key
Data x
key
Data S
Primary Party
Secondary Party St
Secondary Party Sx
Figure 2: Structure of federated transformer (PE: multi-dimensional positional encoding)
multi lilli-dimensional positional encoding [28] to integrate key information into feature vectors. Outputs from the encoders the secondary parties are aggregated and then fed into the decoder. Details regarding the privacy mechanisms employed during this aggregation phase are discussed in Section 6 while the details of the training process are elaborated in Section 5.2) We then elaborate on three techniques designed to improve performance and reduce communication costs.
Dynamic Masking. The size of the neighborhood varies significantly depending on the party and key values. Consequently, including large number of neighbors K for all parties can hinder the model's s ability to extract meaningful information and result in overfitting. To address this, we introduce dynamic "key padding mask" in the transformer, generated from the identifier values using a trainable MLP. This approach allows the model effectively disregard distant data records, thereby eliminating the influence of irrelevant data when K is large. This concept resembles the weight gate in FedSim,
diverges by using identifiers as inputs instead similarities, enhancing privacy by preventing the transmission of similarity data across parties.
The learned dynamic masking visualized Figure We derive two key insights from the visualization: (1) Dynamic masking effectively focuses on localized area around the primary party's 's identifiers. Data records with distant identifiers on secondary parties (in cooler colors) receive small negative mask values, reducing their significance in the attention layers without accessing the primary party's original identifiers. (2) The focus area varies in scale and direction across samples: for example, the left figure concentrates on small bottom area, the middle figure on small top area, and the right figure on broad area all directions.
Party Dropout. Extending the Federated Transformer (FeT) to support multiple parties can be challenging for several reasons. First, the communication bandwidth required by the primary party becomes a significant bottleneck within the SplitAvg framework, increasing linearly with the number of parties. Second, the inclusion of many parties can result in an excessive number of parameters, which may lead to overfitting. To mitigate these issues, we introduce the concept of party dropout. Inspired by traditional dropout [40], we randomly set a portion тa the parties' representations to zero during training. This method serves as a form of regularization, thus helping to reduce overfitting, while also significantly cutting down on communication overhead. In our experiments, demonstrate that increasing the party dropout rate to 0.8 leads to minimal accuracy loss and can
5

---

42
105
033
2.0
6.2
475
-10
Gpy Nareer
230
2.00
1.00
3.50
2.00
0.20 0.32 0400 133
000
Figure 3: Learned dynamic masks of different samples: Each figure displays sample (red star) from the primary party fuzzily linked with 4900 samples (circles) from 49 secondary parties. The position indicates the sample's 's identifier, and colors reflect learned dynamic mask values. Larger mask values signify higher importance in attention layers.
Pearson Corr. 0.5054
Pearson Corr. -0.0210
0.0203
distance
PE inout distance
inpur cizance
1.5
1.5
1.0
Pefo
1.0
0.5
0.5
Pc-P:
0.0
0.0
0.50 output distance
0.75
0.00
0.05
0.10
1.00
1.02
1.04
PE output distance
PE outout distance
(a) Keys: Po, encodings: Pi
P3
(b) Keys: Po. encodings: Po
(c) Keys: Po. encodings: P3
Figure 4: Misalignment of positional encoding (Po: primary party; P1 ~ Py: secondary parties)
even improve accuracy. Consequently, the communication overhead on the primary party can be reduced by up to 80%, enhancing scalability when dealing with large numbers of parties.
Like traditional dropout, it is crucial maintain consistent scaling of the representations during both training and testing phases. This consistency is naturally achieved within the SplitAvg framework. During the averaging process, if ratio rak of parties set to zero, we adjust by dividing only by the number of non-zero parties, (1 - ra)k. This method ensures that the scale of the averaged representations remains consistent across training and testing, regardless of the value of ra.
Positional Encoding Averaging. In positional encoding (PE), is generally expected that the distances between encoded representations are positively correlated with the distances between identifiers. FeT, each party employs its own encoder and PE layer, each tasked with encoding its local identifiers into representations. This configuration leads to significant PE misalignment issues, as illustrated in Figure 4 Although the identifiers and their corresponding encoded representations maintain a positive correlation within the PE layer of each party, there is almost no correlation between the identifiers and encoded representations across different parties. This lack of correlation causes integration issues and affects accuracy. However, directly sharing PE layer among all parties not viable due to privacy concerns. To address this, we propose positional encoding averaging.
Every Tpe epoch, the positional encoding layers are averaged and broadcast to all parties under secure multi-party computation (MPC) scheme, akin to FedAvg (23) in horizontal federated learning (27). While the privacy of the transmitted model can be a concern, this issue is an orthogonal open problem in horizontal federated learning.
5.2 Training
The FeT training process begins with employing evaluate identifier similarities between the primary party P and each secondary party. Secondary parties each contribute random subset for linkage (line 5). For each P's record, K nearest neighbors within these subsets from secondary parties are determined (line 6). The training leverages data embeddings of dimensions is the hidden layer size, following the transformer architecture. In FeT's context, I. for the primary and J. K for secondary parties, linking each primary record with K neighboring records from the secondaries. Identifiers transformed into vectors using multi-dimensional positional
Record Linkage (PPRL) 44
L x H, where B batch size, the sequence length, and H

---

encoding [28 and combined with data representations for processing via self-attention blocks (lines 10). Secondary parties' representations are averaged under the MPC protocol. The primary party then employs attention blocks for forward propagation to compute the final prediction (line 13). Backpropagation sends gradient updates from the primary to secondary parties to refine their local models (lines 14-16). The privacy mechanism including norm clipping (lines 8, 11) and distributed Gaussian noise (line 12) are further discussed in Section
Algorithm 1: Training Process of Federated Transformer
Input :Primary party x secondary parties x s.... x label y, noise scale sampling ratio g, clipping threshold C
:utput:Local models 0f" ,0," ....., and aggregation model 0."
Initialize model parameters , Of", .0f.0.... .0, 05 for epoch E [T] do for instance x." E xx" on primary party do for party {SI..... 3 do
x randomly choose ratio from x² x link K records with uf from x
R < [(0); x(!)
R + / max (1, [
// Norm clipping
end
R -- :0fF=x]]
10
R t RpP max (1. R| 112
11
// Norm clipping
12
H. + MPCAvg
R
R NO, (3,))
13
tK (
Vor + a6(yn.0))
P t 0 1)c Vor
14
30l
15
for party h (P.S ..... Sk] do
16
Vop
0 + V
17
end
18
end
19 end
Privacy
In this section, we address the challenges of privacy in multi-party party fuzzy VFL. First, the risk of transferring raw similarities has been mitigated by the design of the FeT itself. Second, address the inereasing costs when more parties join, we introduce a multi-party privacy-preserving VFL framework, SplitArg, which is compatible with FeT. The architecture of SplitAvg is illustrated in F SplitAvg integrates differential privacy (DP), secure multi-part computation (MPC) [6] and norm clipping to enhance the privacy of representations. Additionally, to further improve the utility of FeT under DP, we employ privacy amplification techniques that reduce the noise scales by incorporating random sampling.
6.1 Differentially Private Split Neural Network SplitAvg
This section outlines three techniques applied to the SplitAvg to improve privacy: representation norm clipping, privacy amplification, and MPC with distributed Gaussian noise. These strategies collectively protect the privacy of each secondary party's 's representations through differential privacy and ensure that privacy risks do not escalate with an increase in the number parties due to MPC.
Representation Norm Clipping. The magnitude of the f2-norm is pivotal in determining the sensi- tivity of differential privacy. To limit the maximum change of the E2-norm, norm clipping essential. Specifically, for representation R. ensure that |||| < C, where C is predefined positive
7

---

real number. This is achieved by scaling R by factor of C, formally, R = R/ max(1, 2 Through this process, any representation R with |R||| exceeding C scaled C, while values of [|||2 below C' remain unaffected.
Secure Multi-Party Computation with Distributed Gaussian Noise. To address the challenges of applying differential privacy in multi-party VFL, we propose method that integrates noise addition into the process of aggregating representations, facilitated through MPC. In this setup, each secondary party first independently conducts representation norm clipping to limit the scale of their data. Subsequently, these clipped representations, along with Gaussian noise (O, a o/k²) independently generated by each of the l secondary parties, are aggregated through averaging under MPC. For the primary party, this aggregation is equivalent adding independent Gaussian noise NO0, to the averaged result. The adoption of MPC in our framework ensures that the secondary parties do not need to individually add (0,, o²) noise to their representations. Instead, as the primary party only has access to the averaged result under MPC, each secondary party can add a smaller amount of noise. This method effectively improves utility with a small efficiency cost due to MPC.
Privacy Amplification by Secondary Subsam- pling. This technique specifically designed for FeT configurations. According to the princi- ple of privacy amplification [5), applying func- tion to a randomly sampled subset of data en- hances privacy compared applying the same function to the entire dataset. By initiating link- age from a randomly sampled subset rather than the full dataset, the privacy parameters effec- tively shift from (s, 6) to (gE, g9), where q < represents the sampling rate. In FeT, the primary party typically selects subsets of candidate data records for training from each secondary party, targeting those with neighboring identifiers. By pre-sampling these subsets at rate of before conducting a k-nearest neighbors (kNN) search, avoid processing the entire dataset, which in turn reduces the noise required to maintain the same privacy level.
FeT Modules
Decoder
SplitAug Privacy Modules
Secure Party Averaging
N(O,
N(O,
Norm Clip
Encoder
Linkage
Linkage
Sempling
Sampling
Primary Party
Secondary Party
Secondary Party Sk
Figure 5: Differentially private split-sum neural network
6.2 Privacy Guarantee
Our analysis of differential privacy focuses on a hypothetical global dataset linked using all secondary parties, denoted as x5 ...,.... S.. Since these datasets are correlated, removing one data record from this global dataset will result in changes to the representations in all secondary parties. Consequently, privacy loss accumulates across secondary parties without the use of MPC. However, with MPC, a single aggregated noise, formed by distributing smaller noise contributions among parties, can be added, effectively reducing the overall required noise. The privacy guarantee for these representations is formally articulated in Theorem 3 with the proof provided in Appendix A
Theorem 3. For certain constants er and ez, given sampling rale q, the total number of epochs T, and the number of batches in each epoch, each representation R achieves (c, 8) 3-diffeerenttal privacy for all æ < T, T, with any 8 > 0, by selecting the standard deviation of the Gaussian noise mechanism as follows:
av BT log(6(1/))
C2
7 Experiments
This section presents the experimental setup and results. We begin by outlining the experimental settings in Section [7.] followed by an assessment performance across varying numbers of parties and neighbors in Section 7.2 We then analyze the privacy of FeT in Section [.3 Additionally, an ablation study conducted in Appendix C to evaluate the contribution of each component to
performance, including dynamic masking, party dropout, positional encoding, key fuzziness, and
8

---

SplitAvg.
The performance of FeT with exact key matching is assessed in Appendix D while the computational and memory efficiency of MPC and training evaluated Appendix Privacy evaluation two-party real datasets is included in Appendix [F] Furthermore, Fe'T's performance under imbalanced feature splits across parties, based on VertiBench [49]. presented Appendix G
7.1
Experimental Settings
Datasets. Our experiments utilize five datasets, including three real-world datasets: house 35
2
bike 8 42 and hdb [8 37,, along with two high-dimensional datasets: gisette [66 and
MNIST [24] Detailed descriptions of these datasets can be found in Appendix B To simulate multi- party fuzzy VFL, we partition the features equally and randomly among the parties. The primary party's feature dimensions are reduced to 4 using principal component analysis (PCA) to serve as universal keys. To create fuzzy linked scenarios, we add independent Gaussian noise with a scale of 0.05 to the keys of each party.
Baselines. We include three baselines in our experiments: (1) Solo: training only on the primary party; (2) Sim: linking each data record in the primary party only with its most similar neighbor in the secondary parties; (3) FedSim [50]: training on the top K neighboring data records.
7.2 Performance
Two-party fuzzy VFI. In this experiment, we evaluate the performance of FeT in two-party settings without privacy mechanisms. The detailed results are presented in Table Our experiments demonstrate that FeT consistently outperforms the leading two-party fuzzy VFL methods. Notably, this performance improvement is achieved while enhancing privacy protections, as FeT does not involve transferring similarity data.
Table 1: Root Mean Squared Error (RMSE) on worldworld two-party fuzzy VFL datasets
Algorithm
house
bike
hdb
Solo
73.27 0.16
244.33
33.97 0.61
ToplSim
58.54
256.19 1.39
31.56 0.21
FedSim 42.12
235.67 0.27
27.13 : 0.06
FeT
39.75 = 0.29
232.98 = 0.62
26.94 # 0.15
Effect of Number of Neighbors K. In this experiment, we
the impact of the number of
neighbors, K, on FeT's performance by varying K from to 100. The results are displayed in Figure 6 The FedSim baseline trained using the optimal K value (i.e.. 50 for hdb and 100 for house and bike). The figure reveals two key insights: First, FeT's performance improves as K increases, demonstrating its ability filter useful information even as the number of unrelated data records grows. Second, FeT consistently outperforms all baselines at larger values of K, highlighting its superiority in fuzzy VFL scenarios.
house
taxi
hdb
Top1Sim
Top1Sim
Top1Sim
RMSE
50
RMSE
RMSE
240
FedSim (Best X)
FedSim (Best
FedSim (Best
40
0
50
100
0
50
100
50 Naighbors
100
Neighbors
Neighbors
Figure 6: Effect of Different Number of Neighbors K on FeT Performance
In this experiment, we assess FeT's performance in fuzzy VFL with various numbers of parties. Due the absence of real multi-party VFI. data, we employ synthetic data for our evaluations. We partition the features equally and randomly among the parties, reducing
Effect of Number of Parties

---

the primary party's 's feature dimensions to 4 using PCA as the universal keys. To simulate fuzzy linked scenarios, we add independent Gaussian noise with a scale of 0.05 to the keys of each party.
Figure 7 shows that Fe'T generally outperforms the baselines, particularly with a larger number of noise-
parties. This advantage is attributed to Solo's lack of informative features and affected linkage. FedSim performs poorly as the top-1 linked secondary parties are unaware of the primary parties' keys, leading to misalignment in subsequent soft linkage and training steps. On the gisette dataset with k = 10, FeT and other models slightly underperform compared to Solo, likely due to overfitting given gisette's small size.
MNIST (noise 0.05)
gisette (noise 0.05)
0.96
Accuucyy
0.6
FedSin
FXT
T2pl
ho2
20
30
40
50
20
30
40
50
Number of Putlies
Number of Parties
Figure 7: Impact of number of parties on FeT performance
7.3 Privacy
In this subsection, analyze how the performance of FeT varies with different noise scales (a) and sampling rates on secondary parties, demonstrating the impact of privacy constraints on its accuracy. The results are depicted in Figure X We observe three key points: First, moderate sampling rate has a negligible effect on model performance and may even slightly improve performance (e.g.. on the MNIST dataset) by reducing overfitting. Second, despite increasing noise levels and enhanced privacy guarantees, FeT consistently outperforms baseline models. Third, the illustrate that solely adding Gaussian noise without MPC, even with advanced analysis theorems such as Rényi Differential Privacy (RDP) (32] would require much larger noise scales compared to our approach that integrates MPC.
privacy-noise curves
0.7
gisette
2.2
MNIST
MNSST
0.59
dndendienao
10
dplars am1
0.00
3.00
10f
0.4
3.00
0.5
10
3.58
Figure 8: Impact of noise scale a on FeT accuracy and relationship between and & under 10-party fuzzy VFL (RDP: without MPC, privacy loss calculated by Rényí differential privacy)
Conclusion
In this study, we introduce the Federated Transformer (FeT), specifically designed to support multi- party VFL while effectively addressing critical challenges related to performance, privacy, and communication. Furthermore, we provide theoretical proof of FeT's differential privacy, ensuring that data representations remain protected from secondary parties. Notably, our experiments demonstrate that FeT surpasses baseline models, even under stringent privacy guarantees and within the traditional two-party setting, establishing its efficacy and robustness in complex federated environments.
Broader Impact. The architecture of FeT, even without privacy mechanisms, has potential applica- tions in multimodal learning. Multimodal tasks often require the alignment of data records across different modalities, which can be quite challenging. For instance, aligning 24Hz video streams with 48kHz audio tracks is complex, as each video frame may correspond to a range of audio samples. FeT has shown its capability to effectively learn from such fuzzily aligned data. Furthermore, the transformer model has proven effective across various data types, including images, text, and tabular data, highlighting FeT's suitability for multimodal learning applications.
10

---

Acknowledgments
This research/project is supported by the National Research Foundation Singapore and DSO National Laboratories under the Al Singapore Programme (AISG Award No: AISG2-RP-2020-013 the National Research Foundation Singapore and Infocomm Media Development Authority under its Trust Tech Funding Initiative. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(() and do not reflect the views of National Research Foundation Singapore, DSO National Laboratories, and Infocomm Media Development Authority. This research is also sponsored by Webank Scholars Program. We thank the AMD Heterogencous Accelerated Compute Clusters (HACC) program (formerly known as the XACC program Xilinx Adaptive Compute Cluster program) for the generous hardware donation.
References
[1] Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. 2016. Deep learning with differential privacy. Proceedings the SIGSAC conference on computer and communications security. 308-318.
the 2016 ACM
[2] Airbnb. 2019, Airbnb prices dataset. http:://hssideiirbbbb..omm the
[3] Manfred Antoni and Rainer Schnell. 2019. The past, present and future of the German Record Linkage Center (GRLC). Jahrbüüchr für Nationalökonomie und Statistik 239, (2019), 319- 331.
[4] Borja Balle and Yu-Xiang Wang. 2018. Improving the gaussian mechanism for differential privacy: Analytical calibration and optimal denoising. In International Conference on Machine Learning. PMLR, 394 403.
[5] Amos Beimel, Hai Brenner, Shiva Prasad Kasiviswanathan, and Kobbi Nissim. 2014. Bounds on the sample complexity for private learning and private data release. Machine learning 94 (2014), 401-437.
[6] Keith Bonawitz, Vladimir Ivanov, Ben Kreuter, Antonio Marcedone, H Brendan McMahan, Sarvar Patel, Daniel Ramage, Aaron Segal, and Karn Seth. 2017. Practical secure aggregation for
machine learning. In proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security. 1175-1191.
[7] Timothy J Castiglia, Anirban Das, Shiqiang Wang, and Stacy Patterson. 2022. Compressed- VFL: Communication- Eifffficatioo--Effcientt Learning with Vertically Partitioned Data. In Proceedings of the 39th International Conference Machine Learning (Proceedings of Machine Learning Research, Vol. 162), Kamalika Chaudhuri, Stefanie Jegelka, Le Song. Csaba Szepesvari, Gang Niu, and Sivan Sabato (Eds.). PMLR, 7738-2766. attps://proceett wtw.igss.m.l..p.....ee. mlr.sssss/ v162 v162/castigll wwwastiglla200 html
[8] CitiBike. 2016. Citi bike system data.
[9] Yuren Cong, Wentong Liao, Hanno Ackermann, Bodo Rosenhahn, and Michael Ying Yang. 2021. Spatial-temporal transformer for dynamic scene graph generation. In Proceedings of the IEEE/CVF international conference on computer vision. 16372-16382.
[10) Changyu Dong, Liqun Chen, and Zikai Wen. 2013. When private set intersection meets big data: an efficient and scalable protocol. In Proceedings of the 2013 ACM SIGSAC conference on Computer & communications security. 789-800.
[11] Johanna Eberle and Michael Weinhardt. 2016. Record linkage of the linked employer-employee survey of the socio-economic panel study (SOEP-LEE) and the establishment history panel (BHP). German Record Linkage Center Working Paper Series, No. WP-GRLC-2016-1 (2016).
[12] Siwei Feng, Han Yu, and Yuebing Zhu. 2024. MMVFL: A Simple Vertical Federated Learning Framework for Multi-Class Multi-Participant Scenarios. Sensors 24, (2024). https
//doi. //doi.org//l orw.o..g..10. 3390/ 3000033300/000000 s24020619
11

---

[13] Chong Fu, Xuhong Zhang, Shouling Ji, Jinyin Chen, Jingzheng Wu, Shanqing Guo, Jun Zhou, Alex X Liu, and Ting Wang. 2022. Label inference attacks against vertical federated learning. In 3/st USENIX Security Symposium (USENIX Security 22). USENIX Association, Boston, MA, 1397-1414.
httpps://wws www.ei..org/e.ee.ee.ee..eeeeeeet /ttp///frrreeeeesseeeeeeererrr//
presentation on oowooooofuuuuuee fu-chhng
[14] Fangcheng Fu, Huanran Xue, Yong Cheng. Yangyu Tao, and Bin Cui. 2022. BlindFL. Vertical Federated Machine Learning without Peeking into Your Data. In Proceedings of the 2022 International Conference on Management of Data (Philadelphia, PA, USA) (SIGMOD '22). Association for Computing Machinery, New York, NY, USA, 1316-1330. https:
org/10. 0145/3514222 35142211122222222 35261227
[15] O. Goldreich, S. Micali, and A. Wigderson. 1987. How to play ANY mental game. In Pro- ceedings of the Nineteenth Annual ACM Symposium on Theory of Computing (New York, New York. USA) (STOC 887) Association for Computing Machinery, New York, NY, USA, 218-229. nttps://doi.org:: wtt.e//r///....////1111 28420
[16] I.
Guyon, S. Gunn, A. Ben-Hur, and G. Dror. 2004. Gisette Dataset. https: /t:::///ww /wwp/www.. csie ntu. edu. http://////aioooo/aaaa/aa///////
bz2
[17] Stephen Hardy, Wilko Henecka, Hamish Ivey-Law, Richard Nock, Giorgio Patrini, Guillaume Smith, and Brian Thomn. 2017. Private federated learning on vertically partitioned data via entity resolution and additively homomorphic encryption. arXiv preprint (2017).
[18] Singapore HDB. 2018. Resale flat prices in Singapore. resale- Sheale-fattane flatfllattriiei prices
[19] Yuanqin He, Yan Kang, Xinyuan Zhao, Jiahuan Luo, Lixin Fan, Yuxing Han, and Qiang Yang. 2022. A hybrid self-supervised learning framework for vertical federated learning. arXiv preprint arXiv:2208. 08934 (2022).
[20] Chung-ju Huang, Leye Wang, and Xiao Han. 2023. Vertical Federated Knowledge Transfer via Representation Distillation for Healthcare Collaboration Networks. In Proceedings of the ACM Web Conference 2023. ACM, 4188-4199.
[21] Yimin Huang, Wanwan Wang, Xingying Zhao, Yukun Wang, Xinyu Feng, Hao He, and Ming Yao. 2023. EFMVFL: an efficient and flexible multi-party vertical federated learning without third party. ACM Transactions on Knowledge Discovery from Data 18, 3 (2023), 1-20.
[22] Yan Kang, Yang Liu, and Xinle Liang. 2022. FedCVT: Semi-supervised vertical federated learning with cross-view training. ACM Transactions on Intelligent Systems and Technology (TIST) 13, 4 (2022), 1-16.
[23] Jakub Konečný, H Brendan McMahan, Felix X Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. 2016. Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv: 1610.0559 (2016).
[24] Yann LeCun and Corinna Cortes. 2005. The mnist database of handwritten digits. https //api. www.ssemaniicccm sww.anticcccccollarr org/ 602826629
[25] Jingtao Li, Adnan Siraj Rakin, Xing Chen, Zhezhi He, Deliang Fan, and Chaitali Chakrabarti. 2022. Ressfl: A resistance transfer framework for defending model inversion attack in split federated learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. 10194-10202.
[26] Jialin Li, Tongjiang Yan, and Pengcheng Ren. 2023. VFL-R: a novel framework for multi-party in vertical federated learning. Applied Intelligence 53, 10 (2023), 12415.
[27] Qinbin Li, Zeyi Wen, Zhaomin Wu, Sixu Hu, Naibo Wang, Yuan Li, Xu Liu, and Bingsheng He. 2021. A survey on federated learning systems: Vision, hype and reality for data privacy and protection. IEEE Transactions on Knowledge and Data Engineering (2021).
12

---

[28] Yang Li, Si Si, Gang Li, Cho-Jui Hsieh, and Samy Bengio. 2021. Learnable fourier features for munti-dimmensoonal spatial positional encoding. Advances in Neural Information Processing Systems 34 (2021), 15816-15881
[29] Yang Liu, Xinwei Zhang, Yan Kang. Liping Li, Tianjian Chen, Mingyi Hong, and Qiang Yang. 2022. FedBCD: A communication--fficiee collaborative learning framework for distributed features. IEEE Transactions on Signal Processing 70 (2022),
[30] Yang Liu, Xiong Zhang, and Libin Wang. 2020. arXiv (2020).
Asymmetrical vertical federated learning.
[31] Xinjian Luo, Yuncheng Wu, Xiaokui Xiao, and Beng Chin Ooi. 2021. Feature inference attack on model predictions in vertical federated learning. In 2021 IEEE 37th International Conference on Data Engineering (ICDE). IEEE, 181-192.
[32] Ilya Mironov. 2017. Rényi differential privacy. In 2017 IEEE 30th computer foundattion symposium (CSF). IEEE, 263-275.
Multi-vfl: A vertical federated [33] Vaikkunth Mugumthan, Pawan Goyal, and Lalana Kagal. 2021.
learning system for multiple data and label owners. arXiv preprint arXiv:2106.00 056666 (2021).
[34] Richard Nock, Stephen Hardy, Wilko Henecka, Hamish Ivey-Lav, Jakub Nabaglo, Giorgio Patrini, Guillaume Smith, and Brian Thome. 2021. The Impact of Record Linkage Learning from Feature Partitioned Data. In Proceedings of the 38th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 139), Marina Meila and Tong Zhang mlr.
(Eds.). PMLR, 8216-8226.
html
[35] Qichen Qiu. 2017. Kaggle dataset: housing price in Beijing. https://www.kat com/ ruiqurm/lianjaa
[36] Xinchi Qiu, Heng Pan, Wanru Zhao, Chenyang Ma, Pedro PB Gusmao, and Nicholas D Lane. 2023. vFedSec: Efficient Secure Aggregation for Vertical Federated Learning via Secure Layer arXiv preprint vrXii:2305.1679 (2023).
[37] Salary.sg. 2020, Secondary school rankings in Singapore, wwwinniiinng20---pe 2aw--22000----- put---pppe--cctoff-.eee
[38] Haoran Shi, Yali Jiang, Han Yu, Yonghui Xu, and Lizhen Cui. 2022. MVFLS: multi-participant vertical federated learning based on secret sharing. The Federate Learning (2022), 1-9.
[39] Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. 2017. Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP). IEEE, 3-18.
[40] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research 15, (2014), 1929-1958.
[41] Jiankai Sun, Xin Yang, Yuanshun Yao, and Chong Wang. 2022. Label leakage and protection from forward embedding vertical federated learning. arXiv preprint a5X (2022).
[42] New York TLC. 2016 TLC trip record data. Elc-trriprrecordddatt.p page.
[43] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017).
[44] Dinusha Vatsalan, Ziad Sehili, Peter Christen, and Erhard Rahm. 2017. Privacy-preserving record linkage for big data: Current approaches and research challenges. In Handbook of Big Data Technologies. Springer.
13

---

[45] Praneeth Vepakomma, Otkrist Gupta, Tristan Swedish, and Ramesh Raskar. 2018. Split learning for health: Distributed deep learning without sharing raw patient data. arXiv preprint arXiv: 1..20.00064 (2018).
[46] Praneeth Vepakomma, Abhishek Singh, Otkrist Gupta, and Ramesh Raskar. 2020. NoPeek: Information leakage reduction to share activations in distributed deep learning. JCDM Workshop (2020).
[47] Chang Wang, Jian Liang, Mingkai Huang, Bing Bai, Kun Bai, and Hao Li. 2020. Hybrid differen- tially private federated learning on vertically partitioned data. arXiv preprint 20XXv:2009.02765 (2020).
[48] Webank. 2023.
The application and development report of open-source privacy comput- ing framework (FATE) in financial industry. https://www. 163. wttp:/.ccoooo///.//////aooo HR7MSK7PO552199HH html
[49] Zhaomin Wu, Junyi Hou, and Bingsheng He. 2024. VertiBench: Advancing Feature Distribution Diversity in Vertical Federated Learning Benchmarks. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 1, 2024. OpenReview.net. httpp://oppnreevie.. wtw.pparrevieee.eeeffrrr.... wig.:rrwuemdddwlwbbeeeeee
[50] Zhaomin Wu, Qinbin Li, and Bingsheng He. 2022. Coupled Design of Exploiting Record Similarity for Practical Vertical Federated Learning. Advances Neural Information Processing Systems 35 (2022), 21087-21 100.
[51] Zhaomin Wu, Qinbin Li, and Bingsheng He. 2022. Practical vertical federated learning with unsupervised representation learning. IEEE Transactions on Big Data (2022).
[52] Yang Yan, Guozheng Yang, Yan Gao, Cheng Zang, Jiajun Chen, and Qiang Wang. 2022. Multi- participant vertical federated learning based time series prediction. In Proceedings of the 8th International Conference on Computing and Artificial Intelligence. 165-171.
[53] Liu Yang, Di Chai, Junxue Zhang, Yilun Jin, Leye Wang, Hao Liu, Han Tian. Qian Xu, and Kai Chen. 2023. Survey on Vertical Federated Learning: From a Layered Perspective. arXiv preprint 2304022304.01829 (2023).
[54] Tianyuan Zou, Yang Liu, Yan Kang, Wenhan Liu, Yuanqin He, Zhihao Yi, Qiang Yang, and Ya-Qin Zhang. 2022. Defending batch-level label inference and replacement attacks in vertical federated learning. IEEE Transactions Big Data (2022).
14

---

Appendix
Table of Contents
A Proof
15
B Experimental Details
16
C Ablation Study
16
C.1 Dynamic Masking
16
Party Dropout
16
C.3 Positional Encoding
17
C.4 Fuzziness of Keys
17
C.5 SplitAvg vs. SplitN
17
Exact Linkage
18
E Efficiency
F Privacy on Two-Party Real Datasets
20
Performance Imbalanced Split
20
Limitations
21
License
21
A Proof
Theorem 3. For certain constants ei and 62, given sampling rate q, the total number of epochs
T, and the number of batches B in each epoch, each representation R achieves (e, 8) 8)ddifeerentla privacy for all E < TT with any > 0, by selecting the standard deviation a of the Gaussian noise mechanism follows:
av BT log(////))
Proof. The proof leverages the moments accountant bound [I. which is applicable to a sequence of Gaussian mechanisms applied to subsampled data. We begin by establishing that the output's f2-norm for each function is constrained by constant C. This constraint ensures that each randomized function adheres to differential privacy under the Gaussian mechanism. By determining the cumulative count of Gaussian mechanisms applied, we can directly invoke Theorem 2 to reach our conclusion.
To clarify the process, we apply norm clipping to each party specified in Section party at a rate of C/k. This scaling guarantees that, for every Rý the condition ||R] ||2 S C/k is satisfied. Using the triangle inequality within normed vector spaces, we derive:
6 scaling each
H 2
Σ
R
Σ
= C
C C
(2)
h=1
Since the f2-norm of H, is bounded by C, adding Gaussian noise H, satisfies the conditions for differential privacy. Throughout the training, B ' independent noises are introduced, resulting in sequence Gaussian mechanisms targeting randomly selected subset at ratio g. Consequently, Equation is derived by directly applying Theorem
15

---

B Experimental Details
Datasets. In this section, we include the detailed information of each dataset used the experiment. These real-world datasets are obtained in the same manner those utilized in FedSim, with each dataset comprising two parties. Details of the real datasets are presented in Table The synthetic dataset, gisette [16], consists of 6,000 instances and 5,000 features and serves binary classifi- cation task with balanced labels. The MNIST dataset [24] consists of 60,000 instances and 28x28 features for 10-class digit classification task.
Table 2: Basic information of real-world VFL datasets
Primary Party (w/ labels)
Secondary Party
Identifiers
Dataset
Task
#samples
#features
ref
#samples
#fcatures
ref
#dims
type
house 141,050
55
35
27,827
25
2
2
float regression
bike 100,000
6
8
200,000
964
42
4
float regression
hdb 92,095
70
18
165
37
2
float regression
Metrics. For regression tasks, Root Mean Square Error (RMSE) is utilized, while accuracy is applied classification tasks. Early stopping is performed based on the validation set, with the corresponding test scores reported.
Hyperparameters. Each algorithm was run until convergence, with maximum of 50 to 100 epochs. The learning rate and weight decay were consistently set at 10-3 and 10-5, respectively. For the Solo model, a multi-layer perceptron (MLP) with two hidden layers 400 units each was employed. In contrast, the Top1Sim utilized single-layer MLP with hidden size of 200 for both local and aggregation models. For FedSim, the number K neighbors was selected from the set (50, 100}. For FeT,, the number of blocks is set to 6 for both local model and aggregation model.
Environments. We evaluate FeT on a server equipped with dual Intel Xeon Gold 6346 CPUs, eight A100 GPUs with CUDA version 12.2, and 1008GB RAM, running Python 3. 3.10.13 with PyTorch 2cll12cu121 on Linux kernel 6.5.0. Efficiency experiments were conducted on a machine powered by 64-core Intel(R) Xeon(R) Gold 6226R CPU @ 2.90GHz and 376 GB of RAM. Each experiment was run five times, and we report the average and standard deviation.
C Ablation Study
In this section, we evaluate the performance improvement of each proposed component of the FeT. Our findings indicate that dynamic masking is crucial for enhancing performance, while both party dropout and positional encoding averaging contribute modestly these improvements. Detailed analyses are provided below.
C.1 Dynamic Masking
We evaluate the performance of FeT with and without dynamic masking, as shown in Table The evaluation includes two-party datasets (house, bike, hdb) and two 50-party synthetic datasets (MNIST and gisette). The results indicate that dynamic masking leads to an improvement of up to 13 percentage points, particularly noticeable in datasets with large number of parties. This suggests that dynamic masking significantly enhances model performance, especially in multi-party settings.
C.2 Party Dropout
Next, we evaluate the effect of the dropout rate under specific hyperparameter settings: the number of parties 50, the number of neighbors dropout rate is demonstrated in Figure and Table Our observations reveal that moderate party dropout rate of 0.6 enhances FeT's generalization by reducing the model size. Notably, FeT maintains stable performance even the dropout rate increases 0.8. This indicates that party dropout not
100, and key noise set 0.05. The impact of the party
16

---

Table 3: Effects of Dynamic Masking (DM) and Positional Encoding (PE) on FeT Performance
Datascts (metric)
Model
house (RMSE)
bike (RMSE)
hdb (RMSH)
MNIST (Aceuracy)
gisette (Accuracy)
FeT w/o PE
43.28 0.74
234.25 + 0.93
27.31 # 0.23
FeT w/o DM
42.48 0.45
236.26 + 0.71
29.13 + 0.18
72.89% : 1.43%
90.32% + 0.52%
FeT
39.75 0.29
232.98 # 0.62
26.94 + 0.15
85.47% # 0.13%
92.43% + 0.24%
only improves generalization but also significantly reduces communication overhead across parties. Based on these findings, we set the dropout rate to 0.6 by default in multi pulti-pprty experiments.
mnist
gisette
0.85
0.93
0.92
0.91
0.90
0.65
0.60
0.8
0.8
Farty Drapout Rate
Party Drapout Rate
Figure 9: Effect of party dropout rate on FeT
Table 4: Effect of Party Dropout Rates on FeT Performance
Party Dropout Rate
Datasct
0.4
0.6
0.8
1.0
gisette
18% 0.47%
92.70% 0.30%
92.48% 0.18%
92.90%
0.41 85,36 % 0.25%
92.52%
0.40%
89.15%
0.64%
MNIST 83.54% + 0.30%
84.39% + 0.35%
85.06% 0.11%
83.85% # 0.11%
61.21% 0.58%
C.3 Positional Encoding
We now assess the effect of the frequency of positional encoding (PE) averaging, as depicted in Figure [o] and Table[5] We find that PE averaging yields improvements, particularly with large number parties, such as 50 on MNIST, where alignment of encodings becomes crucial. Based on our observations, we set the frequency to 1 in most experiments.
Additionally, assess the impact of positional encoding on the performance of FeT, detailed in Table These evaluation of MNIST and gisette are conducted with the number of neighbors K 100 and key noise 0.05. The results indicate that positional encoding is important for enhancing the performance of FeT.
C.4 Fuzziness of Keys
We evaluate the impact identifier fuzziness on FeT's performance by introducing Gaussian noise of varying scales to the keys. The results are presented in Figure [1 From the figure, we derive two key observations: (1) Both FeT and baseline models show improved performance in more balanced scenarios. (2) FeT consistently outperforms the baselines across different levels of heterogeneity. demonstrating robustness varying degrees of noise. These findings highlight the resilience of FeT in the presence of noise, which is critical for practical applications.
C.5 SplitAvg vs. SplitNN
To evaluate the comparative performance of the SplitAvg (without noise) and SplitNN, we conducted training for both models using identical hyperparameters on the same VFL dataset, gisette. The
17

---

mnist
gisette
0.950
0.96
Accuracy
0.925
Accuracy
0.900
0.94
0.875
Party
Party
Parly 5
Parly 5
0,850
Party
Party 20
Party
Party
0.825
10
10
Average PE Frequency
Average PE Frequency
Figure 10: Effect of frequency of positional encoding averaging
Table 5: Ablation study for accuracy with different PE Average Frequency
Dutasel
#parties
Average Frequency
2
3
10
96.40% # 0.25%
96.33% 0.33%
96. 12% 0.29%
96.30% + 0.53%
96.37% + 0. 0.1%%
96.08% # 0.32%
gisette
95.92% 0.31%
0.26% 96.02%
95.87% 0.46%
95.73% 0.12%
95.60% # 0.27%
95.75% 0.37%
20
93.92% a 0.31%
43 83.87% 0.35%
0.60% 91.53% 0.00%
93.85%
94.13% 1 0.27%
93.87% a 0.42%
94.23% = 0.48%
50
92.05% 0.15%
92.50% 0.19%
91.72% 0.34%
91.92% + 0.18%
92.07% = 0.14%
95.57% a: 0.12%
95 88% 0..99%
95.70%
0.16%
95.70% a: 0.201%
95.74% a: 0.13%
95.89% = 0.09%
MNIST
20
88 84% + 0.31%
93.97% 0.22%
93.92%
79% 0.1%%
0.11%
93.84%
0.13% 88.77%
0,18%
94.01%
0.12% 88.8%%
0.20%
88. 79%
93.94% 1 0.24%
+ 0.19%
93.95% 0.09%
88.71% = 0.24%
50
83.43%
0.16%
83.41% 0.44%
83.70% 0.09%
83,48% 0.06%
83.57% 0.22%
83.78% 0.33%
outcomes are illustrated in Figure 12 Analysis of the figure yields two primary observations. Firstly, SplitNN and SplitAvg exhibit very similar loss and accuracy curves during training, indicating that both models behave very similarly. Secondly, upon expanding the number of participating parties to 128, we observe that the performance curves of both models remain closely aligned, albeit with the split-sum model exhibiting marginally lower accuracy. This minor discrepancy is attributed to the increased model parameters in SplitNN, which can typically be compensated for by increasing the number of parameters in SplitAvg.
D Exact Linkage
While FeT primarily focuses scenarios involving fuzzy linkage, we also evaluate its robustness in exact linkage contexts. To achieve this, we synthesize exact linkage data by generating pure random keys within the range of [-1, 1, 1] without introducing any noise. Each party is randomly divided into five or ten groups, with each group containing an equal number of features. Importantly, each party retains the exact keys, ensuring controlled environment for our evaluation.
The results of our experiments are summarized in Table From the table, we observe that Top Sim achieves the highest accuracy, as it is inherently well-suited for exact linkage scenarios. In contrast,
gisette (50 parties)
mnist (50 parties)
FeT
0.90
FeT
0.950
Solo
0.85
Solo
Accuracy
0.925
Top15im
Accuracy
Top15im
0.900
0.75
0.875
0.70
0.850
0.65
0.02
0.04
0.06
0.08
0.10
0.02
0.04
0.06
0.08
0.10
Noise Scale on Identifiers
Noise Scale on Identifiers
Figure 11: Effect of Fuzziness of Identifiers. The x-axis the scale Gaussian noise added to precisely matched identifiers.
18

---

guerta partbss))
gustle 1128 partess
8.5
G49
0.50
6.0
0.0
633
Bpach
Figure 12: Test loss and accuracy curve of SplitAvg and SplitNN under same hyperparameters
the accuracy of FeT shows a slight decrease, which may be attributed to overfitting; however, its performance remains competitive and does not suffer significantly in this context.
Table 6: Performance of FeT under Exact Linkage
Algorithms Top
Dataset
Solo
FeT (ours)
gisette (5-party)
96.58% 0.25%
97.52% 0.26%
94.73% + 0.60%
MNIST (5-party)
95.96% = 0.03%
96.96% % # 0.09% %
96.53% = 0.38%
gisette (10-party)
96.30% 0.25%
97.57%
0.25%
94.58% + 0.41%
MNIST (10-party)
92.09% = 0.11% %
96.95%
0.06%
96.97% + 0.08 %
E
Efficiency
Parameter Efficiency. In our analysis, we assess the computational efficiency standard addition compared to multi-party party computation (MPC) addition, as shown in Table 7 Under the arithmetic GMW protocol [15], and given that the size of the aggregated vector varies by dataset, we use typical size for our experiments. Specifically, we conduct MPC addition to aggregate 10.000-dimensional vectors from multiple parties. Each experiment performed five times, with the average timing reported. Although MPC generally incurs higher computational requirements, the results in Table indicate that aggregating high-dimensional vectors via MPC incurs only a one-second overhead, even the number of parties increases to 100. This minimal time cost is relatively small, especially when compared to other factors such as communication costs. Therefore, our findings suggest that MPC remains feasible and efficient approach for representation aggregation in the context of VFL.
Table 7: Running time of summation with and without MPC in seconds #parties
Sum MPC Sum
Overhead
2
5.29
10
5.09 10 10m4
5.04 10
5
2.10 x 10
2.75 x 10
2.73 X 10-3
10
4.46 x 10-5
1.02
10
1.01 x 10-2
20
1.29 10
4.36 10-2
4.36
10-2
50
2.64 x 10
0.268
0.268
100
5.21 x 10
1.06
1.06
Training Computational and Memory Efficiency. We evaluate the computational and memory efficiency of FeT during training on an RTX3090 GPU with a batch size of 128. The results, shown in T lead to three key observations: (1) FeT has comparable number of parameters to FedSim; (2) FeT demonstrates improved memory efficiency compared FedSim, although this improvement comes with trade-off in training speed; and (3) the additional components, such as dynamic masking (DM) and positional encoding (PE), introduce only minor overhead terms of both parameters and computational cost.
19

---

Table 8: Training efficiency FeT on RTX3090. PE: positional encoding: DM: dynamic masking,
#parameters
Train Seconds / epoch
Peak GPU Memory (MB)
Dataset
house
bike
hdb
house
bike
hdb
house
bike
hdb
FedSim
3,47M
1.85M
1.87M
38
2016
1917
1930
FeT w/o PE
0.98M
3.24M
0.63M
35
50
15
397
691
539
FeT w/o DM FeT
0.98M
2.89M
0.51M
37
54
401
721
569
0.98M
3.29M
0.63M
37
55
17
401
746
571
Privacy on Two-Party Real Datasets
In this section, we explore how the performance FeT varies with different noise scales and secondary sampling rate, illustrating the influence of privacy constraints on its accuracy. The outcomes are depicted in Figure [3) From this figure, we observe two key points. First, for large secondary datasets like bike, moderate sampling rate has negligible effect on model performance. Conversely, for smaller secondary datasets like hdb, performance quite sensitive to sampling rates. Second, the noise scale increases for secondary parties, the performance of FeT does not degrade sharply; instead, it gradually converges to a state where only primary features are utilized due to our dynamic masking design. In this scenario, FeT also outperforms MLP-bases Solo primarily due to transformer's key encoding, which has proven to more effective than incorporating all keys into the training process, evidenced in opatial-temporal prediction tasks (9.
house
taxi
hdb
75
250.0
Salo
65
245.0
Solo
RMSS
242.5
RMS
31
240.0
30
50
29
235.0
232.5
dpSample=0.1
dpSample=0.5
dpSample=1.0
Solo
Solo Std
Figure 13: Impact of noise scale a on FeT performance
Next, we explore the relationship between and as outlined in Theorem 3 setting hyperparameters reflect typical training conditions. The number of epochs chosen based common convergence epochs: 10 for bike, and 50 for house and hdb. We adopt a batch size of 8k and set ô to 1 1/N, with N representing the size of party S1. This correlation between and is depicted in Figure 14 The figure illustrates that reasonable noise levels can yield robust privacy guarantees. For instance, within noise scale conducive maintaining competitive performance, FeT achieves
for hdb and e = 5 for house, indicating effective privacy preservation under practical noise conditions.
G Performance on Imbalanced Split
The preceding experiments were conducted using balanced feature split for VFL. Building on this foundation, we extended our evaluation FeT include datasets with varying levels of imbalance, motivated by the recent benchmarks presented in VertiBench [49]. The MNIST datasets are divided by features according to the methodology described in VertiBench [4), utilizing imbalance parameters E (0.1, (0011,033 0.5, 1.0, 5.0, 10.0, 50.0), where higher value denotes greater balance across parties. The findings, illustrated in Figure [5] lead to two key observations: firstly, both FeT and the baseline algorithms exhibit improved performance in more balanced scenarios. Secondly, despite the varying
20

---

house 104
Laxi
hdb
R0P
10h-
101
Figure 14: Relationship between and noise
levels of data imbalance, FeT consistently shows competitive or superior performance relative to the baselines.
mnist (10 Parties)
mnist (20 Parties)
Accuracy (%)
Accuracy (%)
80
80
Top15im
Top15im
10-1
10a
10a
10
10a
10a
Figure 15: Performance on feature split with different level of imbalance
H
Limitations
The design of FeT includes three primary limitations that warrant careful consideration. First, FeT operates under the assumption that common features exist across all parties. While this assumption valid in many scenarios, it may not hold in more complex situations where the parties lack shared set of features. This limitation necessitates further investigation into alternative frameworks or adaptations that accommodate such cases, particularly in heterogeneous environments.
Second, although FeT facilitates the application scalable differential privacy across multiple parties, the stringent privacy safeguards can lead to significant accuracy reductions when operating with low values of e. This trade-off between privacy and utility is particularly concerning in performance- sensitive applications, where quantifying the extent of accuracy loss is essential for informing users about the potential impacts on their results. Future work should explore methods to balance privacy and accuracy effectively.
Third, similar other fuzzy VFL methods [50], FeT assumes a correlation between identifiers and data representations. This assumption may not hold in cases where identifiers are randomly generated, which could lead to overfitting and minor performance deficits compared to Top1 approaches. Experiments on such datasets (Appendix()) [D) indicate that while FeT performs well in many scenarios, its effectiveness may vary significantly depending on the nature of the data and the key generation process. Therefore, further empirical studies are needed to assess FeT's robustness across diverse datasets and identifier generation strategies.
License
The licenses of the datasets used in this work presented in Table We utilize the code from FedSim [50] as our baseline, which is licensed under the Apache V2 Ticense" Our own code will also be open-sourced under the Apache V2 license.
httpps//ww..ppccceerrg/////////e http://wwww.apaoooggggogoeaeeaaa
21

---

Table 9: Licenses of datasets
Datasct
License
Adapt
Share
Commercial
35
37
CCO 1.0"
NYCBS Data Use Policy"
CC BY-NC-SA 4.0%
Singapore Open Data License"
X
All rights reserved
https://reat..veommmmsss.ccrr http:///ww...oo.oto.ooo//oooooaa
https: httprr/eieeeooooorrrrrrraaeaaaae
httpa: http://ta.ddaa.aggom gov http://eea....aaa..r..e///eeaaae
NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: We have stated in abstract and introduction that the paper proposes novel FeT framework to address the scalability and privacy issues Vertical Federated Learning (VFL).
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
See Section g
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
22

---

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: See Section 6 and Appendix A
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: See Section 7.1 and Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm.
23

---

(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e-g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The codes are available at a GitHub repository https://giite com/ Xtra- Computing/FeT
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.com cc/
mublic/guides/ oaapai/ss/ooSSttuuaassssoooooooo
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips.cc http/uullcgggguugee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permirted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: See Section 7.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
24

---

Answer: [Yes]
Justification:
We repeat the experiments with five different random seeds.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
See the paragraph on environments in Section 7.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We follow the code of ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special cireumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
25

---

Question:
Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See Section g
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (c.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deploymeett of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses such risk.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have cited the used datasets in Section 7.1 and list the licenses in Ap-
pendix]] !1
Guidelines:
26

---

The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: See Appendix
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
27

---

Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
28


Reviews:
[{"review_id": "xgw2ks5Cjn", "reviewer": "Reviewer_jPVu", "summary": "This paper proposes the Federated Transformer (FeT) framework, which aims to address performance and privacy challenges in multi-party fuzzy vertical federated learning (VFL). FeT leverages the Transformer architecture to encode fuzzy identifiers and distribute training across different parties. The authors introduce three innovative techniques—position encoding averaging, dynamic mask module, and party dropout strategy—to enhance model performance while minimizing computational and communication overhead. Additionally, FeT incorporates a scalable privacy framework that combines differential privacy and secure multi-party computation, effectively safeguarding local data representations and ensuring manageable privacy maintenance costs. Experimental results demonstrate that FeT outperforms baseline models, offering superior performance and enhanced privacy protection. Overall, FeT overcomes the limitations of existing models in multi-party fuzzy VFL, showcasing exceptional performance and practicality.", "questions": "same as the weakness", "limitations": "NA", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This manuscript proposes the FeT framework. The framework combines federated learning with the Transformer architecture. At the same time, the dynamic mask technique, participant discarding mechanism, and location coding average techniques are applied to improve the model's performance and privacy protection in multi-party fuzzy vertical federation learning.\n\n2. The FeT framework introduces SplitAvg, a hybrid privacy protection mechanism, which effectively reduces the introduction of noise and improves data utility. In addition, the privacy amplification technology enhances the privacy protection effect. This ensures data privacy as well as the efficiency and robustness of the model.", "weaknesses": "This manuscript proposes a scalable vertical federated learning for practical fuzzy linked data, aiming to solve the performance and privacy issues of real-world VFL in its application. However, there are still some major issues, as follows:\n\n1. In the Introduction section, the definition of fuzzy identifiers and multi-party fuzzy VFL and the relationship between the identifiers, fuzzy identifiers and multi-party fuzzy VFL is not clearly introduced, which may lead to confusing concepts for readers. In addition, the important role of fuzzy identifiers in linking real-world data sets should be emphasized to prove the practical value of studying multi-party fuzzy VFL.\n\n2. In the training of FeT, PPRL was first used to evaluate the identifier similarity between the primary participant P and each secondary participant. However, the author did not mention this method in the previous content. Based on this, a very important question is: in the comparative experiment, the author did not seem to compare the performance gap, privacy gap, and computing resource consumption gap between the training using only the PPRL method and the training based on the transformer structure in this article?\n\n3. In FeT, the combination of dynamic mask, participant discarding mechanism, and hybrid privacy protection mechanism will inevitably affect the training process's time, computing resource consumption, and privacy protection ability. Therefore, it is also important to compare the training efficiency and computing resource consumption, as well as the privacy protection ability of FeT methods with current mainstream methods. This represents whether FeT can surpass other methods in real-world applications. The setting of the experiment may be as follows: 1. Training time of FeT and mainstream methods on the same data set and under the same conditions. 2. CPU/GPU usage between FeT and mainstream methods in the training process to measure memory consumption and network communication overhead, especially in the case of multi-party participation. 3. Evaluate the privacy protection effect of FeT and mainstream methods through quantified privacy protection indicators (such as ε value of differential privacy).\n\nMinor weaknesses:\n1. In lines 27 and 28, the authors describe collaboration between hospitals, financial institutions, and sensors. It is equivalent to juxtaposing the three, which is unreasonable.\n2. In lines 58 and 59, the authors mention that experiments have shown that model accuracy has been improved by up to 13 percentage points on a 50-square fuzzy VFL on the MNIST dataset. It is not given here which method is compared with, please add.\n3. In line 139, the author defines a common feature shared by all parties as an identifier, expressed as \\(x^i = [k^i, d^i] \\), where \\([\\cdot]\\) signifies. \\([,] \\) and\\([\\cdot]\\) is not the same", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments; we have addressed all concerns as follows.\n\nW1. We will include a formal definition in Section 4 and the introduction. Our scenario extends the two-party fuzzy VFL model defined in FedSim [1] to a multi-party setting. Below, we clarify the key terms:\n\n- **Multi-party VFL**: Federated learning tasks involving $k+1$ parties, denoted as $\\{S_h\\}_{h=0}^k$. Each party $S_h$ has $n_h$ data records, represented as $\\mathbf{x}^{S_h} \\in \\mathcal{R}^{n_h \\times (c + m_h)}$, where $c$ is the number of common features shared by all parties, and $m_h$ refers to the unique features held by each party $S_h$.\n\n- **Identifier**: For each data instance $x_i^{S_h} \\in \\mathbf{x}^{S_h}$, the $c$ common features $k_i^{S_h} \\in \\mathcal{R}^{1 \\times c}$ are termed the **identifier** or **key** of $x_i^{S_h}$. All identifiers for $S_h$ are denoted as $\\mathbf{k}_i^{S_h} := \\{k_i^{S_h}\\}_{i=1}^{n_h}$.\n\n- **Fuzzy Identifier**: In multi-party VFL, if two parties $S_a$ and $S_b$ have identifiers such that $\\forall k_i \\in \\mathbf{k}_i^{S_a}$ and $\\forall k_j \\in \\mathbf{k}_j^{S_b}$, $k_i \\neq k_j$, then $\\mathbf{k}_i^{S_a}$ and $\\mathbf{k}_j^{S_b}$ are **fuzzy identifiers**.\n\n- **Multi-party Fuzzy VFL**: A VFL scenario involving parties with fuzzy identifiers.\n\n**Importance of Fuzzy Identifiers**: The critical role of linkage quality in VFL performance has been shown both empirically [1] and theoretically [2]. This impact is even more pronounced in multi-party settings with increasing noise, as demonstrated by our significant improvement over FedSim in such scenarios.\n\n**Prevalence of Multi-party Fuzzy VFL in the Real World**: Traditional VFL assumes that each data record represents a **user**, making the assumption of a universal user ID natural. However, in real-world applications, these records often represent **various real-world objects** - such as a room (e.g., `house` and `hdb` datasets) or a travel route (e.g., `taxi` dataset). Expecting all real-world objects to have a universal, precise ID is unrealistic, which is why case studies in [1] found that over 70% of applications cannot be linked exactly.\n\n\nW2. Privacy-Preserving Record Linkage (PPRL) is a linkage algorithm that calculates identifier similarities but lacks learning mechanisms, serving as a preprocessing step for most VFL algorithms. As such, PPRL is not directly comparable to VFL approaches like FeT.\n\nAn extension of PPRL, known as Top1Sim [3,4], performs VFL on the most similar records identified by PPRL. We have thoroughly compared Top1Sim as a baseline. However, this simple extension overlooks significant information, which led to the development of learning-integrated approaches like FedSim [1] and FeT. Notably, Top1Sim incurs a 42% higher RMSE compared to FeT on the `house` dataset.\n\n\nW3. **Efficiency of FeT**: We compared FeT's computational and memory efficiency with FedSim, the state-of-the-art fuzzy VFL model, as shown in Table 11 of \"rebuttal.pdf,\" and identified three key findings:\n\n1. **Parameter Efficiency**: FeT has a comparable or even smaller (23%-129%) number of parameters than FedSim, indicating that its performance improvement is due to model design rather than parameter scaling.\n\n2. **Memory Efficiency**: FeT is significantly more memory efficient, consuming only 20-39% of the memory compared to FedSim. However, this efficiency comes at the cost of training speed. FeT performs neighbor search in parallel during real-time training, whereas FedSim spends hours linking top-K neighbors and preloads all linked neighbors into GPU memory, leading to longer linkage times, repeated data records, and higher memory usage.\n\n3. **Overhead of New Components**: As detailed in Table 11 of \"rebuttal.pdf,\" the additional components in FeT - dynamic masking and positional encoding - add minimal extra parameters (1k - 0.4M), causing only a slight computational overhead (0-5 seconds per epoch slowdown).\n\nIn summary, FeT delivers **better performance and improved memory efficiency with a similar number of parameters** compared to FedSim, despite slightly lower training speed. Further optimization techniques, such as pipeline parallelism, could enhance FeT’s training speed but are beyond this study's scope.\n\n**Privacy of FeT**: Figures 5(b, d) and 10 in the paper show FeT's privacy superiority. Since related studies use similar Gaussian noise mechanisms, we compare the noise scale at the same $\\varepsilon$ rather than accuracy. At the same $\\varepsilon$, FeT requires much less noise than RDP, a previous pure DP approach like FedOnce [5], generally leading to better model utility.\n\nMW1. **Real Application of Multi-party Fuzzy VFL**: In Figure 15 of \"rebuttal.pdf,\" we present a real-world application involving travel cost prediction in a city through collaboration among taxi, car, bike, and bus companies. Since personal travel information is private and cannot be shared, VFL is essential. Additionally, route identifiers - starting and ending GPS locations - can only be fuzzily linked, but linking closely related source and destination points with multi-party fuzzy VFL can significantly improve prediction accuracy.\n\nMW2. The 13% improvement comes from comparing FeT to FeT without dynamic masking, as shown in Table 2 of Appendix C.1.\n\nMW3. We will correct this formulation in the revision.\n\n**References**\n\n[1] Wu et al. \"A coupled design of exploiting record similarity for practical vertical federated learning.\" NeurIPS 22.\n\n[2] Nock et al. \"The impact of record linkage on learning from feature partitioned data.\" ICML 21.\n\n[3] Hardy et al. Private federated learning on vertically partitioned data via entity resolution and additively homomorphic encryption. arXiv, 17.\n\n[4]  Nock et al. Entity resolution and federated learning get a federated resolution. arXiv, 18.\n\n[5] Wu et al. \"Practical vertical federated learning with unsupervised representation learning.\" IEEE Transactions on Big Data, 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ulvl9KjBXi", "reviewer": "Reviewer_sPbx", "summary": "This paper introduces the Federated Transformer (FeT) framework, which is designed to support multi-party VFL.\nIt enhances the efficiency of model training among multiple parties using fuzzy identifiers while ensuring data privacy.\nExperiment results show that FeT performed well when it scaled to 50 parties and the authors also provide theoretical proof of the differential privacy.", "questions": "**Questions**\n\n1\\. **Scenario Application**:\n\nHow can FeT be adapted to work with unstructured data types such as text or images? Is there any preliminary results or insights on applying FeT to these types of data?\n\n2\\. **Privacy-Performance Trade-off**:\nCan you provide a more detailed analysis of the trade-off between privacy and performance, especially at different levels of $\\epsilon$?\n\n3\\. **Model Performance on datasets with different features**:\nHow does the performance of the Federated Transformer (FeT) vary when applied to datasets with significantly different feature distributions across multiple parties?", "limitations": "The authors have addressed the limitations of their work by identifying three primary limitations related to the assumptions of common features across parties, the potential accuracy reductions due to stringent privacy safeguards, and the correlation between keys and data representations.\n\nHere are some other potential limitations that may be considered:\n\n**Large scale cases**: The model's performance may have the risk of degrading with a large number of parties or larger datasets, which may limit its application in certain large federated learning scenarios.\n\n**Privacy Trade-offs**: While the model is designed to enhance privacy, there may still be some vulnerabilities to be exploited, especially when the underlying assumptions about data sharing are violated.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "**Strengths of the Paper**\n\n**Originality**\n\nThe biggest novelty of the work is the introduction of the Federated Transformer (FeT) framework. It presents a new aspect of handling vertical federated learning (VFL) with fuzzy identifiers.\n\n- **FeT Structure**: The work introduces the architecture of FeT in detail. The model can be split into two parties: the primary party and the secondary party. It also includes several innovative techniques including Dynamic Masking, Party Dropout, and Positional Encoding Averaging.\n\n**Experiment**\n\nThe authors compare the performance of the Federated Transformer (FeT) against multiple baseline methods which is a robust benchmarking. The experimental results demonstrate that FeT significantly outperforms baseline models, achieving improvements of up to 46 percentage points when scaled to 50 parties, which is a significant performance gain.\n\n**Significance and Future Impact**\nFeT is suitable for multimodal learning which aligns many federated learning scenarios in real life while protecting individual privacy.", "weaknesses": "**Weaknesses of the Paper**\n\n**Model Design**\n\n1) The introduction of a trainable dynamic masking module aims to improve the exclusion of incorrectly linked data records. However, if this module is not well-optimized, it could introduce additional noise or errors which degrades the model performance.\n\n2) The proposal of positional encoding averaging may not available in all VFL scenarios, which may cause inconsistencies in data processing which may affect the model training.\n\n**Privacy Trade-offs**\n\nThe paper says that stringent privacy safeguards may cause accuracy reductions, particularly with low values of $\\epsilon$. It would be better if the authors could provide a more detailed analysis of the trade-offs between privacy and performance.\n\n**Theoretical Proof**\n\nWhile the paper provides some theoretical proofs, it may not offer a complete set of proofs for all claims made. The theoretical section did not adequately discuss the limitations of the proposed methods or the implications of violating the underlying assumptions. A more thorough exploration of these aspects would enhance the robustness of the theoretical framework.\n\n**Experiment**\n\nThe experiments are conducted on a small number of datasets, which may not represent the full range of scenarios in VFL. Furthermore, The experiments explore little the trade-off between privacy guarantees and model utility, especially at low values of ε in differential privacy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments; we have addressed all concerns as follows.\n\nW1.1 **Design of Dynamic Masking**: Dynamic masking is a simple two-layer MLP that can be easily optimized and performs well across all five datasets. Our ablation study in Table 3 of Appendix C.1 demonstrates that dynamic masking **consistently and significantly improves performance on all five datasets**. For example, on the `MNIST` dataset, omitting dynamic masking results in a 13 percentage point drop in accuracy. To provide further insight, we **visualize the learned dynamic mask in Figure 11** of the attached \"rebuttal.pdf.\" \n\nThis visualization reveals two key observations: First, dynamic masking effectively focuses on a localized area around the identifiers of the primary party. Data records with more distant identifiers on secondary parties (shown in cooler colors) are assigned small negative mask values, reducing their influence in the attention layers. Second, the focus area varies in scale and direction across different samples, indicating that the dynamic masking layer learns to generate sample-specific masks, which helps to reduce overfitting.\n\nW1.2 **Design of Positional Encoding (PE) Averaging**: This design is specifically applicable to VFL scenarios with independent and identically distributed (i.i.d.) identifiers, similar to the FedAvg approach. VFL with heterogeneous or non-i.i.d. identifiers (e.g., one party using GPS coordinates while another uses postal codes) remains an open problem, posing significant challenges for identifier alignment - an area not yet explored in VFL research. Our ablation study in Table 5 of Appendix C.3 shows that PE averaging improves performance compared to non-PE-averaging FeT (average frequency = 0) in most datasets.\n\nW2 **Privacy Trade-offs**: The trade-off between privacy and performance is illustrated in Figure 5 and Figure 9 of the paper. Both performance (measured by accuracy) and privacy (denoted by $\\varepsilon$) are influenced by the noise scale $\\sigma$. Generally speaking, $\\varepsilon$ reflects the probability bound of determining whether a specific data record exists in the training set based on the representations, regardless of the attack method. Thus, a lower $\\varepsilon$ indicates higher privacy. To achieve high privacy, such as $\\varepsilon = 1$, a noise scale of approximately $\\sigma \\approx 8$ is required, which significantly reduces utility, causing FeT to perform similarly to Solo. Conversely, when $\\varepsilon$ is larger, such as $\\varepsilon = 8$, the required noise scale decreases to $\\sigma = 1$, which allows FeT to maintain relatively good performance.\n\nW3. **Theoretical Proof**: The full proof of Theorem 3 is provided in Appendix A. Background theorems and definitions of differential privacy are covered in Section 2, while the threat model is detailed in Section 4. Since Theorem 3 is based on differential privacy, its assumptions inherit the assumptions of differential privacy.\n\nW4 **Experiments on More Scenarios of VFL**: We have added experiments using the VertiBench [1] datasets to cover a broader range of VFL scenarios, as detailed in the attached \"rebuttal.pdf.\" The results demonstrate that: 1) FeT outperforms baselines across varying levels of imbalance between parties, and 2) FeT shows better performance in VFL scenarios where parties have more balanced features.\n\nQ1. **Scenario Application**: Images and texts can be effectively handled by transformer structures, as demonstrated in many existing works [2]. Thus, adapting FeT to other unstructured data should not pose a significant challenge. The two main ideas - 1) encoding common features as positions and 2) dynamic masking -are particularly useful for multimodal alignment in these scenarios. The primary challenge, however, lies in the significant heterogeneity of features across parties. As shown in Figure 13 of the \"rebuttal.pdf,\" although FeT outperforms baselines, its performance on imbalanced features requires further improvement. We are actively working on addressing this issue.\n\nQ2. Please refer to W2. Privacy Tradeoffs.\n\nQ3. **Significant Heterogeneous Features** We have conducted additional experiments using the VertiBench [1] datasets with varying heterogeneity (balance level $\\alpha$) across parties, as shown in Figure 13 in the attached \"rebuttal.pdf.\" The results demonstrate that while FeT's absolute performance decreases under severely heterogeneous feature distributions (very low $\\alpha$), its improvement over Solo training becomes more pronounced in such heterogeneous scenarios.\n\n\nL1. **Large Scale Cases**: Unlike horizontal federated learning, Vertical Federated Learning (VFL) is primarily observed in cross-silo scenarios [3] rather than cross-device scenarios [4], as it is unlikely that a single user would have features distributed across thousands or even millions of distinct parties. This observation is supported by our communication with industry collaborators that provide commercial VFL service. Therefore, we focus on the cross-silo case, where collaborations among dozens or hundreds of parties are more common.\n\nL2. **Privacy Trade-offs**: We agree that malicious parties, rather than our assumed honest-but-curious parties, would present a greater privacy risk to the model design. We will acknowledge this limitation and plan to explore such scenarios in future work.\n\n**References**\n\n[1] Wu et al. \"VertiBench: Advancing feature distribution diversity in vertical federated learning benchmarks.\" ICLR 24.\n\n[2] Han et al. \"A survey on vision transformer.\" TPAMI 22.\n\n[3] Huang et al. \"Cross-silo federated learning: Challenges and opportunities.\" arXiv 22.\n\n[4] Karimireddy et al. \"Breaking the centralized barrier for cross-device federated learning.\" NeurIPS 21.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uTswYtflmj", "reviewer": "Reviewer_so53", "summary": "This paper investigates vertical federated learning (VFL) linked with fuzzy identifiers and develops a new framework called Federated Transformer (FeT). The proposed framework leverages the transformer architecture to encode the identification information and distributes the subnets of the transformer across parties.", "questions": "1.\tPlease provide more concrete use cases of fuzzy VFL.\n2.\tPlease clarify the key concept in fuzzy VFL.\n3.\tThe design rationale behind the key components in the proposed framework is unclear. Please see my comments above and articulate them in more detail.", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1.\tThe paper tackles the important topic of achieving multi-party fuzzy VFL with both promising performance and robust privacy.\n2.\tThe design of using a transformer to encode fuzzy identifiers is quite novel.\n3.\tThe code is publicly available.", "weaknesses": "1.\tThe paper is not well motivated. It would be great if the paper could provide concrete use cases of fuzzy VFL, that involves fuzzy data/linkages while requiring the privacy needs of distributed machine learning. The German Record Linkage Center might not be an appropriate example for federated learning.\n2.\tSeveral concepts in fuzzy VFL need further clarification. For example, what strategies are used in existing fuzzy linkage? How are the fuzzy identifiers presented? How is privacy preserved in fuzzy identifiers?\n3.\tThe design rationale behind dynamic masking is unclear. Why is dynamic masking necessary? How does it function? \n4.\tThe novelty of party dropout needs further elaboration. Dropout is widely used in federated learning. How does the proposed approach differ from existing designs? \n5.\tIt is unclear why the proposed framework employs two different privacy mechanisms: differential privacy and MPC. What unique challenge is each privacy mechanism designed to address?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments; we have addressed all concerns as follows.\n\nW1, L1. **Real Application of Multi-party Fuzzy VFL**: In Figure 15 of \"rebuttal.pdf,\" we present a real-world application involving travel cost prediction in a city through collaboration among taxi, car, bike, and bus companies. Since personal travel information is private and cannot be shared, VFL is essential. Additionally, route identifiers - starting and ending GPS locations - can only be fuzzily linked, but linking closely related source and destination points with multi-party fuzzy VFL can greatly improve prediction accuracy.\n\n**German Record Linkage Center (GRLC) and VFL**: Previous VFL studies [1-3] show that record linkage is a necessary preprocessing step for VFL, making it inherently involved in all real VFL applications. While VFL studies are relatively new, record linkage has been extensively studied and widely applied. Thus, our study, along with [1], considers GRLC applications to be reflective of VFL applications.\n\n**Prevalence of Multi-party Fuzzy VFL in the Real World**: Traditional VFL assumes that each data record represents a **user**, making the assumption of a universal user ID natural. However, in real-world applications, these records often represent **various real-world objects** - such as a room (e.g., `house` and `hdb` datasets) or a travel route (e.g., `taxi` dataset). Expecting all real-world objects to have a universal, precise ID is unrealistic, which is why case studies in [1] found that over 70% of applications cannot be linked exactly.\n\nW2, L2. We adopt privacy-preserving record linkage (PPRL) [4] for similarity calculation. PPRL is a well-studied area separate from VFL, and we do not impose constraints on specific PPRL approaches, including linkage strategies, representation of fuzzy identifiers, or privacy mechanisms. For example, FEDERAL [5], a PPRL method, transforms fuzzy identifiers into Bloom filters that offer provable privacy guarantees. These Bloom filters' similarities reflect the original fuzzy identifiers' similarities. Our approach, like other fuzzy VFL algorithms [1-3], operates on these calculated similarities. While changing the PPRL method might impact the quality of the similarity measures, it does not affect FeT's superiority compared to other VFL algorithms.\n\nW3, L3. **Design Rationale**: Dynamic masking filters out unrelated data records with low similarities before they reach deep attention layers. FeT feeds the top-K similar records into the transformer, but this can introduce many unrelated records, leading to overfitting. To prevent this, a simple MLP creates a mask based on current identifiers, allowing the attention layers to focus on a narrower neighborhood and reducing overfitting.\n\n**Why Dynamic Masking is Necessary**: In our ablation study (Table 3 of Appendix C.1), removing dynamic masking (FeT w/o DM) results in significant performance loss across all five datasets. For example, on `MNIST`, training without dynamic masking led to a 13 percentage point drop in accuracy, demonstrating its necessity.\n\n**How Dynamic Masking Functions**: The dynamic masking module, a simple two-layer MLP, takes identifiers as input and outputs a mask added to the attention keys (used in `torch.nn.MultiHeadAttention` as `key_padding_mask`). The learned dynamic masks, visualized in Figure 11 of \"rebuttal.pdf,\" reveal two key observations:\n- Dynamic masking effectively focuses on a localized area around the primary party's identifiers without accessing them directly. Records with distant identifiers on secondary parties (shown in cool colors) receive small negative mask values, reducing their significance in the attention layers.\n- The focus area varies in scale and direction across samples, indicating that the dynamic masking layer generates sample-specific masks to reduce overfitting.\n\nW4. Party dropout is a novel feature in VFL, distinct from traditional Dropout, introduced alongside our split-sum neural network design. **Almost all previous VFL** models [1-3,6] use a split-concat design, where representations are **concatenated** at the cut layer and passed to the aggregation model, requiring all parties to be present in each training step. In contrast, we use secure multi-party computation to **average** the representations, allowing some parties to be absent. This enables party dropout, which **disables training of entire encoders** of some parties in each step, rather than specific layers. Combined with SplitAvg, party dropout significantly reduces communication costs in multi-party VFL by the dropout ratio.\n\nW5. Our primary goal is to protect representations from secondary parties against an honest-but-curious primary party. We achieve this by applying differential privacy, adding noise to the representations. In VFL, unlike horizontal FL, data across parties are **related**. Thus, the primary party gains **access to more related representations as more parties join**. This typically requires adding more noise to maintain the same level of differential privacy, thus reducing utility. To address this, we use secure multi-party computation, letting the primary party know only the sum of the representations, keeping noise levels constant regardless of the number of parties.\n\n**References**\n\n[1] Wu et al. \"A coupled design of exploiting record similarity for practical vertical federated learning.\" NeurIPS 22.\n\n[2] Nock et al. \"The impact of record linkage on learning from feature partitioned data.\" ICML 21.\n\n[3]  Nock et al. Entity resolution and federated learning get a federated resolution. arXiv 18.\n\n[4] Vatsalan et al. \"Privacy-preserving record linkage for big data: Current approaches and research challenges.\" Handbook of big data technologies, 17.\n\n[5] Karapiperis et al. \"FEDERAL: A framework for distance-aware privacy-preserving record linkage.\" TKDE 18.\n\n[6] Liu, et al. \"Vertical federated learning: Concepts, advances, and challenges.\" TKDE 24.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dA7fzDeiZk", "reviewer": "Reviewer_P7Jg", "summary": "The paper introduces the Federated Transformer (FeT), a novel framework for Vertical Federated Learning (VFL) that addresses the challenges of fuzzy identifiers in multi-party scenarios. It incorporates three key innovations to enhance performance, privacy, and reduce communication overhead: positional encoding averaging, dynamic masking, and a party dropout strategy. The approach significantly outperforms baseline models in scalability and privacy preservation.", "questions": "Please see Weakness and Limitations.", "limitations": "1.I think you can give us a better case diagram to help us better understand your new task.\n2.I think there is an update error in line 16 of your algorithm.\n3.In the Party Dropout section of the article, around line 188, you said that the communication overhead on the primary party can be reduced by up to 80%, but I did not find the corresponding experiment or proof in the article. I think this part needs to be better explained.\n4.The experimental settings focus on synthetic and controlled environments. Real-world applications might introduce variables not accounted for in this study, potentially affecting the generalizability of the results.\n5.The complexity of the model and the required computational resources are not thoroughly discussed, which could be a limitation for practical deployment in resource-constrained environments.\n6.While the model addresses fuzzy identifiers, the performance heavily relies on the quality of the linkage, which may not always be feasible or accurate in real-world scenarios.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The integration of a transformer architecture for managing fuzzy identifiers in multi-party VFL is innovative. The dynamic masking and party dropout strategies are creative solutions that enhance scalability and reduce communication costs.\n2. The proposed model is well-articulated, with rigorous experimental validation that demonstrates substantial improvements over existing methods. The integration of differential privacy and secure multi-party computation strengthens the privacy aspect.\n3.The paper is well-organized with clear explanations of the problems, proposed solutions, and results. It is accessible to readers with a background in federated learning.\n4.The results show significant improvements in both performance and privacy, making it a valuable contribution to the field of federated learning, especially in applications involving sensitive data across multiple parties.", "weaknesses": "1.The performance improvements reported are impressive; however, how dependent are these improvements on the initial conditions of data alignment and the distribution of fuzzy identifiers? Could the authors provide insights on the robustness of the Federated Transformer (FeT) under less ideal conditions?\n2. While the paper discusses scalability extensively, there is less focus on the computational resources required. Could the authors comment on the computational overhead and the practicality of deploying FeT in real-world scenarios with potentially limited resources?\n3.The FeT introduces several complex mechanisms such as dynamic masking and positional encoding averaging. How do these additions affect the training time and complexity of the model? Is there a significant trade-off between performance and efficiency?\n4.The experiments are conducted primarily on synthetic datasets. How well does FeT generalize to other real-world datasets, particularly those with higher levels of noise and less structured data?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments; we have addressed all concerns below.\n\nW1, L6. **How FeT Relies on Linkage Quality**: FeT is significantly **less reliant** on initial linkage quality than traditional VFL models like Top1Sim [4,5]. While Top1Sim trains only on records linked by privacy-preserving record linkage (PPRL) [3], FeT requires PPRL only to **approximate the K-nearest neighbors** and trains directly on these records, without needing them ordered by distance. Evidence for FeT's robustness includes:\n- **Ablation Study on K** (Figure 14, \"rebuttal.pdf\"): FeT consistently outperforms baselines with many unrelated data records included (K > 50), showing resilience to low linkage quality.\n- **Performance with Different Fuzziness** (Figure 12, \"rebuttal.pdf\"): FeT outperforms baselines with moderate to high noise in identifiers, demonstrating strong resilience to fuzziness.\n- **Dynamic Masking Visualization** (Figure 11, \"rebuttal.pdf\"): The visualization demonstrates that dynamic masking plays a crucial role in handling low-quality linkage, effectively focusing on a few close and relevant records among 4,900 fuzzily linked ones.\n\n**Performance of FeT in \"Less Ideal\" Cases**: FeT performs better in **general fuzzy VFL scenarios** but slightly underperforms Top1Sim [4,5] in **ideal cases with precisely matched identifiers**. This limitation is discussed in Appendix D with experiments on VFL datasets with randomly assigned precise IDs. Moreover, as shown in Figure 12 of \"rebuttal.pdf,\" FeT only slightly underperforms compared to Top1Sim in low-noise conditions of `gisette` dataset.\n\nW2, W3, L5. **Why Focus on Performance Over Efficiency?**: In multi-party fuzzy VFL, the main challenge is fuzzy VFL models like FedSim [2] **perform worse than Solo training** in multi-party setting, instead of computational efficiency. FeT, as the first effective approach for multi-party fuzzy VFL, prioritizes performance to address this critical gap. For clarity, we will change the title to \"_Federated Transformer: **Multi-Party** Vertical Federated Learning on Practical Fuzzily Linked Data_\" and other related contents.\n\n**Efficiency of FeT**: We compared FeT's computational and memory efficiency with FedSim [2], the state-of-the-art fuzzy VFL model, as shown in Table 11 of \"rebuttal.pdf,\" and identified three key findings:\n\n1. **Parameter Efficiency**: FeT has a comparable or even smaller (23%-129%) number of parameters than FedSim, indicating that its performance improvement is due to model design rather than parameter scaling.\n\n2. **Memory Efficiency**: FeT is significantly more memory efficient, consuming only 20-39% of the memory compared to FedSim. However, this efficiency comes at the cost of training speed. FeT performs neighbor search in parallel during real-time training, whereas FedSim spends hours linking top-K neighbors and preloads all linked neighbors into GPU memory, leading to longer linkage times, repeated data records, and higher memory usage.\n\n3. **Overhead of New Components**: As detailed in Table 11 of \"rebuttal.pdf,\" the additional components in FeT - dynamic masking and positional encoding - add minimal extra parameters (1k - 0.4M), causing only a slight computational overhead (0-5 seconds per epoch slowdown).\n\nIn summary, FeT delivers **better performance and improved memory efficiency with a similar number of parameters** compared to FedSim, despite slightly lower training speed. Further optimization techniques, such as pipeline parallelism, could enhance FeT’s training speed but are beyond this study's scope.\n\n**VFL with Limited Resources**:\n1. **FeT's Limited GPU Memory Performance**: Table 11 (\"rebuttal.pdf\") shows that FeT performs well even with GPU memory under 1 GB, making it viable for resource-constrained VFL scenarios.\n\n2. **Cross-Silo VFL Prevalence**: Cross-device VFL with limited resources is rare. VFL typically occurs in cross-silo settings [3], involving dozens to hundreds of parties with adequate computational power. Discussions with industry collaborators providing commercial VFL services confirm that it's uncommon for a single user to have features spread across thousands of distinct parties.\n\nW4, L4. Three of our datasets (`house`, `taxi`, `hdb`) are real-world VFL datasets, identical to those used in FedSim [2]. Each party's data comes from different real sources; for example, the `taxi` dataset includes data from New York taxis and CitiBike. These datasets effectively demonstrate FeT's performance in real VFL applications.\n\nL1.  In Figure 15 of \"rebuttal.pdf,\" we present a real-world application involving travel cost prediction in a city through collaboration among taxi, car, bike, and bus companies. Since personal travel information is private and cannot be shared, VFL is essential. Additionally, route identifiers - starting and ending GPS locations - can only be fuzzily linked, but linking closely related source and destination points with multi-party fuzzy VFL can significantly improve prediction accuracy.\n\nL2. We have fixed this typo in the revision.\n\nL3. The 80% communication reduction is straightforward from calculations, as the communication reduction is nearly proportional to the party dropout rate. Dropped parties don't participate in gradient and representation exchanges, which account for most communication in VFL.\n\n**References**\n\n[1] Li, et al. \"Learnable fourier features for multi-dimensional spatial positional encoding.\" NeurIPS 21.\n\n[2] Wu et al. \"A coupled design of exploiting record similarity for practical vertical federated learning.\" NeurIPS 22.\n\n[3] Vatsalan et al. \"Privacy-preserving record linkage for big data: Current approaches and research challenges.\" Handbook of big data technologies, 17.\n\n[4] Hardy et al. Private federated learning on vertically partitioned data via entity resolution and additively homomorphic encryption. arXiv 17.\n\n[5] Nock et al. Entity resolution and federated learning get a federated resolution. arXiv, 18.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank all the reviewers for their thoughtful efforts in reviewing our manuscript and providing valuable feedback. In response, we have made substantial revisions and added new experiments, visualizations, and real-world applications to the "rebuttal.pdf," including Figures 11-15 and Table 11. Below is a summary of the contents and key findings for each figure and table:

- **Figure 11**: Visualization of learned dynamic masks for different samples. Each figure displays one sample (red star) from the primary party, fuzzily linked with 4,900 samples (circles) from 49 secondary parties. The position indicates the sample's identifier, and colors reflect the learned dynamic mask values, with larger mask values indicating higher importance in the attention layers.
  - **Findings**:
    1. Dynamic masking effectively focuses on a localized area around the primary party's identifiers. Data records with distant identifiers on secondary parties (shown in cooler colors) receive small negative mask values, reducing their significance in the attention layers - without accessing the primary party's original identifiers.
    2. The focus area varies in scale and direction across samples: for example, the left figure concentrates on a small bottom area, the middle figure on a small top area, and the right figure on a broad area in all directions.

- **Figure 12**: Analysis of the effect of identifier fuzziness, showing the scale of Gaussian noise added to precisely matched identifiers.
  - **Findings**: FeT consistently outperforms baselines at moderate fuzzy scales. When identifiers are very noisy, FeT's performance approaches that of Solo. Conversely, when identifiers are highly accurate, FeT's performance approaches that of Top1Sim.

- **Figure 13**: Performance on VertiBench MNIST datasets with varying levels of imbalance ($\alpha \in [0.1, 50]$). A larger $\alpha$ indicates a more balanced feature distribution.
  - **Findings**: Both FeT and baseline models show improved performance in more balanced scenarios. FeT consistently demonstrates better accuracy than baselines across varying levels of heterogeneity.

- **Table 11**: Comparison of training efficiency on an RTX3090 GPU (batch size 128). PE: positional encoding; DM: dynamic masking.
  - **Findings**:
    1. **Parameter Efficiency**: FeT has a comparable or even smaller number of parameters than FedSim, suggesting that its performance improvement is due to model design rather than an increase in parameters.
    2. **Memory Efficiency**: FeT is significantly more memory efficient than FedSim, consuming only 20-39% of the memory. However, this efficiency comes at the cost of training speed, as FeT performs neighbor search in parallel during real-time training. In contrast, FedSim requires hours to link top-K neighbors and preloads all linked neighbors into GPU memory, resulting in longer linkage times and higher memory usage.
    3. **Overhead of New Components**: The additional components in FeT - dynamic masking and positional encoding - add very few extra parameters (1k - 0.4M), resulting in negligible additional computational cost (0-5 seconds per epoch slowdown).

- **Figure 14**: Impact of varying the number of neighbors ($K$) on FeT performance.
  - **Findings**: FeT consistently outperforms all baselines, even when $K > 50$, which involves many unrelated data records in the linkage process. When $K$ is small, the performace of FeT may decrease for the lack of infomation, the similar reason for the low performanae of Top1Sim. This demonstrates the ablity of FeT to filter out unrelated records in large $K$ scenarios.

- **Figure 15**: Real-world application of fuzzy multi-party VFL for travel cost prediction in a city.
  - **Application**: This scenario involves predicting travel costs through collaboration among companies managing taxis, cars, bikes, and buses. Since personal travel information is private and cannot be shared across companies, the identifiers for each route - starting and ending GPS locations - can only be linked fuzzily. Nevertheless, linking routes with closely related source and destination points significantly improves prediction accuracy.

In the detailed response to each reviewer, we use abbreviation to refer to each comment due to the charater contraint. Specifically,
- Wi: Weakness i
- MWi: Minor Weakness i
- Qi: Question i
- Li: Limitation i

Some concerns in the review are tightly related (e.g., Weakness 1 and Limitation 6), thus we have combined them (e.g., W1, L6) in our response.

In our response to each reviewer, we believe we have addressed all concerns and would be grateful if you could consider adjusting your rating if you find our revisions satisfactory.

Meta-review: This paper tackles the problem of multi-party VFL for fuzzily linked data (i.e., where the matching of samples across VFL clients is imperfect).
While multi-party or fuzzy VFL have been researched in prior work, the combination of the two has not.
The paper argues the multi-party problem introduces unique challenges such as more complex fuzzy linkage and higher privacy loss and seeks to mitigate these issues through several strategies including positional encoding averaging, dynamic masking of incorrect pairs, and SplitAvg that combines encryption and noise-based approaches to privacy.

The reviewers were generally positive about the paper after better understanding the problem and motivation. Clarification regarding the practical applicability and context for multi-party fuzzily linked data was noted by multiple reviewers and should be included in the final paper. Additionally, the authors should change the title to include "multi-party" as they suggested and also incorporate this into the paper more carefully to avoid overclaiming their contributions.

While there were some concerns regarding computational issues or experimental conditions, the real-world VFL datasets and author response reasonably addressed these concerns. I would recommend incorporating these discussions into the final paper.

In summary, this paper provides an interesting and effective solution to the multi-party fuzzy linked problem using a distributed transformer architecture that has practical real-world relevance for VFL scenarios.
Decision: Accept (poster)