VFLAIR: A Research Library and Benchmark for Vertical Federated Learning
Authors: Tianyuan Zou
Zixuan GU
Yu He
Hideaki Takahashi
Yang Liu
Ya-Qin Zhang
Venue: ICLR 2024 poster (2024)
Keywords: VFL
Attacks and Defenses
Benchmark
Platform
Primary area: datasets and benchmarks

Abstract: Vertical Federated Learning (VFL) has emerged as a collaborative training paradigm that allows participants with different features of the same group of users to accomplish cooperative training without exposing their raw data or model parameters. VFL has gained significant attention for its research potential and real-world applications in recent years, but still faces substantial challenges, such as in defending various kinds of data inference and backdoor attacks. Moreover, most of existing VFL projects are industry-facing and not easily used for keeping track of the current research progress. To address this need, we present an extensible and lightweight VFL framework VFLAIR (available at https://github.com/FLAIR-THU/VFLAIR), which supports VFL training with a variety of models, datasets and protocols, along with standardized modules for comprehensive evaluations of attacks and defense strategies. We also benchmark $11$ attacks and $8$ defenses performance under different communication and model partition settings and draw concrete insights and recommendations on the choice of defense strategies for different practical VFL deployment scenarios.

Published as a conference paper at ICLR 2024
VFLAIR: A RESEARCH LIBRARY AND BENCHMARK FOR VERTICAL FEDERATED LEARNING
Tianyuan Zou', Zixuan Gu?, Yu He", Hideaki Takahashi', Yang Lis. and Ya-Qin Zhang'
Institute for Industry Research, Tsinghua University, Beijing, China
"eeiynng College. Tsinghua University, Beijing, China School Computer Science, Fudan University, Shanghai, China Coolleee of Arts and Sciences, The University of Tokyo, Tokyo, Japan Shanghai Artificial Intelligence Laboratory, China
ABSTRACT
Vertical Federated Learning (VFL) has emerged as collaborative training paradigm that allows participants with different features of the same group of users accomplish cooperative training without exposing their raw data model parameters. VFL has gained significant attention for its research potential and real-world applications in recent years, but still faces substantial challenges, such as in defending various kinds of data inference and backdoor attacks. More- over, most of existing VFL projects are industry-facing and not easily used for keeping track of the current research progress. To address this need, we present an extensible and lightweight VFL framework VFLAIR (available at THU /VF///FLLL VFI).R). which supports VFL train- ing with a variety of models, datasets and protocols, along with standardized modules for comprehensive evaluations of attacks and defense strategies. We also benchmark 11 attacks and 8 defenses performance under different communication and model partition settings and draw concrete insights and recommendations on the choice of defense strategies for different practical VFI. deployment scenarios.
hattp:://ittuu.
INTRODUCTION
The concept of Federated Learning (FL) was first introduced by Google in 2016 (McMahan et al. 2016a) describing a cross-device scenario where millions of mobile users collaboratively train shared model using their local private data without centralizing these data. This scenario is regarded Horizontal FL (HFL.) (Yang et al. 2019b) as data partitioned by sample. In another type of FL., regarded as Vertical FL. (VFL) (Yang et al.) 2019b). data are partitioned by feature. VFL is often applied in industrial collaborative learning scenarios where each organization controls disjoint features of a common group of users. In VFL, local data and local model are kept private at each participant. Instead, local model outputs and their gradients are transmitted between parties.
VFL has drawn increasing attention from both academic and industry in recent years with hun- dreds of research papers published every year and a number of open-sourced projects released, including FATE (FedAl-maintaal
Liu et al. 2021c). Fedlearner (Byte Dance), PaddleFL (Baidu) Pysyft (Ryffel et al. [0018] Romanini et al. (0221). FedTree (Li et al., 2022c), and FedML (He et al. 2020). Real-world industral cases are also emerged in the field of advertising (Cai) 2020 Lin 2021 and finance (Cheng et al.) 2020) 2022) etc. However, mainstream VFI. projects such as FATE are industrial grade and not designed for keeping up with research advances.
Meanwhile, research interests for VFL have been growing rapidly over the past years, focusing on improving various aspects of VFL protocols, such as communication efficiency (Fu et al. 2021] Liu
al. 2022b Castiglia et al. 2022bb Fu al. 2022c). robustness to attacks (Liu et al. 2021b Cheng
ct al. 2021 Li et al. 2022b] Zou et al.] 2022] 2023 [un et al.] 2022 Yang et al. 2022a), model
utility (Li et al.) 2022 2d Yang et al. 2022b] reng & Yu| 2020 Feng al. 22022), and fair incentive
designs (Liu et al.) 202
et al. 2022). For communication efficiency. methods like decrease
"Corresponds to Yang Liu (liuy03 @ww..tiinghhaaccdu.coo edd.c..

---

Published as a conference paper at ICLR 2024
Evaluation
Recommendation
([Rounds) (MP)
Evallattoo Metrice
turnffrrred infor nation round (Amoontt)
c.g. 'DD has highest
-Eaaaation Thre (Eoac: Tine)
Rainkng
0C5, so la the mout eleciie
Pee lamenca (AP)
Attack
Featuue Recorshuctin (CR2L TEAN
Label Inforence (N6. D6.
Turpeted Blcceoor (LMB]
PMC, AMC)
training
Defense
Acciing ig
-Grad Spenaticaion (F25)
ditonces Comiaatio (dCor)
-Scadient Pertut (GPPr)
(CAE,
Mibal Information Defanse
VFL Setting
Credi, Adi Breact Cancer
Nurnery
Architclure
Modell
Loginric Ragreankar
Xiltuan,, Raardo
Model Partiion
Commu- Protocal
Enery- pion
Citto, Avazu
Foreet
FedilDD
-LISAADDE
CIFARI0&100
CAN LeNeet Rodher
-GO4 -LTTM
CELU-VLL
plon
Cora
Figure 1: An overview of the Components of VFLAIR.
communication rounds using multiple local updates between each round (Liu al. 2022b Fu et al.
2022c) or compress information (Castiglia et al. 2022b) have been proposed. As for data security and privacy, various attacks injected by one multiple parties aiming to either steal other parties' private label et al. 2022b Fuu et al. 2022a Zou et al.] (022), private features (Jin et al.] 202T Luo et al.
2021 Li et al., 2022a Jiang et al. 2022a Ye et al. (022b), sensitive attributes (Song &Shmatikov
2020) and sample relations (Qiu et al.) 2022).
negatively impact the model behavior (Liu et al.
20216 Zou et al. (2022) have been put forward. Multiple defending methods have also been proposed
to tackle these threats, including adding noise (Dwork) 2006) /oo al. 2022 Li al. 2022b).
sparsifying gradients (Aji Heafield 2017 et et. 2002a] /ou et al.22202 discreting gradients (Fu
al. 2022a). label differential privacy (Ghazi al.] [021] Yang al.) 2022a), adding distance
correlation regularizor (Sun et al. 2022 Vepakomma et al. 2019). disguising labels (Zou et al.
2022), adding mutual information regularizer (Zou al. 2023). adversarial training (Sun al.
2021b; et 2022a) or performing robust feature recovery (Liu et al.) 2021b). However, each of these defenses are evaluated under specific tasks and settings, lacking of key insights and metries on evaluating these defense strategies defend all possible attacks in practical deployment.
To facilitate future research for VFL, we introduce a lightweight and comprehensive VFL framework, namely VFLAIR which includes not only basic VFL training and inference for variety of models and settings but also efficiency enhancement techniques and multiple defense methods that mitigate potential threats. Moreover, we perform extensive experiments on combinations of the above settings using multiple datasets provide different perspectives on VFL efficiency and safety. We believe VFLAIR and these benchmark results will provide researchers with useful tools and guidance for their future work. Our contributions summarized in the following:
(1). We design VFLAIR, a lightweight and extensible VFL framework that aims to facilitate research development of VFL (see Fig. D]. We design standardized pipelines for VFL training and validation, supporting 13 datasets, 29 different local model architectures including linear regression, tree and neural networks, 6 different global models, 2 model partition settings, 5 communication protocols, 1 encryption method, 11 attacks and 8 defense methods, each implemented as a distinct module and can be easily extended.
(2). We propose evaluation metrics and modules, and perform extensive experiments to bench- mark various perspectives of VFL, from which we draw key insights on VFL system design choice, in order to promote future development and practical deployment of VFL.
2 RELATED WORK
number of open-source FL projects have been developed supporting VFL. FATE (FedAI maintainers] Liu et al.) 2021c) lc) is an irdustry-grade FL project which supports a variety of model architectures and secure computation protocols: Fedlearner (ByteDance) is specialized in advertising scenarios; PaddleFI. (Baidu) supports 2-party and 3-party VFL. with MPC protection; Pysyft (Ryffel et al. 2018 Romanini al. 2021) introduces Verticaal which focus SplitNN-type of VFL.
2

---

Published as a conference paper at ICLR 2024
settings; FedTree (Li et al.
2022c focuses tree-based VFL only; FedML (He et al. 2020 supports
basic training of VFL with logistic regression models. Real-world industrial applications have been
witnessed in domains such as advertising (Cail 2020: Lin 2021) and finance (Cheng et al. 2020
2022). These works demonstrate the widespread interest and the practical significance of VFL. How- ever, these works are often relatively heavy-weight as they are designed for industrial deployment. On the other hand, most existing benchmarks on FL. focus on HFL. scenario (Chai et al. 2020 Lai et al.
2022; 2022nZZhagg et al. 2023). For VFI. (Kang et al. 2022) evaluates several defense strategies for data reconstruction attacks; SLPerf (Zhou et al. 2023) focuses benchmarking and comparing various kinds of splitNN scenarios like VpLitVFL. No existing work provides a comprchensive evaluation covering variety of key aspects of VFL settings, including model performance, communication efficiency and robustness to attacks. Due to space limitation, we discuss works on VFL definition and emerging fields of research interest in Appendix A
3 VFL FRAMEWORK
In a typical VFL setting with K parties, each party owns their local private feature (X&) and local model {Gx) 1k=1 with parameters (0x ) respectively. Only one party controls the private label information and is referred to as active party while other parties are referred to as passive parties. The active party also controls a global trainable model parameterized by yp (splitVFL) or global non- non-trainable function F (agg VFL) to aggregate each party's local model output. Note in tree-based VFL the global function is an aggregation function that identifies the optimal feature split based on feature splitting information received from all parties. Without loss of generality, we assume that the 1 party is the active party while other K - parties are passive parties.
In the collaborative training process of NN-based VFL, each party computes its local feature em- bedding H₁ = (Xx, 0k), k = I,.... K. The active party collects {Hx K prediction Y = F(H1....
k=1 and gets the final HK. .p). The loss L e(Y, Y) is calculated at the active party. The ac
......,,
gradient w.r.s. to H4 as gk = 2H, A 1..... K are then calculated and transmitted back to each party by the active party. Using these gradients, each party performs local model updates by SGD Dc aHe THE ae,-"
using Vo. ac
h 1..... K. Also the active party performs model update with
SGD on global model I' if trainable using v aØ In the inference procedure, the same is done but without the backward gradient descent to get the prediction of labels. If the exchange of Hş and 9k is performed each round, such VFL protocol is referred as FedSGD protocol. On the other hand, if communication is done every Q >>> 1 steps of local updates, such protocol is referred as FedBCD (Liu et al. 2022b). The training procedure is shown in detail in Algorithm [ in Appendix B Training and inference procedures of tree-based VFL are included in Algorithm Appendix I
4 OVERVIEW OF VFLAIR
Implemented Components. An overview of the components of VF shown in Fig. VFLAIR incorporates not only basic VFL training and testing process for both NN-based and tree- based VFL of various settings, but also multiple existing efficiency enhancement techniques, data leakage and model utility impairing attacks as well as defending methods that aim to mitigate potential threats. VFLA provides support for both VFL. and splitVFL. with easily adjustable model architectures. Currently, VF VFLAIR supports 5 communication protocols to improve communication efficiency. Also, 11 existing attacks and 8 defenses supported. Moreover, VFLAIR supports the comprehensive assessment of defense performance using carefully designed metrics (see Sec. (5.3). based on which defense strategy recommendations can be provided. Paillier Encryption (Cheng et al.) (021) is also supported to further protect transmitted results. In total, 13 datasets from a diverse range of industrial domains, including but not limited to medical, financial, and recommendation are supported. Datasets are partitioned into either balanced or unbalanced subsets to mimic real-world participants based on human knowledge (see Tab. 10 in Appendix H.I for a full list of dataset and partition methods).
How to use and extend. VFLAIR is light- and comprehensive VFL framework that can be launched on a single GPU or CPU (see Tab. its system requirement compared to FATE) VFLAIR facilitates the easy integration of different datasets for model training and inference through simple dataset loading and partitioning functions. New attacks and defenses can be quickly incorporated into the framework thanks to the modular structure. Step-by-step guidance for using and extending VFLAIR are included in Appendix C The workflow of VFLAIR is also shown in Appendix D
3

---

Published as a conference paper at ICLR 2024
Table 1: Summary of attacks for NN-based VFL
Requirements /Limitations
Attacker
Attack Type
Artock
Pany
Attack Penformance (AP)
Noms- Scoting (NS) [L (1022)
binary classification, sample-level sample-level
AUC of inferred labels
Direction-buse busell Scoring (IXS Li 20220
Lahel
Direct Lahel Interence (DLI) (L] et al.
2022h Zou el nl. R022
Inference
Batch-lovel Label
(BLI)
2022
passive
ratio of correctly inferred labels
(LD
Passive Model Completion (
ct
auxtitrry
Active Model Completion (AMC) (P) of 2irees
data for each class
Feature
Generative Rogression Natark (GRNN Land
black-box white-box, auxiliary i.d. data
Recenstracton (FR)
Traininghhsool
Mapping by
active
MSR(Uo, Uvee)
model inversion (TBM) aLl 20225
Targosed Backdooe (TB)
Label Replacement Rackdoer (1 (IRB) (Znn NUM
1 sample of target closs
passive
ratio of triggered samples
inferred target class
Non-turgeled
Noisy-sample Backcloor (NSB)) (555)//ooe 2023
MP difference between total
Backdoor (NTB)
Missing Feature (MF)
Liu d
passive
and noisy/missing samples
5 VFL BENCHMARK
5.1
SETTINGS MODELS AND DATASETS
Using VFLAIR, We benchmark the VFI. main task performance using 13 datasets including
MNIST (Yann LeCun). CIFARIO (Krizhevsky &Hinton] 2009), CIFAR (Krizhevsky &in- ton 2009), NUSWIDE (Chua et al. (009), Breast Cancer (Street et al., 1993). Diabetes (Kahn), Adult Income (Becker &Koaavi| 1996). Criteo (Guo al. 2017). Avazu (Qu al. 2018), Cora (McCallum)
ct al. 2000), News20 (Lang 1995). Credit (Dua &Graff 2017) and Nursery (Dua &Graff] 2017). Detailed data partition strategies are included in Appendix H.. We explore 2 distinct architectures,
namely VFL and splitVFL, and comprehensively benchmark their performance. The local models used for both settings are detailed in Tab. 10 in Appendix H.I For global model F., a global softmax function is applied under agg VFL setting while a I-layer tull-connected model serves as the global model for SLLitVFL setting (except for Cora dataset, for which a I-layer graph convolution layer is applied). Additionally, we investigate the impact of different communication protocols by comparing FedBCD (Liu et al. (022b) (Q 5) and CELU-VFL (Fu et al. 2022c) (Q - 5, W 5), as well as compression mechanisms Quantize (b = 16) (Castiglia et al. (022b) and Top-k = 0.9) (Castiglia ct al. 2022b) to the conventional FedSGD, as discussed in Sec. and further provide insights into the
communication cost reduction achieved by communication efficient protocols, as well as the impact of FedBCD when various attacks and defenses are deployed. We also evaluate the impact of the number of participating parties as well as the type of local model (logistic regression, tree, NN) on the main task performance of VFL. For pe-based VFL. we further benchmark both Random Forrest and XGBoost algorithms. Moreover, for tree-based VFL. we employ Paillier Encryption (Cheng et al. 202T) to protect transmitted information and measure its impact computation efficiency. Details of corresponding datasets and models for tree-based VFL. can be found in Tab. [0]and Appendix H.1 in the appendix.
5.2 ATTACKS AND DEFENSES
We benchmark the performance of 11 attacks with 8 defenses on 3 datasets including MNIST (Yann LeCun), CIFARIO KKrihevssy &Hinton| 2009) and NUSWIDE (Chua et al.) 2009). For these evaluations, we mainly consider a VFL setting with active party and I passive party following original works (Li et al. 20226 Luo et al. 2021 Li et al. 2022a), denoted as party a.p respectively, with each party owning their local feature X X, and local model Ga. Gp. The local model output of the active and passive party are denoted as Ha, Hp respectively. We summarized the evaluated attacks in Tab. [ Note in Tab. I NS and DS attacks can only be applied to binary classification scenarios; "sample-level" indicates that the attack requires gradient information for each sample, whereas "batch-level" means only batch-level gradients information are available; "black-box" indicates that the model is kept private at the party under attack, but can be queried by the attacker and honestly return the output the attacker, whereas "white-box" means the attacker
has access to the model; MSE(U,, Uree) = E[(u) (f) - urec) )1 where un Uree are the fth feature of original input Un and recovered input Uran respectively. LI. FR and NTB attacks are inference time attacks that are launched separately from VFL training procedure while only TB attacks are training time attacks. Detailed descriptions and coper-parameter of all the evaluated attacks are provided in Appendices and H.3) mainly following the original papers. Defense methods are summarized in Tab. 2 with respective hyper-parameters. Detailed descriptions of defense methodologies and implementations are included in Appendices and H.4
(f)
4

---

Published as a conference paper at ICLR 2024
Table 2: Summary of defense methods and tested hyper- hyper-parameter values for NN- MN-based VFL.
Methodology
Hoprrpmmmtter
Hoperpammeter
GPP Duank Jank Te 1022r Zoy L Doser Crcc
add gradiets
prediction
Sirength
0,000, (OMM, 0:1 0:1
Poiter Zour Forcor
sadd máse to gralicats local praliction
DP Strungh
6,0001, 0.00,, 0.1.1.111
08 []) Ammonna
drop gradient elements close to
Sperecictoon Rate
90.3% US 47% (ff, to
Cr Ealte
penuil pralleus with that of ofher cluss
Patarbuio Soonath
0,0001, 0.001, 8.01, 1.1.0.1
60rr (5m al
distaner comrllation regalarsotionn
Regularoeer Smength
(10001) 0.00, 0.3.3
diegnine
Confucon Straggh
DCNL PR
discrsee pradient nutsal inforssatios (MI regulariaaioon
Contussnn Streigth A
0,0.0.00..111
20E (U25)
Regalormer Strength
0.0, Rett, Le"" 10,0000,00,,
5.3 EVALUATION METRICS
Main Task Performance (MP). MP is defined as the final model prediction accuracy on the test dataset, which reveals the utility of the VFL system.
Communication and Computation Efficiency. Number of communication rounds (#Rounds) and the amount of data transferred for each round (Amount) used for measuring communication efficiency. Execution Time (Exec. Time) is used to measure computation efficiency.
Attack Performance (AP). The definition of AP varies with respect to the type of the attack and is summarized in Tab. 1 Definition details can be seen in Appendix E
Defense Capability Score (DCS). Intuitively, an ideal defense should not compromise the utility of the original main task and should thwart the attack completely. Therefore, considering that both AP and MP are key metries to evaluate defenses. We further propose Defense Capability Score (DCS), to directly compare all the defenses under one unified metric. Let df (AP, MP) represents the performance of a defense on an AP-MP graph, then we define its defense capability score (DCS) based on the distance between of to an ideal defense df = (AP", MP*). MP* is the MP of VFL without defense and AP" is set to 0.0 representing the performance of completely incapable attacker. Then, we formulate the definition of DCS as:
DCS
(I)
1 D(df. df*))
I+
2 3(MP MP*)²
1.0
where D(-) is user-defined distance function. Here Euclidean distance with an adjustable trade-off weighting pa- rameter 3. A visualization of DCS on an AP-MP graph with 0.5 can be seen in Fig. 2 A point closer to the bottom- right corner of an AP-MP graph has a higher DCS score in- dicating a better defense capability, consistent with intuition. 8 = 0.5 is used in our experiments.
* Ideal Dalorse
Attack Performance (AP)
0.8
=05.
0.6
72
0.4
0.70
Type-level Defense Capability Score (T-DCS). T-DCS is the DCS score averaged by attack type. Treating all 17 attacks of the same attack type as equally important, we average DCS for each attack i get T-DCS for attack type j:
0.2
DidT. df
0.0
AP
MP
MEc
Task Performance (MP)
1,
T-DCS; -,, Σ DCS,
Figure 2: A visual illustration ex- ample of DCS. The numbers on the contour lines are DCSs calculated with 3 = 0.5.
(2)
Comprehensive Defense Capability Score (C-DCS), C-DCS is comprehensive assessment of the capability of a defense strategy with respect to all kinds of attacks and is a weighted average of T-DCS shown in Eq. 3
C-DCS
Σ T.DCSS with Σ w = 1.0.
JEA
jEA
Weights (wr))e. can be tailored to user preference. In our experiments, we simply use an unbiased weight wi- [A] for each attack type j E A = FL,, FR, TTB}
6 EVALUATION RESULTS
6.1 VFL MAIN TASK PERFORMANCE
We first comprehensively evaluate the impact of various settings on the performance of VFL. Details on training model and training hyper-parameters for the following experiments are included in Tab, T0 (in Appendix H.T and Appendix H.2 respectively.
5

---

Published as a conference paper at ICLR 2024
Table 3: MP under 4 different settings of NN-based VFL. Q when FedBCD is applied. In
"#Rounds" column, the first and second numbers are the communication rounds needed reach the specified MP for FedSGD and FedBCD respectively.
Dataset
aggVFL, FedSGD
Vgg FedBCD
#Rounds
splitVEL, FedSGD
splicVEL., FedBCD
#Rounds
MNIST
0.972-40.001
0,971 +00001
150/113
0.973-00000
0.974=00000
180 143
NUSWIDE
0.887=0.001
0.882:+0.001
60/26
0.888=0.001
0.884=10.001
12
Breast Cancer
0.914=0.033
0.919
5/3
0.025
0.907
5/4
Diabetes
0.755=00.43
0.736:+0.021
15/13
0.766:0.024
0.746-10.039
15/11
Adult Income
0.839=0.006
0.8411:00005
17/15
0.8421:.004
0.842:10.005
30 13
Table 4: MP under 2-party VFL verses MP under 4-party VFL under 4 different settings of NN-based VFI. using FedSGD communication protocol. "#Rounds" has the same meaning as in Tab. 3
Datasct
aggVFL, 2-party
aggVFL, 4-party || splitVEL, 2-party splitVEL, 4-party
MP 0.790-100033
0.747=10.03
0.798
0.762-10.003
CIFARIO
#Rounds
244+-16
205++12
238++14
173+3
0.454 130++11
0.417:100008 124+22
0.423:10.005 125+2
0.382:10.004
CIFAR100
#Rounds
100+1
Table 5: MP and execution time under 2 different types of tree-based VFL.
XGBoost w/ Encryption
Random Forest w/o Encryption
XGBoost w/o Eneryption
Random Forest
Dataset
w/ Encryption
(a.k.a. SecureBoost)
MP
0.816=0.005 138+14
0.816=20.044
0.816#0.005
0.816+0.004
Credit
Exec. Time [s]
366-116
410--10
881-16
MP
0.814
0.890=:0.011
0.884
0.890=10.011
Nursery
Time [s]
29:1-2
69:14
243:-5
1194:121
The SPLitVFL setting yields a comparable or slightly higher MP compared to aggVFL on most datasets, due to the additional trainable layer serving global model, evidenced by results from Tabs. and
Model Partition.
Communication Protocols. As shown in Tab. 3 and Tab. 6 compared to FedSGD, FedBCD and CELU-VFL. exhibit comparable MP across all datasets with fewer communication rounds, supporting their efficacy reducing communication overhead. Quantize and Top-k compress the transmitted data and successfully reduce the communication cost per round, but may result in increase in communication rounds.
Encryption. For tree-based VFL, we consider two models with and without Paillier Encryption
using 512-bit key size in Tab. 5 Note that XGBoost with Paillier Encryption is equivalent to SecureBoost (Cheng et al. 2021). Although MP values are consistent regardless of encryption, the execution time experiences notable increase of 3 to 20 times when encryption is applied due the additional encryption and decryption process.
Number of Participants. Impact of number of participants are shown in Tab. 4 A slightly lower MP achieved using fewer communication rounds as the number of participants increases, demonstrating the increasing challenges brought by multi-party collaboration.
Model Architectures. Tabs. 5 and 13 (in Appendix compare the MP of different model architectures and show that different model architectures result slightly different MP for each dataset, highlighting the importance of selecting best performing model for each dataset.
Real world datasets. Additional results on Criteo (Guo et al., 2017). Avazu (Qu et al. 2018). Cora (McCallum et 2000) and News20 (Lang) 1995) datasets using domain specific models (e.g. Wide& Deep Model (Cheng et al. 2016) for Criteo and Avazu, GNN for Cora) are provided in Tab.
as they are considered for typical VFL applications, such as in recommendation problems.
6.2 ATTACK AND DEFENSE PERFORMANCE
We demonstrate attack and defense results VFL on the AP-MP graph for each attack on MNIST. CIFARIO 10 and NUSWIDE datasets under VFL setting using FedSGD protocol in Figs. 3 II and 12 Each point in the figure represents (MP, AP) pair with the size of markers representing

---

Published as a conference paper at ICLR 2024
Table 6: MP, communication rounds (#Rounds), amount of information exchanged per round
(Amount) under different communication protocols of NN-based VFL. under agg setting. "Total" column is the total amount that equals #Rounds x Amount.
MNIST
NUSWIDE
MP
#Rounds
Amount (MB)
Total (MB)
#Rounds
Amount (MB)
Total (MB)
FodSGD
0.972=0.001
150
0.156
23.438
0.887=0.001
0.039
2.344
FedBCD
0.971=0.001
113
0.156
17.656
0. 882+=0.00
0.039
1,016
Quantize
0.959=0.006
161
0.117
18.867
0. 0.88110.002
0.029
2.754
Top-k
0.968=0.001
150
0.148
22.266
0. 0.887000001
0.037
2.227
CELU-VFL
0.971=0.002
0.156
16.406
0.88010.0011
0.039
0,977
Table 7: Comparison of agg VFL and splitVFL on MP. #Rounds, Amount, total communication cost, Txee for reaching specified MP with 4 real-world datasets of NN-based VFL with FedSGD
communication protocol.
VFL
splitVFL
Dataset
Amountt (MB)
Total (MB)
Arnouni
Toial (MB)
MP
#Rounds
Exce.Time [s]
MP
#Rounds
(MB)
Exce. [8]
Critco
0.715=0.053
2
0.125
0.250
0.190=0.132
0.744=0.001
0.125
0.375
0.234+0.126
Avazu
0.832-0.001
5
0.125
0.625
0.517=0.185
0.832:000011
0.125
1.125
1.203:11.516
Com
0.721=0.004
11
0.145
1.591
0.205=00085
0.7241:0001
13
0.145
1,880
0.2701:0.082
News20-S5
0.882=0.014
57
0.005
0.278
0,430=0.076
0,895=0.013
0.298
0.613+0.269
LI-DS(2-class)
L-DLi(10-class)
U-BLi(10-class)
LI-MM((10-class)
1.0
13
64
C4
02
612
8:0
GZS
035
CBS
EN
ER5
EN0
LI-AM((10-class)
(10-caa))
FR-TBM(10-class)
TB-LRB(10-class)
6043
1134
1125
6.22
0.02
1.000
0.04
6.60
6.91
0.009
114
0100
030
NTB-N5B(100 (50ccaassss
NTB-MF (10-clF))
0.34
0.5
0.4
G-DP
CAE
8:00
GS
0.0
8.32
GPer
w/o defense
066
8:31
dCor
Figure MPs and APs for different attacks under defenses [CIFARI0 dataset, Vgg, FedSGD]
the relative magnitude of the corresponding defense hyper-parameter listed in Tab. Note that although we try to provide comprehensive evaluation for various defenses, we do not force defense onto attacks, meaning that if defense mechanism is designed for mitigating label inference attacks only, do not assess its effectiveness against FR attacks backdoor attacks.
We further rank all the defenses of different limitation, show representative results for NUSWIDE dataset in Tab. 8 Full results are shown Tabs. to 17 in Appendix 1.2 for MNIST, CIFARIO and NUSWIDE dataset respectively, with calculation detail of each column provided in Appendix H.S
based on their C-DCS. Due to space
Attacks pose great threat to VFL. Comparing the black squares illustrating the MP and AP the attack against a VFL system without any defense in the sub-figures, we can observe that DS, DLI, BLI and TBM attacks strong attacks with AP higher than 0.97, while MF attacks quite weak
7

---

Published as a conference paper at ICLR 2024
Table 8: T-DCS and C-DCS for All Defenses INUSWIDE dataset, agg VFL, FedSGD]
Defense
Defense Parameter
T-DCSu,
T-DCSu,
T-DCS T-DCSpR
T-DCST8
T-DCSxTE
C-DCS
Name
MID
10000
0.7358
0.8559
0,8159
0.5833
0.7333
0.8707
0.7508
MID
1.0
0.7476
0.8472
0.8140
0.5833
0.7331
0,8700
0,7501
MID
100
0.7320
0.8536
0.8130
0.5833
0.7326
0,8711
0.7500
G-DP
0.1
0.7375
0.8262
0,7966
0.5863
0.7282
0.8675
0,7447
L-DP
0.1
0.7389
0.8177
0.7915
0.5863
0.7258
0.8603
0.7410
MID
0.1
0.7516
0,8259
0.8011
0.5833
0.7172
0.8563
0.7395
MID
0.7280
0.8092
0.7822
0.5844
0.7151
0.8627
0.7361
dCor
0.3
0.7641
0.8411
0.8155
0,5834
0.7289
0,8051
0.7332
dCor
0.0001
0.6496
0.6340
0.6392
0.5864
0.6307
0.8287
0,6712
GS
99.0
0,7404
0.8060
0,7841
0.6415
0.8408
CAE
0.6863
0.7822
0.7502
0.6830
DCAE
0.0
0.6669
0,8660
0.7996
0.6816
GPer
0.01
0.7386
0.8412
0.8070
0.7193
with AP below 0.1. Other attacks, including NS (see Fig. 12 in Appendix 1.2). and LRB, also pose great threat to the VFL system.
PMC. AMC, GRN
Defenses exhibit trade-offs between MP and AP. For most of the attacks and defenses, we can observe an apparent trade-off between MP and AP, i.e. lower AP is often gained with increasing harm of MP defense strength grows, which can be controlled by adjusting defense syper-parameters. An increase of noise level in DP-G and DP-L, sparsification rate in GS, regularization typer-parameter ag in dCor, confusional strength X2 in CAE and DCAE, regularization hyper-paramend À in MID or a decrease of DP budget é in GPer will lead to lower MP and AP.
DCS rankings are consistent across various datasets and settings. As shown in Tabs. 15 to 17 the results of the C-DCS rankings are generally consistent across all 3 datasets. In addition, the C-DCS ranking of the defense methods are still generally consistent even when the VFL model partition and communication protocol changes, as shown in Tabs. 15 and 19 in Appendix As summarised in Fig. 22 these results demonstrate the robustness of the proposed DCS metrics, as well as the stableness of relative performance of different defense methods. Note that, T-DCSyR values much lower than the T-DCS of other types, indicating that FR attacks are harder to defend than other attacks, which are consistent with human observation (see Fig. 13 in Appendix 1.2.2).
MID, L-DP and G-DP are effective on a wide spectrum of attacks. MID demonstrates its capability of achieving a relatively lower AP while maintaining a higher MP compared to most other defenses shown in Figs. 3 and 12 and Tabs. to 17 L-DP and G-DP are also generally effective under most attacks with above average T-DCS and CDDCS; DCAE is effective in defending against LI attacks; GS demonstrates strong defense ability for most of the LI attacks but performs less than satisfactory on LRB attacks; GPer performs similar to DP-G and DP-L in defending against label related attacks; dCor is less effective in limiting AP under NTB attacks but is largely effective against PMC and AMC attacks as shown in Figs.
and 12
Change in 8 does not significantly im- pact the C-DCS ranking. / in Eq. ①T) represents users' trade-off preference on AP and MP when evaluating defenses, and can be adjusted. Here we use for our main results. Figs, 4 14 and T5 show the change of the ranking results with the change of B. Overall the relative rank- ings are not significantly impacted by B, demonstrating the stableness of the com- parison results among various defenses. As 3 grows to large values, e.g. 2 0.9, the metric places overly strong weight on MP, resulting in more variations on the rank- ings. Specifically, dCor ranks higher with the increase of 3 thanks its better MP preservation the cost of weaker AP limitation.
0.5
Bank
#0334
21
PDC3
Figure 4: Change of C-DCS ranking with the change of
8. [MNIST dataset, agg FedSGDJ
splitVFL. is less vulnerable to attacks than agg VLL. Using DCS metrics, we directly compare all the aforementioned attacks and defenses under VLL and splitVFL settings to understand the impact changing the model partition strategy on VFL's vulnerability against attacks. We mainly use the DCS gap, defined as SppiL VEL for each attack-defense point. Figs. 16 and
8

---

Published as a conference paper at ICLR 2024
show the DCS gap for all the attack-defense points among the 11 attacks and 8 defenses with all the evaluated parameters using MNIST and
(see Appendix 1.2.4
defense wib difirie
NUSWIDE dataset respectively. Figs. 5 and 17 displays the distribution of the DCS gaps depicted in Figs. T6 and [8] respec- tively. As all the black square points in Fig. 16 appear above or close to the red horizontal line at value of 0.0 (see also the blue histograms that appear mostly at the right of the vertical line at a value of 0.0 in Fig. 5 we can conclude that splitVFL is less vulnerable to attacks than agg VFL when defense is applied. In addition, splitVFL has an overall positive effect on boosting defense performance against attacks as well, as most of the DCS gap is positive in the last subplot of Fig. when no attack is applied. Similar results can be seen from Figs. a7d 18 Appendix [.2.4 Further analysis are included in Appendix [.2.4]
A:SCSopto DCStegg
Figure 5: DCS gap Distribution, y-axis represents den- sity IMNIST dataset, splitVLLaggVVLL, FedSGD]
FedBCD is less vulnerable to attacks than FedSGD. In addition, we compare DCS gap under FedSGD setting and Fed- BCD with Q 5 to assess the impact different communication protocols on model's vulnerability to attacks. DCS gap is defined as DC FeaSGD for each attack-defense point. Figs. and 2nd200povvided the distribution of the DCS gaps plotted in Figs. 19 and 21 (see Ap- pendix (.2.5) which includes the DCS gap of all the attack-defens points among the attacks and the 8 defenses using MÄIST and NUSWIDE dataset respectively. As shown in Fig. 6 the blue histograms gen- erally appear on the right of the vertical line of value 0.0. indicating that a system with FedBCD protocol is less vulnerable attacks when defense method is applied. In addition, system with FedBCD also has an overall positive effect on boosting defense performance against FR and NTB attacks. This is evidenced by the fact that that the majority of DCS gaps are positive for FR and NTB attacks as shown in Fig. [ Similar conclusions can be drawn from Figs. 20 and 21 Further analysis
widetree
LIDCSOO0 DYNOOL
PRODCS000 DENOLL
TR.Dc (e0esoon
ATRL noron
Figure 6: DCS gap Distribution, y-axis represents den- sity IMNIST dataset, VFL, FedBCI/FedSGD
included in Appendix 1.2.5
7 CONCLUSIONS AND LIMITATIONS
In this work, introduce light-weight VFL framework VFLAIR that implements basic VFL training and evaluation flow under multiple model partition, model protocols and attacks and defenses algorithms using datasets of different modality. We also introduce unified evaluation metrics and benchmark model utility, communication and computation efficiency, and defense performance under various VFL settings, which sheds lights on choosing partition, communication and defense techniques in practical deployment. Currently, the library has limited implementations on cryptographic techniques. Combination of non-cryptograhic and cryptographic techniques would be interesting next step and we plan to add more advanced and communication-efficia methods to our library.
REPRODUCIBILITY STATEMENT
We include all the hyper-parameters used for the benchmark experiments in Appendix Our code is also available at htttt:://gtthub. wttp://www....///AAAAAT////////

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
This work was supported by the National Key R&D Program of China under Grant No.20222I006038 60504, the Tsinghua UniversityAARR)-siaande Technologies (China) Inc. Joint Re- search Center, and Singhua-Toyook Joint Research Institute inter-disciplinary Program.
REFERENCES
Alham Fikri Aji and Kenneth Heafield. Sparse communication for distributed gradient descent. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 440-445, 2017.
Baidu. Paddlefl. http://ssggghh....cc/////aaaeeea
Barry Becker and Ronny Kohavi. Adult. UCI hitps://doi.org/1..244444 2442224422/5
Machine
Learning Repository, 1996. DOI:
ByteDance. Fedlearner github. httpss//atttiaa.c.oo///eeeeeeeer
Fangfang Cai. Bytedance breaks federated learning: Open source fedlearner framework, 209% increase in advertising efficiency. Available online: http://wwwwigghhhcc....oootttttd 03-9 (accessed on 15 March 2021), 2020.
Timothy Castiglia, Shiqiang Wang, and Stacy Patterson. Flexible vertical federated learning with heterogeneous parties. ArXiv, abs/2208. 12272208..1662, 2022a.
Timothy J Castiglia, Anirban Das, Shiqiang Wang, and Stacy Patterson. Compressed-vie Communicationn-eficcient learning with vertically partitioned data. In International Conference on Machine Learning, pp. 2738-2766. PMLR, 2022b.
Di Chai, Leye Wang, Kai Chen, and Qiang Yang. Fedeval: A benchmark system with comprehensive evaluation model for federated learning. arXiv preprint arXiv:20/ 1.09655, 2020.
Javad Ghareh Chamani and Dimitrios Papadopoulos. Mitigating leakage in federated learning with trusted hardware. arXiv preprint arXiv:2011.0425
2020.
Weijing Chen, Guoqiang Ma Tao Fan, Yan Kang, Qian Xu, and Qiang Yang. Secureboost4: A high performance gradient boosting tree framework for large scale vertical federated learning, arXiv preprint arXiv:21
10927, I0.1222,, 2021.
Heng-Tre Tze Cheng, Levent Koe, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, et al. Wide & deep learning for recommender systems. In Proceedings of the workshop on deep learning for recommender systems, pp. 7-10, 2016.
Kewei Cheng, Tao Fan, Yilun Jin, Yang Liu, Tianjian Chen, Dimitrios Papadopoulos, and Qiang Yang. Secureboost: A lossless federated learning framework. IEEE Intelligent Systems, pp. 1-1, 2021. doi: 10.
Yong Cheng, Yang Liu, Tianjian Chen, and Qiang Yang. Federated learning for privacy-preserving ai. Communications of the ACM, 63(12)333-36, 2020.
Yuanhong Cheng, chi Jin, Peizhe Cheng, and Boliang Lv. The prevention, control and thinking of the new type of "money laundering" in the digital age. Financial Computer of China, (6):4, 2022.
URL https: //mp::://mpp mp. wei x xin.nn.. gg. com/ 8/ s/PPyylldvgG.ggu1100cbbb
Tat-Seng Chua, Jinhui Tang, Richang Hong, Haojie Li, Zhiping Luo, and Yan-Tao Zheng. NUS- WIDE: A real-world web image database from national university of singapore. Proc. of ACM Conf. on Image and Video Retrieval (CIVR '09), Santorini. Greece., July 2009.
Nikoli Dryden, Tim Moon, Sam Ade Jacobs, and Brian Van Essen. Communication quantization for data-parallel training of deep neural networks. In 2016 2nd Workshop on Machine Learning in HPC Environments (MLHPC), pp. 1-8. IEFE, 2016.
10

---

Published as a conference paper at ICLR 2024
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL http::/archiii/
ics.uci moci.edu/m
Cynthia Dwork. Differential privacy. Proceedings of the 33rd International Conference on Automata, Languages and Programming- Volume Part 11, ICALP"06, pp. 1-12, Berlin, Heidelberg, 2006. Springer- Verlag. ISBN 0-540-3590779, 378-3-540-33907-44 doi: 00.1007/11787000 URL
Wenjing Fang. Derun Zhao, Jin Tan, Chaochao Chen, Chaofan Yu, Li Wang, Lei Wang, Jun Zhou, and Benyu Zhang. Large-scale secure xgb for vertical federated learning. CIKM '21, pp. 443-452, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450384469. doi:
10.1145/345963773882311
FedAI-maintainers, Fate github. http://///gittuub.booeeeeeeeeeee
Siwei Feng and Han Yu. Multi-participan multi-class vertical federated learning. arXiv preprint arXiv:2001.11154, 2020.
Siwci Feng, Boyang Li, Han Yu, Yang Liu, and Qiang Yang. Semi-Supervised Federated Heteroge- neous Transfer Learning. Knowledge-Based Systems, 252:109384 2022.
Zhi Feng, Haoyi Xiong, Chuanyuan Song, Sijia Yang, Baoxin Zhao. Licheng Wang, Zeyu Chen, Shengwen Yang, Liping Liu, and Jun Huan. Securegbm: Secure multi-party gradient boosting. In 2019 IEEE International Conference Big Data (Big Data), pp. 1312-1321, 2019. doi: 10.1 10t./iiaaaa4709...222
Chong Fu, Xuhong Zhang, Shouling Ji, Jinyin Chen, Jingzheng Wu, Shanqing Guo, Jun Zhou, Alex X Liu, and Ting Wang. Label inference attacks against vertical federated learning. In 31st USENIX Security Symposium (USENIX Security 22), Boston, MA, August 2022a. USENIX Association. URL httpp////ww...eeee..ooo/oooeeeee
presentation/S
Fangcheng Fu, Yingxia Shao, Lele Yu, Jiawei Jiang, Huanran Xue, Yangyu Tao, and Bin Cui. V12boost: Very fast vertical federated gradient boosting for cross-enterprise learning. In Proceed- ings of the 2021 International Conference on Management of Data, pp. 563-576, 2021.
Fangcheng Fu, Xupeng Miao, Jiawei Jiang, Huanran Xue, and Bin Cui. Towards Communication- efficient Vertical Federated Learning Training via Cache-enabled Local Updates. arXiv preprint arXiv:2207. 14628, 2022b.
Fangcheng Fu, Xupeng Miao, Jiawei Jiang, Huanran Xue, and Bin Cui. Towards communication- efficient vertical federated learning training via cache-enabled local updates. arXiv preprint N.Xi::2207.14628, 2022c.
Badih Ghazi, Noah Golowich, Ravi Kumar, Pasin Manurangsi, and Chiyuan Zhang. Deep learning with label differential privacy. Advances in neural information processing systems, 34:27131- 27145, 2021.
Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. Deepfm: a factorization- machine based neural network for ctr prediction. arXiv preprint
2017.
Chaoyang He, Songze Li, Jinhyun So. Xiao Zeng, Mi Zhang, Hongyi Wang, Xiaoyang Wang, Praneeth Vepakomma, Abhishek Singh, Hang Qiu, et al. Fedml: A research library and benchmark for federated machine learning. arXiv preprint arXiv: 13518, 2020.
Tin Kam Ho. Random decision forests. In Proceedings of 3rd International Conference on Document Analysis and Recognition, volume 1, pp. 278 272-282 vol.1. 1995. doi: 10.1109//DARR.1990 .598994.
Xue Jiang, Xuebing Zhou, and Jens Grossklags. Comprehensive analysis of privacy leakage in vertical federated learning during prediction. Proc. Priv. Enhancing Technol. 2022(2):263-281, 2022a.
Xue Jiang, Xuebing Zhou, and Jens Grossklags. Signds-fl: Local differentially private federated learning with sign-based dimension selection. ACM Transactions on Intelligent Systems and Technology (TIST), 2022b.
11

---

Published as a conference paper at ICLR 2024
Xiao Jin, Pin- Yu Chen, Chia- Yi Hsu, Chia-Mu Yu, and Tianyi Chen. Cafe: Catastrophic data leakage in vertical federated learning. Advances in Neural Information Processing Systems, 14:994-1006, 2021.
Michael Kahn. Diabetes. UCI Machine Learning Repository. DOI: https///biirrgg/0022333777717
Yan Kang, Jiahuan Luo, Yuanqin He, Xiaojin Zhang, Lixin Fan, and Qiang Yang. A framework for evaluating privacy-utility trade-off in vertical federated learning. arXiv preprint arXiv:2209.03885, 2022.
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph networs networks. In International Conference on Learning Representations, 2017. URL htttps://operreviie.
net / Eor.mmiddd
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical Report University of Toronto, Toronto, Ontario, 2009.
Fan Lai, Yinwei Dai, Sanjay Singapuram, Jiachen Liu, Xiangfeng Zhu, Harsha Madhyastha, and Mosharaf Chowdhury. Fedscale: Benchmarking model and system performance of federated learning scale. In International Conference on Machine Learning, pp. 11814-11827. PMLR, 2022.
Ken Lang. Newsweeder: Learning to filter netnews. In Proceedings of the Twelfth International Conference on Machine Learning, pp. 331-339, 1995.
Jingtao Li, Adnan Siraj Rakin, Xing Chen, Zhezhi He, Deliang Fan, and Chaitali Chakrabarti. Ressfl: A resistance transfer framework for defending model inversion attack in split federated learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 10294, 2022a.
Oscar Li, Jiankai Sun, Xin Yang, Weihao Gao, Hongyi Zhang, Junyuan Xie, Virginia Smith, and Chong Wang. Label leakage and protection in two-party split learning. In International Confer- ence on Learning Representations, 2022b. URL cCtBRgsf2f0,
net/ wontt/formmmiii.
Qinbin Li, Yanzheng Cai, Yuxuan Han, Ching Man Yung. Tianyuan Fu, and Bingsheng He. Fedtree: fast, effective, and secure tree-based federated learning system. 2022c.
Wenjie Li, Qiaolin Xia, Junfeng Deng, Hao Cheng. Jiangming Liu, Kouying Xue, Yong Cheng, and Shu'Tao Xia. Vfed-ssd: Towards practical vertical federated advertising. International Workshop on Trustworthy Federated Learning in Conjunction with IJCAI, 2022d.
Xiaochen Li, Yuke Hu, Weiran Liu, Hanwen Feng, Li Peng, Yuan Hong, Kui Ren, and Zhan Qin. Opboost: A vertical federated tree boosting framework based on order- onder-preservig desensitization. In Proceedings of the 49th International Conference on Very Large Data Bases, 2022c.
Yizhen Lin. The practice of federated learning tencent wesee advertising. Available online: https://cloud.tencce tencent. www////ebbpee/arrce///88/////// 2021.
Yujun Lin, Song Han, Huizi Mao, Yu Wang, and Bill Dally. Deep gradient compression: Reducing the communication bandwidth for distributed training. In International Conference on Learning Representations, 2018
Changxin Liu, Zirui Zhou, Yang Shi, Jian Pei, Lingyang Chu, and Yong Zhang. Achieving model 2021a.
fairness in vertical federated learning. arXiv preprint
Jing Liu, Chulin Xie, Krishnaram Kenthapadi, Sanmi Koyejo, and Bo Li. Rvfr: Robust vertical federated learning via feature subspace recovery. In NeurIPS Workshop New Frontiers in Federated Learning: Privacy, Fairness, Robustness, Personalization and Data Ownership, 2021b.
Yang Liu, Tianjian Chen, and Qiang Yang. Secure federated transfer learning, CoRR, abs/1812.003377, 2018. URL bttttt/tp//axxi / abs / 1812. C3337
Yang Liu, Yingting Liu, Zhijie Liu, Yuxuan Liang, Chuishi Meng, Junbo Zhang, and Yu Zheng. Federated forest. IEEE Transactions on Big Data, 2020a.
12

---

Published as a conference paper at ICLR 2024
Yang Liu, Zhihao Yi, and Tianjian Chen. Backdoor attacks and defenses in feature feature-partiiioned collaborative learning. arXiv preprint wrXiv:207..06000 2020b.
Yang Liu, Tao Fan, Tianjian Chen, Qian Xu, and Qiang Yang. Fate: An industrial grade platform for collaborative learning with data protection. J. Mach. Learn. Res., 22(226):1-6, 2021c.
Yang Liu, Yan Kang, Tianyuan Zou, Yanhong Pu, Yuanqin He, Xiaozhou Ye, Ye Ouyang. Ya-Qin Zhang, and Qiang Yang. Vertical federated learning. arXiv preprint arXiv:2211. 11/2211..21444. /2814, 2022a.
Yang Liu, Xinwei Zhang, Yan Kang, Liping Li, Tianjian Chen, Mingyi Hong, and Qiang Yang. Fedbed: A communication-eficienn collaborative learning framework for distributed features. IEEE Transactions on Signal Processing, 70:4277-42900 2022b.
Xinjian Luo, Yuncheng Wu, Xiaokui Xiao, and Beng Chin Ooi. Feature inference attack on model predictions in vertical federated learning. In 202/ IEEE 37th International Conference on Data Engineering (ICDE). pp. 181-192 IEEE, 2021.
Andrew Kachites McCallum, Kamal Nigam, Jason Rennie, and Kristie Seymore. Automating the construction of internet portals with machine learning. Information Retrieval, 3:127-163, 2000.
H. Brendan McMahan, Eider Moore, Daniel Ramage, and Blaise Agüera Arcas. Federated learning of deep networks using model averaging. CoRR, abs/1602.05629, 2016a. URL http:
//arxiv. org/ hbtp///...116... 05629
H Brendan McMahan, Eider Moore, Daniel Ramage, and Blaise Agüera Arcas. Federated learning of deep networks using model averaging. arXiv preprint (xXv::6602.0569, 2, 2016b.
Qi Pang. Yuanyuan Yuan, Shuai Wang, and Wenting Zheng. Adi: Adversarial dominating inputs in vertical federated learning systems. In 2023 IEEE Symposium on Security and Privacy (SP), pp. 1875-1892. IEEE Computer Society, 2023.
Tao Qi, Fangzhao Wu, Chuhan Wu, Lingjuan Lyu. Tong Xu, Hao Liao, Zhongliang Yang, Yongfeng Huang, and Xing Xie. Fairvfl: A fair vertical federated learning framework with contrastive adversarial learning. Advances in Neural Information Processing Systems, 35:7852-7865, 2022.
Pengyu Qiu, Xuhong Zhang, Shouling Ji, Tianyu Du, Yuwen Pu, Jun Zhou, and Ting Wang. Your labels are selling you out: Relation leaks in vertical federated learning. IEEE Transactions on Dependable and Secure Computing, 2022.
Yanru Qu, Bohui Fang, Weinan Zhang, Ruiming Tang, Minzhe Niu, Huifeng Guo, Yong Yu, and Xi- uqiang He. Product-basee neural networks for user response prediction over multi-field categorical data. ACM Transactions Information Systems (TOIS), 37(1):1-35, 2018.
Daniele Romanini, Adam James Hall, Pavlos Papadopoulos. Tom Titcombe, Abbas Ismail, Tudor Cebere, Robert Sandmann, Robin Rochm, and Michael A Hoch. Pyvertical: A vertical federated learning framework for multi-headed splitnn. arXiv preprint arXiv:2 104. 004899 2021.
Theo Ryffel, Andrew Trask, Morten Dahl, Bobby Wagner, Jason Mancuso, Daniel Rueckert, and Jonathan Maaserat-Palmbach A generic framework for privacy preserving deep learning. arXiv preprint arXiv: /8/ 1.04017, 2018.
Reza Shokri and Vitaly Shmatikov. Privacy-preserving deep learning. In Proceedings of the 22nd ACM SIGSAC conference on computer and communications security, pp. 1310-1321, 2015.
Congzheng Song and Vitaly Shmatikov. Overlearning reveals sensitive attributes. In International Conference on Learning Representations, 2020. /tllhttts////opeeee..ee.. f
id= Sd=SJeNZ04EDS
W Nick Street, William H Wolberg, and Olvi I. Mangasarian. Nuclear feature extraction for breast tumor diagnosis. In Biomedical image processing and biomedical visualization, volume 1905, pp. 861-870. SPIE, 1993.
Jiankai Sun, Xin Yang. Yuanshun Yao, Aonan Zhang, Weihao Gao, Junyuan Xie, and Chong Wang. Vertical federated learning without revealing intersection membership. arXiv preprint arXiv:2/066 05508, 2021a.
13

---

Published as a conference paper at ICLR 2024
Jiankai Sun, Yuanshun Yao, Weihao Gao, Junyuan Xie, and Chong Wang. Defending against reconstruction attack vertical federated learning. arXiv preprini urXiv:2107.09880 2021b.
Jiankai Sun, Xin Yang, Yuanshun Yao, and Chong Wang. Label leakage and protection from forward embedding in vertical federated learning. arXiv preprint arXiv:2203.0145,, 2022.
Hideaki Takahashi, Jingjing Liu, and Yang Liu. Eliminating label leakage in tree-based vertical federated learning. arXiv preprint TrXiv:2307.10313 2023.
Zhihua Tian, Rui Zhang, Xiaoyang Hou, Jian Liu, and Kui Ren. Federboost: Private federated learning for gbdt. arXiv preprint vrXiv:2011.02777 1.0279, 2020.
Praneeth Vepakomma, Otkrist Gupta, Abhimanyu Dubey, and Ramesh Raskar. Reducing leakage in distributed deep learning for sensitive health data. arXiv preprint 1811008812.00544, 2, 2019.
Yilei Wang. Qingzhe Lv. Huang Zhang, Minghao Zhao, Yuhong Sun, Lingkai Ran, and Tao Li. Beyond model splitting: Preventing label inference attacks in vertical federated learning with dispersed training. World Wide Web, pp. 1-17, 2023.
Yuncheng Wu, Shaofeng Cai, Xiaokui Xiao, Gang Chen, and Beng Chin Ooi. Privacy preserving vertical federated learning for tree-hased models. Proc. VLDB Endow., 13(12):2000-2033 jul 2020. ISSN 2150-8097. doi: 10. 1447788/440077994477881.
Lunchen Xie, Jiaqi Liu, Songtao Lu, Tsung-Hui Chang, and Qingjiang Shi. An efficient learning framework for federated xgboost using secret sharing and distributed optimization. ACM Trans. doi: 10.1 10./355//53366..
Intell. Syst. Technol., 13(5), sep 2022. ISSN 2157-6904.
Qiang Yang, Yang Liu, Tianjian Chen, and Yongxin Tong. Federated machine learning: Concept and applications. ACM Transactions Intelligent Systems and Technology, 10(2):12:1-12:11 2019a.
Qiang Yang, Yang Liu, Yong Cheng, Yan Kang, Tianjian Chen, and Han Yu. Federated learning. Synthesis Lectures on Artificial Intelligence and Machine Learning, 13()) 2019b.
Xin Yang, Jiankai Sun, Yuanshun Yao, Junyuan Xie, and Chong Wang. Differentially private label protection in split learning. arXiv preprint 2=Xix2223300207: 2022a.
Yitao Yang, Xiucai Ye, and Tetsuya Sakurai. Multi-view federated learning with data collaboration. ICMLC 2022, pp. 178-183, New York, NY, USA, 2022b. Association for Computing Machinery. ISBN 978 1784495955 doi: 10. 00.11447/5298663529990
Christopher J.C. Burges Yann LeCun, Corinna Cortes. The mnist dataset. http://yann. Com htmm//xdb/misst/inccooo html
Houpu Yao, Jiazhou Wang, Peng Dai, Liefeng Bo, and Yanqing Chen, An efficient and robust system for vertically federated random forest. arXiv preprint arXiv:2201.107611 2022.
Peng Ye, Zhifeng Jiang, Wei Wang. Bo Li, and Baochun Li. Feature reconstruction attacks and countermeasures of dnn training in vertical federated learning, 2022a. httts://t////x
orq/abs/2210. 22100. 06771
Peng Ye, Zhifeng Jiang, Wei Wang. Bo Li, and Baochun Li. Feature reconstruction attacks and countermeasures of dnn training in vertical federated learning. arXiv preprint arXiv:2210.06771, 2022b.
Jie Zhang, Song Guo, Zhihao Qu, Deze Zeng, Haozhao Wang. Qifeng Liu, and Albert Y. Zomaya. Adaptive vertical federated learning on unbalanced features. IEEE Transactions on Parallel and Distributed Systems, 33(12):4006-4018, 2022. doi: 1099/DSS 2.0.1..5...0....8844....
Tuo Zhang, Tiantian Feng, Samiul Alam, Sunwoo Lee, Mi Zhang, Shrikanth Narayanan, and Salman Avestimehr. Fedaudio: A federated learning benchmark for audio tasks. In ICASSP 2023-2023 IEEF International Conference Acoustics, Speech and Signal Processing (ICASSP). pp. 1-5. IEEE, 2023.
Tianchen Zhou, Zhanyi Hu, Bingzhe Wu, and Cen Chen. Slperf: a unified framework for benchmark- ing split learning. arXiv preprint 204.02304 2023.
14

---

Published as a conference paper at ICLR 2024
Tianyuan Zou, Yang Liu, Yan Kang, Wenhan Liu, Yuanqin He, Zhihao Yi, Qiang Yang, and Ya- fa-Qin Zhang. Defending batch-level label inference and replacement attacks vertical federated learning. IEEE Transactions Big Data, 2022.
Tianyuan Zou, Yang Liu, and Ya-Qin Zhang. Mutual information regularization for vertical federated learning. arXiv preprint arXiv:2301.01 /42, 2023.
15

---

Published as a conference paper at ICLR 2024
A RELATED WORK
Vertical Federated Learning. Federated Learning (FL) (McMahan et al., 2016b; Yang et al. 2019b a) is a learning paradigm which allows multiple parties to build a machine learning model collaboratively without centralizing each data owner's local private data. Depending on whether data are partitioned by sample or by feature, FL can be further categorized into Horizontal FL (HFL) and Vertical FL. (VFL.) (Yang et al., 2019a). VFI. (Liu et al., 20226; Cheng et al.) 2021 Jiang et al. 2022b) is commonly applied in real-world applications in the field of finance and advertising (Cheng ct al. 2020; FedAI-maintannes a common group of users jointly build machine learning models while keeping both local data and local model private. In VFI., only one participant possesses sensitive label information and is often referred to as active party while others are referred to as passive parties. Depending on how model is partitioned among parties, the VFL architecture can be further categorized into VggVFL and splitVFL (Liu et al., 2022a). In Vgg each party possesses local sub-model, and a non-trainable aggregation f'unction is used as the global model; while in splitVFL. trainable aggregation model is used. Communication efficiency issue is key bottleneck in VFL training. Approaches such as FedBCD (Liu et al. 2022b), Compressed VFL (Castiglia et al. 2022b) (including Quantize and Top-k compression), Flex- VFL (Castiglia et al. 2022a), VdL (Zhang et al. (022). and CELU-VFL (Fu (ct al. 2022b) enhance VFL system efficiency by enabling each party to perform multiple local updates during each communication iteration. Although raw data and model parameters are not shared in NN-based VFL training and inference procedure, the threats of data leakage and model integrity remain. Existing attacks target either the reconstruction of private data (Li et al. 20226
where cross-silo participants holding different sets of features of
Jiang al. 2022a) or the compromise of model robustness (Liu et al.) 2020b Zou et al. 2022
Pang et al.] 2023). Private label or private features are both potential targets for data leakage attacks.
Sample-leve gradients (SLI) (Li et al. 2022b Fu et al.) [022at Sun al. [02 Yang et al. 2022a)
or batch-level gradients (BL.) (Zou et al. 2022) or trained local models (Fu et al.
(022aa) can all be
2021 Luo et al. 202T; Li et al. 2022a Jiang et al. 2022a) for white-box or black-box oracle setting with image or tabular data; or linear equation solving technique (Ye et al. (022b]a) a) for black-box setting with binary value data can be applied to conduct feature reconstruction attacks. Assigning specific label to triggered samples (Zou et al. 2022) or adding noise to randomly selected samples (Liu et al., 2021b failing to transmit collaboration information (Liu et al. 2021b) can all result in successful backdoor attacks. These attacks pose significant challenges to VFL settings and necessitate effective countermeasures. To mitigate these threats, multiple defense methods have been proposed. Aside from cryptography techniques such as HE or Secure Multi-Party Computation (MPC) (Yang et al.
exploit to conduct label inference attacks. Model inversion technique (Jin et al.
2019a). proorraptyg techniques such as reducing information by Adding Noise (Dwork [2006]
Li et al. 2022b [uu al. 2022a Zou al. 2022). J Gradient Sparsification (Aji &eafield (017))
Gradient Discretization (Dyyden et al.) 2016; Fu et al., 2022a), Gradient Compression (Lin et al.
2018), or the combination of these information reduction techniques (Shokri &Shmatikov 2015 Fu
ct al. 2022a) are often applied. Additionally, emerging defense mechanisms have been proposed
either for specific attacks or multiple type of attacks. Disguising labels (Zou et al., 2022 Jin et al.
2021). label DP (Ghazi et al., 202T Yang et al. 2022a), Dispersed Training (Wang et al., 2023)
have proven effective in defending against label inference attacks; Dropout and Rounding (Luo
ctt al. 2021), Adversarial Training (Sun et al., 20216 Li et al.) 2022a). Fabricated Features (Ye
ct al. 2022b) are proposed to prevent input feature leakage; Robust Feature Recovery (Liu et al.
202 has demonstrated its effectiveness for eliminating backdoor attacks, while Mutual Information
Regularization (Zou et al. (022) and Distance Correlation Regularization (Vepakomma ct al. 2019
Sun et al. 2022) seek to mitigate various types of attacks.
Tree-based VFL. Due to its exceptional model performance and its inherent explainability, tree- VFL has garnered widespread applications. Gradient Boosting Decision Tree (GBDT) is the most commonly employed approach for constructing trees-based VFL with various related algorithms
having been proposed (Cheng et al. 2021 Chen et al. 2021 Fang et al. [202 Tian et al. 2020
Xie et al. 2022). Another noteworthy based ensemble algorithm in VFL is Random Forest
(RF) Ho 1995) which leverages bagging and parallelism optimization techniques to improve training and inference efficiency (Liu et al.) 2020al Yao et al. 2022). While most tree-based VFL methods employ cryptographic approaches to protect sensitive data, some encrypt only partial information to
enhance efficiency (Cheng et al. 2021 Wu et al. 2020) but results in data leakage threats (Chamani
&Papadopoulos) [2020] Takahashi et al. 2023). Th Tus. various methods have been investigated aiming
to solve the privacy (Cheng et al. 2022 Chen et al. 2021 |Fng et al. 2019 Fang et al. 2021 Tian
et al. 2020
Li al. 2022e) and efficiency (Li al. 2022e) issue.
16

---

Published as a conference paper at ICLR 2024
B VFL FRAMEWORK
We include the training algorithm of NN-based VFL (Algorithm D as well as the description and algorithm (Algorithm of the training and inference process of tree-based VFL in this section.
Algorithm 1 A Basic VFL Training Procedure using FedSGD.
Input:learning rates !/ and 7/2
Oufput: Model parameters 01, 02 ... 0g. 4
1: Party ..2....,K' initialize 01, 02 0x, 4.
2: for each iteration j 1, 2.... do
3: Randomly sample a mmini-batch of samples {x,y} C D of size n;
4: for each party k=1,2,.... ..K in parallel do
5: Party l computes H = GR 0n);
6: Party l sends H₃ to active party K;
7:
end for
8:
Active party computes the prediction
02 F(H,, HK, vp) and loss L f ,))
9:
Active party K updates p لي - 171 0p*
10:
Active party K computes and sends ac aH,
to all other parties;
11:
for each party k=1,2...,,k in parallel do
12:
Party i computes Vo, L 09
ac
aC aH* aH* 86
Party A updates 0f+11 0g 1/2 Vo L;
13:
14: end for
15:
end for
We also support using Homomorphic Encryption (HE) with Paillier Encryption to protect transmitted results during training VFL Zou et al. (2022) (see Algorithm 2 In this protocol, Homomorphic Encryption (HE), denoted as [[]. is applied to the communicated information, i.e., local model outputs H₁ in forward propagation and respective gradients 8Hf, in backward propagation. The active party
then computes the gradient of loss with respect to [[Hx]] under encryption, i.e. [I ]]. and sends the results back to the passive parties for gradient updates. Note Taylor Expansion used for gradient approximation as Paillier Encryption supports only addition and multiplication, following (Liu et al., 2018). The computed encrypted gradients [8 are subsequently added with a random local mask and transmitted Trusted Third Party (TTP) for decryption. The public key for the encryption generated and distributed to each party by TTP, while the paired private keys are kept at TTP for decryption.
Algorithm 2 vertical federated learning framework with Homomorphic Encryption (Zou al. 2022]
Require: Learning rate 7)
Ensure: Model parameters f1, .......00x
1: Party 1, 2..... K initializes 0,, 02,02,-- .2........ . 0x
2: Trusted Third Party (TTP) creates encryption pairs, sends public key each party
3: for j = 1,2,... do
4: for each passive party k * K in parallel do
5: 1: computes, encrypts, and sends [[+x]] to the active party.
6: end for
7: Active party K computes and sends 0f all other parties
8: for each party 1e 1, 2, ... in parallel do
9: k computes [[ 0 ]] and sends them with random mask TTP for decryption.
10: ic receives and unmasks Va, C. 20,
ac and updates 02+1 g++1_ 0% Ve, C
11: end for
end for
In tree-based VFL. the active party first broadcasts the set of record indices for the current node. Next, each passive party calculates the percentiles for each feature based those indices. The passive party
17

---

Published as a conference paper at ICLR 2024
then proceeds to create binary splits for each feature by comparing the feature value of each instance to the percentile values. After that, the passive party sends back the statistics of each split necessary for evaluation such as purity to the active party, and the active party selects the best split using specific evaluation function. To be precise, Random Forest employs gini impurity for classification, while XGBoost utilizes its gain function, which is based on the gradient and hessian. Finally, the active party requests the owner of the best split send the of record indices for the children nodes generated by the best split. Tree-based VFL system continues these procedures recursively until certain stop conditions, like depth constraints, are satisfied. Algorithm demonstrates training details of tree-based VFL.
Algorithm 3 A Basic Training Process of Tree-based VFL. Input: Evaluation function for split Output Trained trees:
1: for each tree j = 1,2,... do
2: Root Node f- randomly sample subset of record indices;
3: Nodes = [Root Node] [A list of nodes to be divided)
4: while Nodes is not empty do
5: Current node + pop one element of Nodes
6: if Current node satisfies the terminate conditions then
7: Continue
8:
end if
9:
Active party broadcasts the set of record indices of the current node to all other parties; for each party ..,, in parallel do
11:
Party k receives the set of record indices of node to be divided; end for
Party i calculates the statistics for all possible splits and sends them to the active party;
13:
14:
Active party gathers the statistics of possible splits and picks the best one; Party is selected as the best party then Party & sends the sets of record indices of children nodes generated by the best split; end if
15:
16:
17:
18:
Active party receives the sets of record indices of children nodes and appends them to Node; end while
20:
end for
C QUICK GUIDE TO USE AND EXTEND VFLAIR
In this section, we give step-by-step user guidance on how to use and extend VFLAIR.
How to Use. Using VFLAIR requires the following steps:
1. Build. Download our code repository from GitHub and prepare all the required environ- ments using commands shown in Fig. 7(a) Hardware requirements are listed in Tab, i in Appendix D
2. Prepare Dataset. Prepare or download the dataset into folder / the default folder that contains all the datasets for experiments.
3. Configure. Modify the configuration JSON file under ./src/ corc//oofigs///// folder to specify settings, including learning hyper-parameters (e.g. 'epochs' and 'Ir' for learn- ing rate etc.), training dataset ('dataset'), ), training model ('model list for model of each party), model partition (ilobal_modde) trainable not), communication pro- tocol (' ('communication')) as well as attacks ('attack_list') and defense method ('de- fense"). Then rename (e.g. my my_config ison) the configuration file and save it under folder. An example is shown Fig. (b) Detailed explanations of all the parameters are provided in /src/ con f gs READDE. md for NN-based VFL and / //src/contt corc/configee
SERADM__TREES md for Tree-based VFI.
4. Train. Use command ed src and python main pipeline py --configs my_confia consecutively to start a VFL training. Attack and defense evaluation will also be performed if specified in configuration file. MP, AP, communication rounds, amount of
18

---

Published as a conference paper at ICLR 2024
clone the repository
git clore <iink-to-orrggithubrree
install required packages
conda
VFLAIR python=3.8
conda activate VFLAIR
pip install upgrade pip
ed VFLAIR
pip "r reuuttenntts,
instell related pyterch
pip torch=2.11.. torchvision--8.12
torchaudio--0.28.
https://downloed..
(a) Code download and environment preparation.
(
"communication":"
"epochs": 16, "Ir": 9.et,
catpiniiattoootttccoeo "Quaniization"
"iteration wwr.ttteenn___ppeeeeeeeeee per_aggreeation"
"iet 2,
"quaat__eeeee""""""" 16, "vecdin':
"batchusice" 2045,
"dataset":
29
"dataset_ namse" "endst",
38
"defense":
"nun_classes":
31
"name": "GoussiantP",
}
"mndel_ [
32
"paraneters": {
33
"party": [e],
"8":
34
"dp_strongth": 0.1
"type": "MLP2", "input_din":
35
}.
14
"output_din"'
"attack, list":
15
}
16
"1"1
"nare": "BirectLabelscorinn,,
17
"type": "MLP2", "Input_din": 392, "output_din":
"parasotars":
18
"party": [0]
19
28
b
21
"asply_trainaas" Jayen": 8,
22
"gloeal_rodel": "Clesstficeriooooelllllllltt
45
23
}
(b) Example ISON file.
Figure 7: User guidance.
information transmitted each communication round as well as execution time for reaching the specified MP recorded.
5. Evaluate. After training is finished, use file to perform evaluations, e.g. calculate DCS, T-DCS, C-DCS.
ipynb
We summarize the above steps in Fig.
Build
Prepare Dataset
Configure hyper-parameters dataset
Train train with command python by confige corda
Evaluate
download realted
Jaro/metrics/da
from GitHub
dataset
communication
____prccss.ppp lynbb
calculating
protocol
install all the necessarly requirement
place under
number of participants attacks defense encryption
MPAP communication cost & rounds,
DCS, T-DCS and C-DCS.
folder
J.Jshare dataset/
execution time recorded
Figure 8: Step-by-step user guidance for using VFLAIR.
How to Extend. Extending our VFLAIR platform also easy thanks to our modularized design of all the relative components.
19

---

Published as a conference paper at ICLR 2024
New ryper-parameted configurations (e.g. new training optimizer specification) can be file.
added by modifying functions / //tcc//oaa/Laaacooffff...o
New datasets and data partitioning methods can be added by first placing the raw data file under / ssaaee Latiared_atsee//e/aa folder and then modifying functions in /src/load htt.//oa///////// wowa//LoaaDeeeecoo LoadDaaDDDaaaasaaa Pyy file.
New VFL models can be added under //cc/modees//////////////// folder and loaded via modifying the / /wad// LoadModel1 1 py file with proper personalized modification in the 'model_lis _ist' part (see line 10-23 in Fig. 7(b) of the configuration file.
New communication protocols can be added by modifying the provided training flow in /ttc////vvaaaaeee eet.eveallaeesss.eeeee / TannaasVVFraa py file to realize of the new communication protocol or by adding new python file under the same directory of that file if the user finds the modification significant.
New attacks can be added by inheriting Attacker class under httc//vallatee//attaaaaa the main training file
folder for testing time attacks, or by modifying for training time attack.
New
defense methods can be added by modifying functiins py file and the main necessary.
training file http://ceeeaaaaattte///aaaaaaaaa
New party behaviors. e.g. new information exchange strategies for new communication pro- tocols, can be added by modifying folder.
under //rc/ htce/prty////
Use Case: Adding new attack. Below is a simple example for adding new attack named Batchhabblleeconsstuciion:
1. New syper--arameters for Batchhlabel
attack, e.g. attacker party ('party)) ) learning rate ('Ir') and attack model training epochs ('epochs') in this case, can be added to the configuration file in the @attacklistt part (line 29-37 in Fig. (())). Rename the new configuration file, e.g. my my_new_config.. json, and save under //rc// configs/.
2. Implement necessary code for the new attack LaatchLabelRecoo. struction' by inheriting Attacker class (see Fig. wssc//vallaats//atteel/eaa
9()) under
folder for testing time attacks. Save it in file / wswrr//ewlaates//eeeeeeeeeaeeeee wttpaattte/ttttttt/aeaaaeeeeeeee /Batttacccc/aaaiiaiaa chLab Reelleeeoosssseoeo ionttion.. con.Pt py. Note that, the name of the attacker class implemented the file should be aligned with the name of file, BatthhabbeRReoonstuuttii
in this case.
3. Run the new attack with command python main__ipeline, py --configs my_new_confis under //src/ folder.
Use Case: Adding a new local model. We further provide a second example here on adding local model 'MLP2 Softmax" VFLAIR. Assume this local model is for the new attack Buuchhabcleecoonstuuctttnt added above.
1. New hyper- paremeerrssssssss for specifying the usage of 'MLP2 Softmax' model as local model for each party are 'type' attribute model list' part (line 12 and 17 in Fig. 9a)). which should be specified the name of the model. Then save the modified configuration file mny_ne__conff json still under /src/ con configs/.
2. New VFL bottom model 'MLP2 Softmax'
be implemented in htt///oolll///e peooelssmmpp.pp like shown in Fig. [c] In this case, loading this new model does not need to modify ./src/load LoadModel B. file.
3. Run command python main main_pipeling wipeline....eeeem.....p --configs my_new_conte under ./src/ folder.
attack with new local model 'MI.P2_Sfttmx' with
D VFLAIR WORKFLOW
In this section, we present the key code modules and workflow of VFLAIR.
20

---

Published as a conference paper at ICLR 2024
I
"epocns": 38, "Ie": 0.01, "k": 2,
21
"apply_ "apply_traainblle www.ttraaaalee..cc Jayer"; 6,
22
"slobai_roeel'; "Casaaffiattoorooollltttttsss
25
b
"batcrrstze"; 2048, "dataset": "dotaset_nane"" "mnist"
"comunication":
25
"comunication ptt.octttooorsstooo...... "Fed5G0",
"Hteration peg.eggreeatttiooooooo
"nun_classes";
1.
atttck__ist"'
(
-9":
"name": "Barchiabooleccnssttttttto "paroneters"; {
"type"s "NLP2_Softmex", New Model
13
"Inpat dir": 392,
"party": [e],
14
"out 18
33
"In"s 0.85,
15
34
"epochs": 10000
16
"1":
17
"sype": "WLP2 Softwxx", New Model
36
New Attack
18
"inptt_dar"; 392,
19
"outout_din":
(a) New JSON file.
ULF2_Softrexx(n.nr Module):
init (seif, input din, output, dim): super(MLP2 Softnax, self) soif)___noe init_ self.layerd mn.Sqquential m..latten(),
nn, LinLinneriinputtddin din, 32, bias:True), nn. (m.RelU(inplace--Trr))
class bacchaabeleeccnsttuciiinttttt
def [ieit__eee,, top_vfl, args):
self self.exp_res_pat eet_p___tht_thhhhhh
folf.laye22 mn.Sequential
def set, seed[self, seedda)) torch.backends. www.h.aacceeesscddnnnnn. cudnn, benchrsark
nn, LinLeener322, output_din, biss:True), nn, (din:x
True
def calr_laae _ab____bl_____ haic__abl__scccorry.oo (attorrr,att(tsoff,,, label, la_lab return success total
def Forkard x):
self, (xyer
def atelc::
Layer
return best cer__ec_nat rate
return
(b) New attack.
(c) New bottom model.
Figure 9: Adding new class.
VFLAIR
Arciconfit
Areload
Arcinsllaaes
Arc/mstrics
Config Module
Load
Train & Evaluate Module
Metries
Module
Module
Anrodes Medel Librrey
Janach Aeffenee
Shitts Communication Libeary
Datiset Libnary
Library
X
Load Dataset
X Load Model
Joed ponl__rtttett_tre pattttttttt.... model pww.el_eerr_srrrrgo parnyy
coteset_portttiine
Attack&
% Communication
date aate_procsss.oo
class Artaciertiaal
cranees)
ME, AP
Training Pipelines
URounds Amount
Party
Paintacion my das Paintaceen
DCS,
Losd Party ay
Lose partiesl)
Figure Code modules and Workflow of VFLAIR.
As shown in Fig. 10 & Evaluate Module and Metrics Module.
VF IFLAIR contains 4 key modules, namely Config Module, Load Module, Train
21

---

negative correlation Published as a conference paper at ICLR 2024
The Config Module first processes the user-specified configurations (see Appendix for detail), then passes to the Load Module for party preparation. With function load_partiee (), each party separately loads its dataset and local model with function load saee_ddattset_pe_prrt. () and
lcad models_perr parttparttyyyy () respectively.
Afterwards, the training pipeline starts in the Train & Evaluate Module. Communication proto- cols implemented within the training pipeline and are controlled by pred_transe t. () and pradient_transee
() functions. All the supported defense are integrated into the training pipeline. For attacks, training-time attacks (e.g. TB and NTB attacks) are integrated and evaluated within training pipeline while inference-time attacks that do not affect the training flow (e-g. LI and FR attacks) are launched and evaluated after the training pipeline. However, they are all prepared by an At attackerLoader object before the training pipeline starts.
Finally, the Metrics Module processes the recorded results using functions implemented in data_process wrocess......................... py file and produces evaluation metrics including MP, AP, communication rounds (#Rounds), amount of information transmitted between parties each round (Amount), execution time (EXec.Time), DCS, T-DCS and C-DCS.
We compared our framework VFLAIR with FATE, one of the most widely used FL plat- form supporting a broad range of VFL functionalities, on the system requirements for de- ployment in Tab. 99. demonstrate that VFLAIR is lightweight framework. Accord- ing to FATE httpps:/// thub. thub. com hemm/FeeeaaedddA / ob // master/deployy standalone-dae
README mc), deploying a stand-alone version of the FATE framework requires at least a 8 core CPU with 16G memory and 500G hard disk and the downloaded docker package for deployment is of size 4.92G for version 1.7. However, for our VFLAIE LIR, core CPU with less than 4G memory and less than 4.0G hard disk required for installation and environment preparation.
Table 9: Comparison of hardware requirements for installation between FATE and VFLAIR.
CPU
memory
Installation required hard disk
FATE (stand-alone, version 1.7.1.1) VFLAIR
8 core
16G
4.92G
core
4G
<4G
E DETAIL DEFINITION OF ATTACK PERFORMANCE AND ATTACK PERFORMANCE FOR IDEAL DEFENSES
In this section, we give detailed definition of Attack Performance (AP) for each type of attack we studied in the literature. In this work, we use the AP for ideal defense AP* 0.0 indicate a complete failure of the attack..
1. In Label Inference (LI) Attacks, AP is defined as the ratio of correctly inferred labels which can be regarded as the label inference accuracy. That is,
ART Number of Samples Correctly Inferred
Number of Total Samples
One exception is that for LI attacks under binary classification tasks, i.c. NS and DS (see Appendix [ for explanation), we use the AUC of inferred labels instead for better measurement especially for unbalanced dataset. For both settings, high label inference accuracy indicates a successful attack.
2. In Feature Reconstruction (FR) Attacks, Mean Squared Error (MSE) is commonly em- ployed as measure to evaluate the quality of reconstruction, where a smaller MSE signifies higher-quality reconstruction. So define the AP for FR attacks of MSE with a constant shift. Let C denote the constant shift, Uo be the real data stored in the attacked party, Uree be the data reconstructed by the attacker and function MSE(.)) -) for calculating the mean square error between the 2 function inputs, AP can be expressed as:
(5) In our evaluation, we use C = 1.0 and normalize all raw input features to [0.0, 1.0] before training and testing. In this case, MSE(Uo, Uree) [(( ( u(fd)? 2 E [0.0, 1.0) where
APER = - MSE(U,, Urec).
22

---

Published as a conference paper at ICLR 2024
U0 Wree the fth feature of Uo, Uree respectively. Thus, APFR [0.0, 1.0] and a high AP indicates a strong attack.
(5) (f)
3. In Targeted Backdoor Attacks, AP is defined the number of trigger inserted samples that is regarded as the target class by the VFL model to the total number of trigger inserted samples, also the backdoor accuracy, achieved by the final VFL model, that is:
Number of Triggered Samples Inferred as Target Class Number of Triggered Samples
APT8
(6)
Same the? 2 kinds of attacks above, high AP indicates successful backdoor attack.
4. In Non-targeted Backdoor (NTB) Attacks, the gap of MP on attacked samples relative to the overall MP on all the samples is defined as AP. That is:
APNTB Pa Patacked
(7)
Still, a higher AP indicates a more successful attack.
F EVALUATED ATTACKS
In VFLAIR, all the attacks listed below are supported and evaluated in the benchmark experiments. Users can easily extend other attacks and evaluate them using our framework.
F.1 LABEL INFERENCE (LI) ATTACKS
In Label Inference (LI) Attacks, the attacker (also the passive party) hopes to steal the sensitive label information kept active part. There are multiple ways for the passive attacker infer private labels. In this work, we evaluate 6 label inference attacks to assess the vulnerability of a VFL system under label leakage threat, including: Norm-based Scoring (NS) (Li et al., 2022b). Direction-basee
Scoring
(DS) (Li et al.) 20226) and Direct Label Inference (DLI) (Li et al. 20226 Zou et al. (022) that exploit sample-leve gradients, Batch-level Label Inference (BLI) (Zou et al. (022) that uses batch- level gradients, Passive Model Completion (PMC) (Fu al.) 2022a) and Active Model Completion (AMC) (Fu et al. 2022a) that infer label from trained local models.
1. Norm-based Scoring (NS) eLi 2022b). NS is designed for binary classification task, and is based on an experimental observation that the norm of gradient |g]|2 l2 for a positive sample is generally larger than that of negative sample in an unbalanced dataset. Thus the passive party can calculate the norm of sample-level gradients transmitted back by the active party and then infer the sensitive label information according to the norm values.
2. Direction-based Scoring (DS) Li et al. 2022b). Similar to NS, DS also aims to infer label in binary classification task. DS is based on the observation that for any given sample pairs Fas Wh, with their sample-level gradient denoted as 9a. the cosine similarity of these gradients cos 9b) = -h gb/ (119al12ll 2119)lle) has positive value if Fa, Zb are of the same class, and negative value if za. 26 are of opposite classes. Attack can be launched using merely gradients that are transmitted back from the active party. However, to make the attacker more powerful, in the implementation of the attack, the gradient of one positive sample 9+ is additionally given.
3. Direct Label Inference (DLI) (Li al. 2022b Zou et al. 2022 DLI is based the observation that at passive party, for sample {x;, J. yf" element the gradient transmitted back from active party 9i is the only element having opposite sign compared to others, thus the label information is revealed.
4. Batch-level Label Inference (BLI) (Zou et al. 2022). There cases where sample- gradient information is protected, when using encryption techniques for example, and only batch-level gradient is revealed to passive party. BLI is designed to infer labels from this kind of setting. Inversion model is constructed and trained at passive party to invert label information from batch-level gradients.
5. Passive Model Completion (PMC) (Fu et al. 2022a). PMC exploits the information of label inherit in the trained local model at passive party. By adding randomly initialized linear layer on top of the trained local model to get "completion model" and fine-tune the "completion model" with auxiliary label data, the passive party is able to guess the label of each sample.
23

---

Published as a conference paper at ICLR 2024
6. Active Model Completion (AMC) (Fu et al.
2022a AMC is enhanced version of PMC where the attacker (passive party) use malicious local optimizer which adaptively scales up the gradient of each parameter in the adversary's bottom model. This results in more accurate trained local model as the overall VFL model is tricked rely more on the maliciously optimized local model at passive party. With more accurate local model, passive party is expected obtain better attack using the same model completion techniques as PMC.
F.2 FEATURE RECONSTRUCTION ATTACKS
In Feature Reconstruction (FR) Attacks, the attacker aims to recover other parties' local features from its local model and all information received from other innocent parties. Both active and passive party can be the attacker when labels are not needed for completing the attack. When neural network models serves as local models, targeted features are limited to binary values (Ye et al. 2022b a) under a black-box setting in which the attacker neither has any knowledge of nor has access for querying the local model of the party under attack. Generation based model inversion can be employed to reconstruct data by querying the trained model a black-box oracle manner to recover tabular data (Luo et al., )021). For white-box setting that has access to the trained model, model inversion techniques can also be used to recover image data (Jin al. (02T] with the knowledge of trained local model, or with prior knowledge about data (Li ct al. 2022a Jiang et al. 2022a). In this work, test the following 2 FR attacks:
1. Generative Regression Network (GRN) (Luo al.
202T). By querying the trained VFL model, GRN attack reconstructs data features by matching the VFL prediction of real and reconstructed data. A generative model is trained to map random noise to targeted features.
2. Training-basek Back Mapping by model inversion (TBM) (Li et al. 2022a). When auxiliary i.i.d. data of the local private data is available at the attacker, TBM utilizes these data to train generative model in order to map the embedding feature of the victim party back to the original input feature. A strong assumption is used for TBM attack in which the attacker can query the whole trained VFL model with the data it obtained.
F.3 TARGETED BACKDOOR ATTACKS
Targeted Backdoor (TB) Attacks have a clear incorrect leading target and aims to manipulate the VFI. model's 's behavior on samples marked with backdoor related features. We evaluated Label Replacement Backdoor (LRB) (Zou et al. 2022) this work to assess the vulnerability of a VFL system under targeted backdoor threat.
1. Label Replacement Backdoor (LRB) (ou et al. 2022) aims to assign an attacker-chosen label (target label) to input data with a specific pattern (i.e. a trigger). The passive attacker is assumed of knowing a few clean samples from the target class. The triggered poison samples are created locally by adding triggers randomly selected samples from the passive attacker's s own data. In training, the attacker replaces the embedding a known clean sample from the target class with that of triggered poison sample replace the corresponding label of the poisoned sample.
F.4 VON-TARGETED BACKDOOR ATTACK
Unlike TB attacks, Non-targeted Backdoor (NTB) Attacks only aim to affect the utility of the VFL model. Methods like adding noise some randomly selected samples (Zou et al. (2022) or by adding missing features (Liu et al. 2021b) can be exploit during inference. The attacks used to assess the safety of a VFL system under NTB are listed below:
1. Noisy-Sample Backdoor (NSB) (Zou al. 2022 In NSB attack, random noise A/ 2) is attached arbitrarily selected samples by passive party to harm the model utility of VFL.
2. Missing Feature (MF) (Liu et al. 20216). In MF attack, the embedding of passive party's 's local features of some samples are missing either due to the unstable network issue due to the intentional hiding by the passive attacker. Missed features are treated as all in the implementation.
Detail attack hyper-parameters settings can be seen in Appendix
H3
24

---

Published as a conference paper at ICLR 2024
G EVALUATED DEFENSES
In VILAIR, all the defense methods listed in below are supported and evaluated in the benchmark experiments. Like for the attacks, users can easily extend other defense techniques and evaluate them using our framework.
1. Differential privacy with Gaussian noise (G-DP) (woork 2006 Li et al. 2022b; Zou et al., (022)) or Laplace gradients or local model predictions to defend against attacks launched at passive or active party.
noise (L-DD) (Dwork) 2006 Fu et al.
2022a Zou et al. 2022) added to
2. Gradient Sparsification (GS) (Aji Heafield 2017 Fu et al. 2022a Zou et 2022) defends against attacks by dropping elements in gradients that are close to O)
3. Gradient Perturb (GPer) (Yang et al. 2022a) defends against label leakage attacks by perturbing the gradients with the sum of gradients from each class added with random scalars. Label-DP is guaranteed using GPer.
4. Distance Correlation (dCor) (Sun et al. 2022 Vepakomma al. 2019) defends against
attacks by applying correlation regularr laguaaniaation, When the passive party applies this defense to defend against feature reconstruction attacks, distance correlation is calculated between input features X2 and local embedding G(Xk. g: f&) at each party in order to limit redundant information of X3 kept in G(X,, 0&). J. In contrast, when active party applies this defense to defend against label inference attacks or backdoor attacks, distance correlation is calculated between label Y and G(X,,, 02) of each party to limit redundant information of Y kept in G(X², 0x). log(dCor(Xx, G(X,, 0x)) and log(dCor(Y, G(X,, 8x)) are used in practice instead of dCor dCorrXX, G(Xx,60 0%) and dCor((Y, G(xx, 0%) to stabilize training.
5. Confusional AutoEncoder (CAE) (Zou et al. 2022) defends against label related attacks by disguising labels with an encoder and reconstruct the original label with the paired decoder. Confusion is added to map one class to multiple classes to further disguise label information.
6. Discrete-gres teteee-rrrinnetenaneeeee CAE (DCAE) (Zou et al. 2022) defends against attacks by applying discrete gradients along with CAE get stronger defense.
7. Mutual Information regularization Defense method (MID) (Zou al. 2023) defends against attacks by limiting the information of label contained in local embedding.
Detailed paraeeess of these defenses can be found in Appendix H.4
H EXPERIMENTAL SETTINGS
We mainly use NVIDIA GeForce RTX 3090 for all the benchmark experiments except for tree-based VFI related experiments for which we use Intel(R) Xeon(R) CPU E5-2650 v2 instead. All the experiments are repeated times with different random seeds. The stopping criterion is determined as reaching a predefined number of epochs, while ensuring convergence at the same time. The reported results include both the mean values and the corresponding standard deviations. Other experimental settings are listed below.
H.1 MODELS AND DATASETS
We construct our benchmark experiments on the following 9 datasets. The local models used for each dataset are listed in Tab, TO Additionally, for the SpLitVFL setting, we employ a 1-layer MLP model as the global model for datasets other than Cora, for which I-layer graph convolution layer is adopted pair with the GCN it adopts local model, while non-trainable global softmax function is used for the VFL setting.
MNIST Yann LeCun). The MNIST dataset comprises handwritten digits and consists of a training set with 60, 000 examples, along with a test set containing 10, 000 examples, distributed across 10 classes. All samples have been standardized in size and centered within 32 x 32 pixel grayscale images. In our experiments, we utilize the entire dataset, and each sample is horizontally divided into equal halves and assigned to respective parties.
25

---

Published as a conference paper at ICLR 2024
Table 10: Summary of evaluated datasets under NN-based VFI.. In "#Samples" column, the values denote the number of training and testing samples separately. In "Feature Partition" column, if the number of features is equal among each party, present only one number: '[]' marks the number of features after extending discrete feature to one-hot features; '/' is used to separate the feature partition for 2-party VFL. and 4-party VFI., otherwise the number stands for the feature partition of 2- 2-party VFL. "#Parameters/ #Nodes" column, the value denotes the number of trainable parameters for each local model for neural network or logistic regression local model and denotes the number of nodes in total of each side for tree-based local model. and C' stands for the number of input features specified in Appendix H. respectively.
and #Class respectively. (p) and (a) refer to passive and active party
Feature Paritionn
Local Moddl (sctive & passive party)
Dataser
*Class
*Samples
PParometers #Nodes
MNIST Yann LeCun]
10
60000; 10006
392
MLP (1-32-C)
12.9K
CIFAKIO Knahhvasy Fiitinn 2009
10
50M0M
5127
Resnet
11.17M
CIFAR (Kohhessty
1O000
Resnet18
11. 11M
NUSWIDE (Chua el 2009
5
GODDN, WO00
10M (p), 63E (9)
(1-22
32 (p).
20.5K (a)
Breast Cancer Sttrect et ai.
1995)
2
114
15
MLP 12
2.5k
Dubhies
154
Logisric Regression
1D
Adall Income BBcker KKllvii 1996
54153; 14637
(15
(5))
MLP-447--11128-44
17.7K (pi, 22 (a)
Chiic (Jao ot al. 2017
LIAM62; 1630268
26
Wide&DDepp (Cheng el 2016
10G0 (pL. 1.22 la)
(Vu 2013
NUNGT
9 (3
Widc&Deap
2016
MMCCallomm ml 20000
140; 1000
716
CCN
kipl Weling 2017)
News2U-S5 [ang 1955
5
Jiiii,; 1000
19638
MLP-3
3.2M
Logistic Regression
26
Credit (Dua &Gmlf 2017
24000; 6000
(p). 60)
ML
7.4K (p), (a)
Ranioral worest XCBoost
5807 6200
Logistic Regression
(p 65 (a)
Nurkery Tun Gmmff 2017
10368; 2502
MLP33 (I 200- 100-C
22.2K 25.28 33.2K
RuadorsForest XGBoast
/416
CIFARIO (Krizhevsky &Hinton 2009). The CIFARIO dataset consists of 60, 000 colour images, each of size 32 32, of 10 classes, with 6, 000 images per class: 5, 000 for training and 1, 0,0 for testing. All the data are used in our experiments. Each sample is horizontally split in to equal halves and assigned to respective parties under 2-party VFL setting, or is equally split into 4 parts of size 16 x 16 and assigned to respective parties under 4-party VFL setting.
CIFARIO0 (Krizhevsky &Hinton 2009). Similar to CIFARIO dataset, CIFARIO0 100 dataset consists of 60, 000 32 x 32 colour images, but are distributed across 100 classes containing 600 images each: 500 for training and 100 for testing. All the data are used in our experi- ments. Each sample horizontally split in to equal halves and assigned to each party under 2-party VFL setting, or is equally split into 4 parts of size 16 x 16 and assigned to each party under 4-party VFL setting.
NUSWIDE (Chua et al., (009). NUSWIDE dataset is web image dataset that includes: (1) 269, 648 images and the associated tags from Flickr; (2) 5 types of low-level features extracted from the images and 1 bag of words feature of 500 dimension; (3) ground-truth label for 81 concepts. In our experiments, we use all the 5 low-level image features, that is a total of 634 features, for the active party and use the top 1000 frequent tags associated to the images for the passive party. We use only data from 'buildings', 'grass', 'animal', "water' and "person" in our experiments. For binary classification tasks, we use only data from 'clouds' and 'person'.
Breast Cancer (Street al. 1993). Breast Cancer dataset is tabular dataset consists of 30 statistical descriptions of 568 breast tumors which are categorized as either malignant In our experiments, we use 20% of the whole dataset
(cancerous) or seniin(non-caacerooe samples for testing and the rest for training. Each party possesses 15 statistical description features.
Diabetes (Kahn). The Diabetes dataset comprises 768 samples, each accompanied by 8 diagnostic measurements corresponding to individual patients, along with the diagnosis indicating whether the patient has diabetes. We also use 20% of the whole dataset samples for testing and the rest for training. Each party possesses 4 diagnostic measurements.
Adult Income (Becker &Kohavi 1996). The Adult Income dataset consists of 14 distinct fea- tures, including demographic information such as age, education, and occupation, collected from a dataset of 48, 790 individuals. The primary task associated with this dataset is to predict whether person earns an annual income exceeding 50K. We use 30% of the whole
26

---

Published as a conference paper at ICLR 2024
dataset samples for testing and the rest for training. Category features first changed to one-hot features before sending into the model for prediction. Passive party obtains the 6 non-category features well as a category feature 'workclass', while the active party controls the rest category features. After extending the categorical discrete features, 15 and 93 features are kept separately at each party.
Criteo (Guo et al. 2017) The Criteo dataset contains real world click-through data of display advertisements served by Criteo of 7 days and whether the advertisement has been clicked or not. The primary task is to predict whether clicktion is done. Only 1, 833, 625 samples from the whole dataset is used in our experiments with 90% used for training and the rest 10% used for testing following previous work (Fu et al. 2022c). Each sample has 26 anonymous categorical features assigned to passive party and 13 continuous features assigned to active party also following previous work (Fu et al. 2022c
Avazu (Q et al. 2018) The Avazu dataset contains 11 days real world click-through data from Avazu. The primary task is also to predict whether clicktion is done. We use only 808, 579 samples from the whole dataset in our experiments with 90% for training and rest 10% for testing following previous work (Fu et al. (022c). Each sample has 9 anonymous categorical features assigned passive party and 13 categorical features that are not anonymous assigned to active party also following previous work (Fu et al. 2022c).
Cora (MCCallum et al. 2000) The Cora dataset is a citation network dataset with nodes representing computer science research papers and edges representing citations between them. The task is to predict the category of a node, which falls into one of the following 7 categories: 'Neural_ NNtwolkss'''' 'Probabilisticc Methods', 'Genetic Bariic_llggmssss 'Theory's and "Rule_Learnine Learning", A total of 2708 nodes are provided in the dataset, with 140 nodes with labels and are used as training data. We use 1000 of the rest for testing. total of 1 1432 features are provided for each node and we split them equally and signed to each party.
255 (Lang) 1995) The News20 dataset, also the 20 Newsgroups dataset, is collection of approximately 20, 000 newsgroup documents, partitioned (nearly) evenly across 20 differ- ent newsgroups. We use only the first 5 categories, namely 'alt.atheism', "comp.os.me coomp.os.m--vee www.wwiiddwwiiiioo misc', our experiments, resulting in a total of 5, 000 samples. We use 80% of the data for training and the rest 20% for testing. Each party controls 49, 658 features in our setting.
pomp.graphics Nomp.sys.ibme wtt.simmmm...ddwwe.. pc.hardware' and "comp.sys.. www.ss............e....... hardware' for
Credit (Dua &raff 2017). Credit dataset comprises 30, 000 samples, each with 23 features, including attributes like age and education level. The primary objective here is to predict the likelihood of default payment. For our experimentation, we allocate 20% of the entire dataset for testing, while the remaining 80% is designated for training. While the passive party owns the 12 features related to the amount of bill statement and previous statement, the active party possesses the other 11 features related background information and repay statement.
Nursery (Dua &Graft 2017). Nursery dataset consists of 12, 960 samples of 8 features with the task of predicting the recommendation level of applications for nursery schools. We also use 20% of the whole dataset samples for testing and the rest for training. While the active party owns 4 features related to family structure and financial standings, the passive party has other 4 features.
H.2 VFL MAIN TASK TRAINING
We introduce the papaeeparameters uesd for MP evaluations and benchmarks showed in Tabs. t
and 13 in the following.
For NN-based VFL, the learning rate and training epochs use for reporting the MP listed in Tabs. 3
and and are are included in Tabs. and 12 batchsize of 1024 is used throughout all the experiments
(except for MNIST, Criteo, Avazau and News20-S5 S5 which uses batchsize of 2048, 8192, 8192, 128
respectively) and is not listed in the table. For tree-based VFL, for reporting the MP listed in Tab, 5
each party is equipped with a number of trees each of depth 6 under all circumstances. Note that learning rate is only utilized for XGBoost and is set to 0.003 in the experiments.
27

---

Published as a conference paper at ICLR 2024
Table 11: Hyper-parameters for Tabs. and
13
LR is short for learning rate.
Dataset
VLL,
FedSGD epochs
VLL
FedBCD epochs
splitVFL,
FedSGD
splitVFL, FedBCD
LR
LR
LR
epochs
LR
epochs
MNIST
0.01
30
0.005
30
0.01
30
0.005
30
CIFARIO )
0.001
30
0.001
30
CIFARIO (4-parly)
0.001
30
0.001
1024
CIFARION 100 (2-party)
0.01
30
0.01
30
CIFAR TO0 (4-party)
0.001
40
0.001
40
NUSWIDE
0.003
10
0.003
10
0.006
10
0.003
10
Breast Cancer
0.05
50
0.005
50
0.05
50
0,005
50
Diabetes
0.05
80
0.01
80
0.05
80
0.01
80
Adult Income
0.01
50
0.001
50
0.01
50
0.001
50
Criteo
0.0001
5
0.0001
5
Avazu
0.0001
10
0.0001
10
Cora
0.01
20
0.01
20
News20-S5
0.001
80
0.002
80
Credit (LR)
0.6
300
0.6
300
Credit (NN)
0.01
40
0.01
40
Nursery (LR)
0.5
40
0.5
40
Nursery (NN)
0.01
40
0.01
40
Table 12: Hyper-parameters for Tab. 6
Note that the hyper-parameters for the FedSGD and FedBCD
Tab, are the same that Tab.
Quantize
Top-k
CELU-VFL epochs
Dataset
LR
epochs
LR
epochs
LR
5
MNIST
0.03
30
0.05
30
0.5 0.008
30
NUSWIDE
0.003
10
0.003
10
0.008
10
0.8
H.3 ATTACK HYPER-PARE PAREER-PAAMMEERSS
Our evaluated attacks can be categorized into Label Inference (LI) attacks, Feature Reconstruction (FR) attacks, Targeted Backdoor (TB) attacks and Non-targeted Backdoor (NTB) attacks. Each attack is launched separately on the VFL systems trained with the hyper-parameters listed above in Appendix H.2 Specific hyper-parameters for each attack is listed in below if exist.
For LI attacks, attacker is the passive party. In NS (Li al. 2022b), DS (Li al. 2022b)
and DLI (Li et al. [2022b] Zou et al. 2022), no other attack related typer-parameter needs to be specified. However, for MNIST dataset, in order to achieve higher MP, we use learning rate of 0.001 for VFL model training under binary classification tasks (DS attack). In BLI Zoou al. 2022), learning rate and number of training epoch for the inference model are set 0.05 and 10000 respectively.
AMC and PMC (Fu et al. 2022a), we randomly take 4 training samples from each class to form the auxiliary labeled dataset for training of the "completion model". Both the trained local model and the classification head used to complete the model are fine-tuned using the auxiliary labeled dataset and some non-labeled data from the training dataset. Number of "completion model" training epochs, learning rate and training batchsize are set to 25, 0.002, 16 separately for MNIST and CIFAR 10 datasets and set to 20, 0.002, 16 separately for NUSWIDE dataset. Note that for better attack stability, we change the VFL main task learning epoch to 100 for PMC and AMC under all the datasets.
For FR attacks, attacker is the active party. In GRN (Luo al. 2021), ) following the original paper, the reconstruction model is trained for 60 epochs with batchsize of 1024 for all datasets, well as learning rate of 0.005 for MNIST dataset and 0.0001 for CIFAR 10 dataset. While TBM (Li al. 2022a), the reconstruction model trained for 50 epochs with a learning rate of 0.0001 and a batchsize of 32. Note that, since TBM requires auxiliary data with the same distribution as the training data, we use 10% of the training data form the auxiliary dataset and train the VFL model using only the rest 90% following the original work (Li et al. 2022a)
28

---

Published as a conference paper at ICLR 2024
For TB attacks, LRB (Zou al. 2022 is evaluated with the passive party being the attacker. Target class randomly selected for each training. Following previous work (Zou et al. 2022). 1 % of the data is randomly selected and attached with specific trigger of 4 pixels to form backdoor samples for MNIST and CIFARIO datasets, while samples that have value 1 at the last bit of the tag data, which take up less than 1% of the whole dataset, are treated as backdoor samples for NUSWIDE dataset.
For NTB attacks, attack is done by the passive party inference time. In NSB (Zou et al. 2023), 1% of the testing samples are randomly selected and added with random noise sampled from (0, 2) by the attacker party. In MF (Liu al. 2021b), a missing rate of 0.25 is evaluated which means during inference, a quarter of the intermediate local model output from the passive attacker is replaced with before transmitted to the active party for aggregation.
H.4 DEFENSE
To ensure a comprehensive benchmark, we evaluate a total of 8 defense methods using our proposed benchmark pipeline. For each defense method, we systematically test different parameters to ensure thorough and adequate evaluation. When defenses are applied to a VFL under particular attack, all the hyper-pate pareee-ersmeeerss are not changed compared to that when that attack is applied. Detailed defense related experimental hyper-peee apprr-parmmttr settings are listed below:
L-DP or G-DP (Dwork 2006; Li et al., 2022b Zou et al., 2022), gradients are first 2-norm clipped with 0.2 when the defense is applied at the active party. When passive party applies this defense, noises are added normalized local prediction results. Then noise with "diversity" scale parameter or standard deviation ranging from 0.0001 to 1.0 is added clipped gradients.
In GS (Aji &Heafield 2017 Fu et al., 2022a; Zou al. 2022), drop rate ranging from 95.0% to 99.5% are evaluated in the experiments.
In GPer (Yang et al. 2022a), conduct tests with various standard deviations for the noise variable u with the value of range 0.0001 to 0.1.
In dCor (Sun et al.
2022
Vepakomma et al. 2019), we set distance correlation regularizer to 0.3 and record the corresponding results.
coefficient À from 0.0001
In CAE (Zou al. 2022).
we consider various confusional levels with A2 value ranging from 0.0 to 1.0 while in DCAE (Zou et al.) 2023), we select the same set of confusional levels and fix the bin number for gradient discrete to 12.
In MID Zou al. 2023), we test the scenario where both active party and passive party apply MID, which suit the real world application since both sides need to protect themselves against potential attacks. Hyper-parameter that signifies the strength of the defense ranges from 0.0 to 1000.0.
H.5 C-DCS CALCULATION DETAIL
T-DCS values and C-DCS values are all calculated with respect to 1 single dataset. All T-DCS scores. iec T-DCSL,, T-DCS T-DCS TT-DCS I-DDSp DTDDCCPPR.-DDSS/... T-CCTTDD T-DCS are calculated by averaging the DCS values of each evaluated attack belonging to that type under the particular dataset. Then C-DCS values calculated with Eq. 3 using only T-DCSu, wtw.:TDDDSSS....SSSr......... T-DDS
I ADDITIONAL EXPERIMENTAL RESULTS
I.1 ADDITIONAL MAIN TASK PERFORMANCE RESULTS
We place the comparison results of MP and communication rounds (#Rounds) of NN-based VFL with different local models in Tab. 13 here in the appendix. Together with Tab. 5 we compare the performance of linear regression, tree and neural network model architecture under 2 different datasets.
29

---

Published as a conference paper at ICLR 2024
Table 13: Comparison of MP and communication rounds using different local models in NN-based (Ye et al. 2022b) for the selection of
VFL with FedSGD communication protocol. We mainly follow
MLP model architecture.
Credit
Nursery
Linear Regression
Neural Network (MLP-4)
Linear Regression
Neural Network (MLP-3)
aggVFl.
MP #Rounds
0.820+00001
0.826+0.001
0.938+0.001
0.999+0.001
22=10
56::2
58#19
77
MP #Rounds
0.821=0.0000
0.826:10.001
0.0.05
0.999=10.001
splitVFL.
21:1.6
85:1:20
52:14
102:1
Table 14 presents the MP and running times with and without encryption. The number of epochs for Credit with LR is reduced 40 since the training with HE takes a lot of time. The MP is different from those of Table 13 since the experiment of Table 14 uses the logit-loss even for multi-class classification. In terms of MP, a slight performance degradation occurs due to the approximation error caused by Taylor Expansion. The increase in running time due to encryption is more than 1000 times, do not currently support GPU for matrix operations on encrypted values.
Table 14: MP and execution time under agg VFL with Homomorphic Encryption.
Dataset
Lincar Regression (wlo eacryption)
Lincar Regression (w/ encryption)
Credit
MP
0.821 = 0.000
0,805 0.002 11149 1. 357
Exec, Time [s]
3 3 0
Nursory
0.918 0.001
0.912 0.001 2971 57
Exec. [s]
0
1.2 ADDITIONAL ATTACK AND DEFENSE PERFORMANCE WITH MORE DATASETS
Due to space limit, we place the plot for MPs and APs under MNIST (Fig. I and NUSWIDE (Fig. [2] datasets in this section with detailed analysis of the results. A brief summary of the below analysis is already included in Sec. 6.2
1.2.1
MP AND AP
MID defense is capable of achieving relatively lower AP while maintaining a similar MP compared to most other defenses, demonstrating its effectiveness in defending against a wide spectrum of attacks by reducing the information of label Y and local feature X, kept in Hp with a mutual information (MI) regularizer.
dCor targets at attacks that explore the information contained in Hp for deducing Y or X, by limiting the distance correlation between H, and Y or X, for defending against attacks launched by passive active party respectively. This defense ideology is similar to that of MID, which directly regularized the mutual information between Y and Hp. MID performs better than dCor on gradient-based LI attacks like DS, DLI and BLI, likely due to the fact that limiting the MI between Hp and Y simultaneously limits the dependency between the correlated gradient 9p at passive party and Y, which is better than reducing the distance correlation between H, and Y. Additionally, when compared other attacks, dCor appears less effective in limiting AP under NTB attacks across all the 3 datasets which is reasonable since this defense is not designed for defending against attacks that introduce information loss into transmitted data.
CAE and DCAE focus on disguising label by mapping real one-hot labels to soft-fake labels. CAE consistently performs well across all datasets when defending against DLI, BLI and LRB attacks, which utilize merely the information of the current sample without any auxiliary information or data, For PMC and AMC attacks that rely on auxiliary label information, DCAE is more effective with the help of information reduction from quantization of gradients. The limited effectiveness of DCAE against DS attack can be attributed to the fact that the quantization is not effective in perturbing the direction of the gradient, while DS attack relies on the cosine similarity between the gradient of each sample and a known sample. These 2 attacks are not designed for FR and NTB attacks.
GS is a defense that injects information loss by sparsifying gradients, i.c. setting gradient elements with small absolute value 0.0, during training. As the gradients, which is directly related to labels but not input features, that GS modifies to defend against potential attacks, we do not benchmark its performance FR attacks following previous works (Luo al. 2021 Li al.
30

---

Published as a conference paper at ICLR 2024
2022a). GS shows strong defense ability for most of the LI attacks but exhibits less than satisfactory defense results against LRB attack, which replace the gradients of selected samples with those related to target labels to alter their labels. This is likely due to the fact that GS still preserves the gradient elements with larger absolute values which are responsible for learning the triggered patterns injected by the backdoor attack.
DP-G and DP-L defenses that inject noise gradients local model output defend against attacks launched by passive or active party respectively. They show similar trend and display a clear trade off between MP and AP in all the attacks.
GPer targets at defending label inference attacks so we apply it only on label related attacks, i.e. LI and LRB attacks. GPer guarantees label-DP by perturbing the gradients and performs similarly to DP-G and DP-L in defending against these attacks, consistent with the original work (Yang et al. 2022ab which shows that GPer performs the same as adding isotropical noise to gradients.
LI-DS(2-class)
U-DU((00-class)
LI-BL((1O-class)
LI-PMC(10-class
14
1.0
0.9
0.
0.
19KK SIDE LIDO C9R0 Edsa
626
2:20
6N
030
512
3.36
Gol
0.30
6G
006
LI-MCC(1--CasS,
FR-GRN(10-class)
FR-TBM(10-Class)
TB-LRB(10-class)
com
0.50
66
64
6543
82
6.4
6.85
84
6040
NTB-NSB(10-class)
(11--M((00-claS
0.87
G-DP
CAE
DCAE
0.33
GS
MID
defense
038
232
334 3.31 DU0 ESD
EXT
Figure 11: MPs and APs for different attacks under defenses [MNIST dataset, VFL, FedSGD]
I.2.2 VISUALIZATION OF TBM ATTACK
We show the visualization of reconstructed features of TBM attack under various defenses with MNIST dataset in Fig. 13, All the defense hyper-parameters selected are the most effective one with highest DCS among the particular defense method, demonstrated in Fig. [T (sub-figure for TBM) and Tab. [5 It's clear from Fig. [3 that, although the reconstructed images are noisy when defense exits, the contour can still be seen easily. This implies the difficulty for defending against TBM attack which exploits auxiliary i.i.d. data for feature reconstruction guidance.
I.2.3 DCS, T-DCS AND C-DCS RANKING
We also place the full T-DCS and C-DCS table for MNIST (Tab. 15), CIFARIO (Tab. 16.
NUSWIDE
(Tab. [7) datasets here due to limit of space. Basic analysis has already been included in Sec. 6.2
The overall relative rankings all the defenses are still similar that of other datasets.
We also show the change in C-DCS ranking under different 33 for CIFAR 10 and NUSWIDE here due to space limit. Similar results can be seen from Figs. 14 and 15 while the ranking fluctuates more
31

---

Published as a conference paper at ICLR 2024
U-NS((-class)
U-DS(2-class)
L--LLIS-class)
UIBLI((S-llase
0.55
6433
0.7
6393
0.53
LI-PMC(S-class)
LI-MM((S-class)
FR-TeM(S-class)
TB-LRB(5-class)
0.30
0.00223
0.70
1.0613
0.00
0.3
0.3
0.30
D15 AUTH DAES DSS ARE
Par 0.N FAST one oxn
37%
2331
0.11
NTB-NSB(S-class)
NTB-MF (TB-Ma(5-
140
L-DP
DCAE
GS
MID
w/o defense
120
dCor
375
0.80
372 634 076 TR
Figure 12: MPs and APs for different attacks under defenses INUSWIDE dataset, agg FedSGD)
No Defense
5
C
G-DP
G
0.1
L-DP
0.1
dCor
0.3
MID 0.01
Figure 13:
Visualization of the feature reconstruction results of TBM attack for MNIST dataset.
All the defense apper-parameters selected the most effective ones with highest DCS among the particular defense method.
under NUSWIDE dataset compared to the other 2 datasets, probably because the larger drop of MP for all the defenses when applied NUSWIDE dataset.
32

---

Published as a conference paper at ICLR 2024
6020
0.0%0.0
LOPA1
Rank
LEPOUEES
2000
17-
0.95
0.99
Figure 14: Change of C-DCS ranking with the change of B3. [CIFARIO dataset, agg FedSGD]
COCS
LO0 03
Pank
COCAL
DOCO
17
0.1
0.3
0.5
0.7
0.9
0.95
0.99
Figure 15: Change C-DCS ranking with the change of 8. [NUSWIDE dataset, Fed,
U-DS/2-class)
LL-DLI10-ciass))
LLPMCC10-ccass)
0.30
0.00
0.20
0.15
0.10
-0.15
0.0
0.00
8 of ged, o
0/ of db tel
5 6 geth Sh Pugd
1-AMC(10-c65s)
2
FR-CRNI M0RNN10--1190
FR-TBM (0-less)
Be T-- (1-cbb
0.25
0.00
0.100
0.075
-010
0.025
0.15
00003
0.000
0.023
6-5P
6/8
NTD
clccss
0.20
0.04
0.03
C/OP
Use
0.10
0.03
MCC
0.02
Che
000
0.02
0.00
1.9
6
6
Figure 16: DCS gap for each datack-defenn point [MNIST dataset, spliVVFLaggVVFL..eed000
33

---

Published as a conference paper at ICLR 2024
Table 15: T-DCS and C-DCS for All Defenses IMNIST dataset, agg FedSGD]
Defense
Name
Defense Parameter
T-DCSE,,
T-DCSE
T-DCStr
T-DCSER
T-DCSyn
T-DCSN7n
C-DCS
Ranking
MID
100
0.7371
0,8810
0.8523
0.6184
0.9077
0.9093
0.8217
MID
1.0
0.7395
0.8731
0.8464
0.6186
0,9080
0:9070
0.8200
MID
10000
0.6989
0.8939
0.8549
0.6145
0.9071
0.9026
0,8197
L-PP
0.1
0,7407
0.8566
08334
0.6190
0,8957
0.8855
0,8084
MID
0.1
0.6937
0.8577
0.8249
0.6205
0.8961
0.8885
0.8076
5
L-DP
0.01
0,7244
0.8577
0.8310
0.6029
0.8920
0.8918
0,80444
MID
0.01
0,8185
0.8504
G-DP
0.1
0.7392
0.8528
08:01
0.6220
0.8913
0.8208
0.2910
G-DP
0.01
0.7350
0.8461
0.8238
0.60226
0
0.8167
0.78
9
MID
0.0
0,7335
0,8058
0.7913
0.6095
0,7862
0.8676
0,7636
MID
1c-06
0.7382
0,8048
0.7914
0.6125
0.7798
0.8680
0.7629
11
MID
08
0.7382
0,8063
0.7927
0.6110
0.7714
0.3669
0,7605
12
MID
0.0001
0.6564
0.8057
0.7742
0.6168
0,7720
0.8653
0.7571
L.DP
0,001
0.7720
0.7911
dCor
0.3
0.5901
0.7999
0.7579
0.6083
0.8277
0.8241
0.7545
15
dCor
0.1
0.5881
0,7241
0.6999
0,6046
0.8234
0.8203
0.7366
16
G-DP
0.001
0,7038
0.7564
0.7677
0,8200
dCor
0.01
0.5858
0.6305
0.6216
0.6022
0.7999
0.8189
0.7106
18
L-DP
0.0001
0.5858
0.6727
0.6553
0.6022
0.6533
0.82388
016834
19
G-DP
0.0001
0.5858
0.6632
0.6477
0.6026
0,6640
0.8196
0.6835
dCor
0.0001
0,5858
0.6258
0.6178
0.0020
0.6645
08816
0.6751
21
GS
99.5
0,7358
0.8873
0.8576
0,7189
0.8454
G8
99.0
0,7388
0,8780
0,852
0,7038
0.8313
GS
97.0
0.7388
0.7573
0.7536
0.6825
0.8305
GS
95.0
0.7388
0.7232
0.7263
0.6732
0.8268
CAE
1.0
0,5858
0.7960
0.7539
0,7632
CAR
0,5858
0,7124
CAE
0.5
0.8858
0.7957
0.7521
0.7280
CAE
0.0
0,5858
0.8218
0.7746
0.6837
DCAE
1.0
0.5858
0,9021
0,8389
0.7648
DCAE
0.5
0.5858
0.9039
0,8402
0.7396
ICAF
0.1
0.5858
0,8990
0.83664
0,7299
DCAE
0.0
0,5858
0,9094
0.8447
0,7111
GPer
10.0
0,5903
0,7016
0.6794
0.8031
GPar
1.0
0.7306
0,8097
0.7939
0.9045
GPer
0.1
0,7357
0.8546
0.8308
0.8926
GPer
0.7360
0.8525
0.8857
Table 16: TDCC and C-DCS for All Defenses [CIFAR10 dataset, agg FedSGD]
Defense Name
Defense Parameter
T-DUS T-DCSE,,
T-DCStr
T-DCSFR
T-DC'yn
T-DCSw7n
C-DCS
Ranking
MID
0.01
0.7232
0.9172
0.8784
0.6035
0.8942
0.9286
0.8262
MID
1.0
0.7260
0,9173
0.8791
0.6039
0.8931
0.9286
0.8262
MID
100
0.7270
0.9161
0.8733
0.6039
0,8931
0.9280
0,8258
MID
10000
0.7276
0,9159
08782
0.6016
0.8945
0.9284
0,8257
MID
0.1
0.7001
0.9175
0.8700
0.6016
0.8906
0.9284
0,8237
MID
Ic-06
0,7132
0,8600
0.8306
0.6024
0.9123
0.9323
0.8194
MID
1s-08
0.7132
0.8594
0.9123
MID
0.0
0.7132
0.8552
088288
0.6123
0.9121
0.9289
0.8175
MID
0.0001
0,7118
0,8575
0.8283
0.6034
0.906
0.9302
0.81 77
9
G-DP
0.01
0,7316
0,8965
86835
0,6019
0,8809
0.8154
L-DP
0.01
0.7365
0,8987
0,8663
0.6021
0.8751
0.9146
0.8145
11
L-DP
0.001
0.7041
08787
0.8437
0.6023
0.8892
0.9211
0.8141
12
G-DP
0.1
0,7324
0.8947
0.8622
0.6097
0,8670
0.9083
0.81
G.DP
0.001
0.6704
0,8713
L-DP
0.1
0.7353
0,8919
8606
0.6034
0.8696
0.9058
0.8098
15
dCor
0.3
0.6116
0.8109
0.7710
0.6012
0,8883
0.9245
0.79K'3
16
0.5879
0,7695
0.6030
0.8962
L-DP
0.0001
0.5876
0.7611
0.7264
0.6023
0.9049
0.9083
0.7855
G-DP
0.0001
0.5868
0.7467
0.7166
0.6025
0.9046
0.8993
0.7807
19
aCor
0.0001
0.5858
0.6578
0.6434
0.6031
0.9162
0.9558
0.7796
dCor
0.01
0,5859
0,6839
0.6643
0.6042
0,9162
0,8982
0,7708
GS
99.5
0,7387
0.8956
0.8642
0,8877
0.9276
G8
99.0
0,7388
0,8810
0.8526
0,8838
GS
97.0
0.7388
0.8367
0.8171
0.8882
0.9286
GS
95.0
0.7388
0,8049
0.7916
0,8895
0.9295
CAE
1.0
0.5918
0.8592
0.8057
0,8482
CAE
0.5
0,8633
0,8079
0.8557
CAB
0.1
0.8858
0,8498
0.7970
0.9179
CAF
0.0
0.5858
0,8419
D. 0.7907
0.9214
DCAE
1.0
0.6080
0,8678
0.8464
DCAE
0.5
0.6131
0,8780
0.8250
0.8556
DCAE
0.1
0.6090
0,8977
0.8400
0.9039
DCAE
0.0
0.6090
0,9038
0.8449
0.9066
GPer
10.0
0.6422
0,8082
0.7750
0.8957
GPar
1.0
0.6976
0,8913
0.8526
0.8838
GPe
0.1
0,7385
0.8946
0.8634
0,8715
GPe
0.7392
0.8931
0.8700
34

---

Published as a conference paper at ICLR 2024
Table 17: T-DCS and C-DCS for All Defenses INUSWIDE dataset, agg VFI., FedSGD]
Defense Name
Defense Parameter
T-DCSp,,
T-DCSE,,
T T-DCSrn T-DCSpR T-DUSS
T-DCSSwT
C-DC'S
Ranking
MID
10000
0.7358
0.8559
0.8159
0.5833
0.7333
0.8707
0.7508
MID
1.0
0.7476
0.8472
0.8140
0.5833
0,7331
0.8700
0.7501
MID
100
0.7320
0.8536
0.8130
0.5833
0.7326
G-DP
0.1
0.7375
0.8262
0.7966
0.5863
0.7282
0.8675
0.7447
L.DP
D.1
0.7389
0.8177
0.7915
0.5863
0.7258
0.86013
0.7410
MID
0.1
0.7516
0.8259
0.8011
0.5833
0.7172
0.8563
0.7395
MID
0.01
0,7280
0,8092
0.7822
0.5844
0.2151
0.8627
0.7361
MID
0.0001
0.7144
0,8097
0.7779
0.5855
0.7040
0.8680
0.7339
dCor
0.3
0.7641
0.8411
0.8155
0.5834
0.7289
0.8051
0.7332
G-DP
0.01
0.7991
0.76000
0.7530
0,2061
0.7251
L-DP
0.01
0.7395
0.7525
0.7482
0.5863
0.7148
0.8485
0.7244
11
MID
1c-06
0,7022
0.8201
0,7808
0.5860
0 0.6880
0,8408
0.7239
12
dCor
0.1
0.7442
0.7617
0.7559
0.7259
MID
1s-08
0.7066
0.8147
0.7787
0.5862
0.6593
0.8410
0.7163
MID
D.O
0.6599
0,8097
0.7598
0.6862
0.6590
0,8414
0.2116
15
L-DP
0.001
0.7291
0.7234
0.7253
0.6424
G-DP
0.001
0.7175
0.7237
0.7216
0.5863
0.6379
0.83344
0.6948
17
JCor
0.01
0.7445
0.7021
0.7162
0.5863
0.6336
0.8295
0.6914
18
L-DP
0.0001
0,6783
0.6470
0,6574
0.5863
0.6313
0.8293
0.6761
G-DP
0.0001
0.6495
0.6381
0.6419
0.8863
0.6309
0,8290
0.6720
20
JCor
0.0001
0.6496
0.6340
0.6392
0,5864
0.6307
0,8287
0.6712
21
GS
99.5
0
0.7888
0.6456
GS
99.0
7.7004
0.8000
0.7841
0.6415
0.8408
GS
97.0
0.7414
0.7672
0,7586
0.6376
0.8392
GS
95.0
0.7423
0.7399
0,7407
0.6375
0.8385
CAE
1.0
0.6863
0.7822
0.7502
0.6830
CAE
0.5
0.6808
0.7848
0.7501
0.6733
CAF
0.1
0 6868
D.8249
0.7768
0.6734
CAE
0.0
0.6808
0.8212
0.7744
0,6807
ICAE
1.0
0.6716
0.8156
0.7676
0.6771
DCAE
0.5
0.6672
0,$108
0,7629
0.6668
DCAE
0.1
0.6669
0.8651
0.7991
0.6746
IXAE
0.0
0.6669
0.8660
0.7996
0.68166
GPer
10.0
0.6877
0.6722
0.6773
0.6222
GPer
1.0
0.7230
0,7383
0.6315
GPer
0.1
0.7395
0.8007
0.7803
0.7042
GPer
0.01
0.7386
0.8412
0.8070
0,7193
Table 18: T-DCS and C-DCS for All Defenses IMNIST dataset, splitVFL. FedSGD]
Defense Name
Defense Parameter
T-DCS
T-DCSLL.
T-DCSu
T-DGSER
T-DCStB
T-DCS
C-DCS
Ranking
MID
1.0
0,7380
0,8846
0.8553
0.6065
0.9051
0:9237
0.8226
L-DP
0,7398
0,8700
0,9092
G-DP
0.1
0.7350
0.8654
0.8393
0.6139
0,9058
0.9271
0.8215
MID
10000
0.7372
0.8921
0,8611
0.6064
0.9071
0.9062
0.8202
MID
100
0.7371
0.8896
0.8591
0.6082
0,9085
0.9034
0,8198
L-DP
0.01
0.7381
0,8603
0.835
0,6026
0.9099
0.9147
0.8158
MID
0.1
0.7398
0.8652
0.801
0.6241
0.9113
0.8871
0.8156
G-DP
0.01
0,7393
0.8522
0.8328
0.6031
0.9060
0.9170
0.8147
&Cor
0.8876
0.8288
L-DP
0
0.7189
0.8365
0.8130
0.6038
0,8682
0.8663
0.7878
10
MID
0.01
0,7364
0.8455
0.2377
0.6140
0,8421
0.8536
0.7834
11
G-DP
0.001
0.7404
0,8269
8096
0.6041
0,7755
dCor
0.5981
0,8519
0
0.6133
0,8519
0.7719
MID
1c-06
0.7388
08810
0.7990
0.6173
0.7641
0.8394
0.7549
14
MID
1c-08
0,738%
0.8158
0,8004
0.6135
0,7615
0.8421
0.7544
MID
0.0
0.7388
0,8147
0.7524
dCor
0.01
0.5870
0,8020
0.7590
0.6048
0.8111
0.8174
0.7481
17
MID
0.0001
0.7388
0,8137
0.7988
0.6182
0,7324
0.8312
0.7451
18
L-DP
0.0001
0.6472
0.6046
0,7699
G-DP
0.0001
0.6253
0.7972
0.7628
0.6036
0.7586
0.8212
0,7368
&Cor
0.0001
0,5858
0.7906
0.7496
0.6053
0.7558
0.8164
0.7313
21
GS
95.0
0,7388
0,8080
0,7380
GS
97.0
0.7388
0,8133
0.7984
0.7355
0.8428
GS
99.0
0,7388
0,8334
0.8145
0,7590
0.8558
GS
0.7388
0,8487
0.8267
0,7872
0.8636
CAE
1.0
0.5858
0,8252
0.7773
0.7741
CAB
0.5
0.5858
08222
0.7797
0,7439
CAE
0.1
0,5858
0.8365
0.7863
0,7535
CAR
0.0
0.5858
0,8410
0,7574
DCAE
1.0
0.5858
0.7824
0.7431
0.7882
ICAF
0.5
0,5858
0,7862
0.7461
0,7882
DCAE
0.5858
0.7773
0,7882
DCAE
0.0
0.5858
0.7758
0.7378
0.7882
GPer
10.0
0.5916
0,8037
0.7613
0.8490
GPer
1.0
0.7295
0.8460
0.8227
0.9064
GPer
0.1
0.7369
0,8632
0.8379
0.9039
GPer
0.01
0.7365
0.8712
0.8442
0.9025
35

---

Published as a conference paper at ICLR 2024
Table 19: T-DCS and C-DCS for All Defenses [MNIST dataset, agg FedBCD(Q=5))
Defense
Defense Parameter
T-DCSE,,
T-DCStr T-DCSm
T-DCS#R T-DCSrn
T-DCSw7n
C-DCS
Ranking
Name
MID
1.0
0.6707
0.8794
0.8376
0.6166
0.9034
0.9399
0.8243
MID
10000
0.6917
0.8729
0.8366
0.6133
0.90134
0.9417
0,8238
MID
100
0.6908
0,8750
0,9039
MID
0.1
0.6121
0,8300
08208
0.6199
0,9030
0.9413
0.8212
L-DP
0.1
0.7403
0,8369
0.8176
0.6525
0.8669
0.8506
0.7969
5
G-DP
0.1
0,7385
0.84012
0.8199
0.6262
0,8704
0.8532
0,7924
G-DP
0.8376
L-DP
0.01
0.7387
0.8331
0884
0,0055
0.8959
0.8472
0.79977
MID
0.01
0.6119
0.8357
0.7910
0.6166
0.8075
0.9118
0.7817
L-DP
0.001
0.6821
0,7915
0.7696
0,6033
0,8447
0.8668
0.2711
G-DP
0.001
0.7132
0.7836
0.7696
0.6051
0,8032
0.8527
0.7576
11
dCor
0.3
0.5900
0.8329
0.7843
0.6071
0,8000
0.8154
0.7517
12
MID
1e-06
0.6156
0,8089
0.7702
0.6144
0.7441
0.8727
0,75004
MID
0.6155
0.6217
MID
0.0
0.61556
0.8072
0,7689
0.6176
0.7396
0.8342
0.7451
15
MID
Ic-08
0.6156
0.8092
0.7705
0.6221
0,7237
0.8522
0.7422
dCor
0.5960
0.7971
0.7569
0.6077
0,6955
L-DP
0.0001
0.5858
0.7246
0.6969
0.6044
0,7083
0.8241
0.7084
G-DP
0.0001
0.5858
0.7194
0.6927
0.6134
0,69963
0.8196
0,7030
19
OCor
0.01
0.5858
0.6900
06699
0,6894
0.6940
dCor
0.0001
0,5858
0.6763
0,6522
0.0085
0,6144
0.8086
0.6859
GS
99.5
0,7358
0,9005
0.8682
0,7086
0.8587
0,7388
0,7093
GS
97.0
0.7388
0.7718
0.7652
0.6683
0.8483
GS
95.0
0.7388
0,7458
0.7444
0.6585
0.8359
CAR
1.0
0,5858
0,9319
0,7450
CAR
0.5
0,9367
CAE
0.1
0.5858
0,9664
O 8903
0.6757
CAE
0.0
0,5858
0,9681
0.8917
0.6724
DCAE
1.0
0.5858
0,7311
DCAE
0.5
0.5858
0.8776
0.8192
0,8041
DCAF
0.1
0,5858
0,8867
0.8265
0.6619
DCAE
0.0
0.5858
0,8919
0,8307
0.6896
GPer
10.0
0.5901
0,7562
0,7230
0.7515
GPer
1.0
0,7317
082216
0.8036
0
GPer
0.1
0,7341
0.8398
0.8186
0,8829
GPer
0.01
0.7366
0,8381
0.8178
0,8683
defensee defense
125
-02
Al:ocSopp Robcctogg)
TE:0C5 (CCCagg
N1B:3C50pp (-pCClegg))
Figure 17: DCS gap Distribution, y-axis represents density (NUSWIDE dataset, VLL FedSGD]
1.2.4 ADDITIONAL RESULTS ON SPLITVFL AND AGGVFL COMPARISON
We further expand the discussion of the conclusions on the comparison between SpLit and Vgg as well as the comparison between FedBCD and FedSGD given in Sec. here due to space limitations.
The conclusion that when no defense is applied, splitVFL system is less vulnerable to attacks than Vg is also evident by the results under NUSWIDE dataset, with all the black square points in Fig. appeering above or close to the red horizontal line at a value of 0.0 and the blue histograms appearing mostly the right of the vertical line at value of 0.0 in Fig. I7 indicating positive DCS gap. This shows that global trainable model can be beneficial to model robustness and safety. Specifically, the DCS gap is pronounced for attacks that directly exploit the gradients, i.e., DLI and BLI attacks.
The conclusion that spliVVFL. has an overall positive effect on boosting defense performance against atlacks is also shown from Figs. and 17 in which most DCS gaps for defenses are positive, especially
36

---

Published as a conference paper at ICLR 2024
NS/2 class)
US/2 class)
DLIS cless)
UI BLXS class)
0.20
6.08
0.06
033
5:04
0.10
0.02
0.03
6:00
0.00
36
-0.02
-0.3
0,000 cF Nog LI-MMCIS-c (-55)
0 0 &
c5 2
U.AMC(S-CGEs)
Te. [B-
8.00
0:00
00000
0.000
0.06
6.04
20005
0.015
c.22
00000
0.010
0.50
c.0
p0o2s
n.om
4.02
-0.00
-0.04
G 1.P. of 2a
6 OF get.oll wo Ras
6C
NTB NTB-NSB(5--eess)
NTB-MF/5-clisg)
w/o attack(5-cless)
0.003
8.00
G-OP LOF
DCAE
8.06
0000
0.003
0.52
022
004
6-0 L.0
dter
Figure 18: DCS gap for each attack-defense point INUSWIDE dataset, splitVFL/aggVEL, FedSGD]
for LI, FR and TB attacks. This implies that splitVFL architecture exhibits greater robustness against potential attacks. However, not all defenses are enhanced in splitVFL setting. For example, in Fig, [6 MID results in minor negative DCS gaps in several attacks like GRN and NSB, so is GS in MC attacks.
The DCS ranking under splitVEL setting with FedSGD communication protocol using MNIST dataset is also included in Tab. 18, which is quite similar that in Tab. [5 under agg VFL setting with FedSGD communication protocol using MNIST dataset. This indicates the robustness of our DCS evaluation metrics as well as the inner consistency of defense abilities under different settings.
1.2.5 ADDITIONAL RESULTS ON FEDBCD AND FEDSGD COMPARISON
The conclusion given in Sec. 6.2 that a system is less vulnerable to attacks under FedBCD setting when no defense method is applied is also evident from Fig. 19 with all the black square points, except the one for MF attack, appear on or above the horzzontal line at a value of 0.0. Also, the vulnerability of a system under FedBCD setting to attacks differs between different attacks. Specifically, as shown Figs. 6 and [9] evaluating with MNIST dataset, VFL trained with FedBCD is much less vulnerable to BLI attack, which exploits batch-level gradient to recover labels. This is because gradients from earlier epochs of un-trained model shared under FedSGD reveal more information about labels compared to FedBCD which only shares gradients every Q > iterations. Similar results can be seen from Figs. 20 and 21 that compares the DCS between FedSGD and FedBCD under agg VFL setting with NUSWID dataset.
The DCS ranking using FedBCD communication protocol under agg VFL setting using MNIST dataset also included in Tab. 19 which is quite similar that in Tab, 15 using FedSGD communication protocol under agg VFL setting using MNIST dataset. This also indicates the robustness of our DCS evaluation metries as well as the inner consistency of defense abilities under different settings.
1.2.6 CONSISTENCY OF C-DCS RANKING.
We conduct a comparative analysis of the C-DCS rankings that are presented in Tabs. 15 to 19 across multiple datasets, communication protocols, and model partition strategies. The mean and standard
37

---

Published as a conference paper at ICLR 2024
LI DS12 class)
LI
BLI(1O class)
PMC class)
0.43
0.01
0.20
0.00
0.00
0.02
0.00
36
-0.0
0.12
0.00
-0.00
0 b
6 S
05
gg R TB-LABi (o-Pisss)
(1ARC (0-chass)
FR-GRN(1--Ca5S
FR-TBM(1-<cas)
0.00
0.00
6,0055
0.04
0.02
0.00
0.02
06
0.00
6 38 & 2o
200f
v00
XTB-N55(10-Zleg
NTB-MF(10-2) (10-class)
who attack (alacc(10-cisss))
0.300
0.00
DCAE
-0.01
05
-0.3
0.02
0.04
Figure 19: DCS gap for each attack-defense point [MNIST dataset, agg VFL, VL
celennee
Figure 20: DCS gap Distribution, y-axis represents density [NUSWIDE dataset, VFL, Fed- BCD/FedSGD]
deviation of the 5 rankings for each defense are calculated and visualized in Fig. 22 Remarkably, despite the diversity datasets, communication methods, and model partitioning, the variations in the rankings remain at a low level. This suggests that the C-DCS rankings are generally consistent across various datasets, communication protocol and model partition settings, and that the relative performance of different defense methods is relatively stable under different datasets and settings.
J SOCIAL IMPACT OF THE WORK
Our work introduces an extensible and lightweight VFL platform for research which will for sure facilitate the research considering VFL. Moreover, our work encourages not only the development of stronger defense methods, but also the development of new attacks for practical VFL scenarios. These attacks can be used for either benign or malicious purpose. Developing stronger defense or exploring related regulations and laws possible approaches for alleviating the potential negative impacts.
38

---

Published as a conference paper at ICLR 2024
U-NS(2-clss)
1-05(2-cla55
U-DLIS--las5
L--UlS-class)
0,20025
0,0000
0,50000
0002
-0 0036
0.03
-0 -.000775
-002
-0. -0.00000
0010
0.
(.05 goyct I-PMCIS-
3. geoee Ru UNA
ge o
gol TB-(AE;5-Chss)
FA-TBM(S-class)
6.10
0.10
0.0125
0.05
0.08
e posos
-000
2207s
pooso
0.05
-0.0
-0.10
0.10
06 ae NTB-N5BI5-cless
6 wo Aom NTB-M NTB-MFIS-C (iScclsss
sore
attack(S:class)
0.10
0.65
DCAE
0.00
0.04
0.60
0.02
-0.00
-045
0.00
-0.0
-0.00
-042
00.4
645
64P
LDR
Figure 21: DCS gap for each attack defeene point INUSWIDE dataset, agg FedBCD/FedSGD)
21
39301
0.00
BEBC
Figure 22: C-DCS ranking comparison across 3 datasets, 2 communication protocols and 2 dataset partition strategies presented in Tabs. to T9
39


Reviews:
[{"review_id": "kzw0g8RJd1", "reviewer": "Reviewer_z5tx", "summary": "The paper presents a research library and benchmark named VFLAIR for vertical federated learning. VFLAIR contains 9 datasets, 29 models, 2 communication protocols, 3 data partitioning, 11 attacks, and 8 defense methods. Model performance, attack and defense performance, and communication protocol comparison are comprehensively evaluated.", "questions": "1. Can you demonstrate how to use VFLIR and how to extend it? \n\n2. Can you summarize and highlight interesting findings at the beginning of each subsection of Section 6.1?\n\n3. Will you consider including more communication protocols, real-world vertical federated datasets, and data partitioning methods in the library?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. While vertical federated learning is a promising research direction with many real-world applications, it is less exploited compared with horizontal federated learning. Unlike horizontal federated learning systems, there is a lack of a comprehensive vertical FL library. This work is a significant contribution to the FL community.\n\n2. The library is comprehensive and includes many models, attacks, and defense methods.\n\n3. Experiments are extensive, especially for the attack and defense part.", "weaknesses": "1. The writing needs to be further improved. The paper claims that VFLAIR is a lightweight and extensible framework but does not demonstrate why it is. The introduction of the framework is limited. Besides introducing the components of VFLAIR in Figure 1, the paper should also introduce what is the systematic design of VFLAIR and demonstrate why it is very easy to use and extend.\n\n2. The insights in Section 6 should be highlighted. Currently, the paragraph is too long (especially Section 6.2) and readers are hard to find interesting results from the experiments. I suggest the authors put the insights at the beginning of each subsection.\n\n3. It seems that the paper divides the datasets into multiple subsets equally. Non-IID data partitioning is an important factor in FL and would be good to be included in VFLAIR. Also, it’d be better to include real-world vertical federated datasets besides partitioning a centralized dataset.\n\n4. The communication protocols in the library are not rich. Only two methods are considered.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "p1x3Mj4799", "reviewer": "Reviewer_1nCt", "summary": "This paper proposed a benchmarking framework for vertical federated learning. It proposes new evaluation metrics such as defence capability score (DCS)", "questions": "See the weakness part", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "This paper aims to propose a framework that can provide universal benchmarking solution for vertical federated learning. \n\nThe literature review is commendable, especially on the attack and defence part.", "weaknesses": "- The paper predominantly centers on evaluating attacks and defence strategies. But the paper title implies a broader scope – VFL in its entirety. The paper title could be more specific to align with the focus of the paper. \n\n- A notable contribution is the introduction of new evaluation metrics such as defence capability score (DCS). However, the experiments did not validate the effectiveness of the proposed metrics. What is the evidence that shows that the proposed metrics indeed work, representing the real ability of the evaluated algorithms?\n\n- The paper claims to “implement basic VFL training and evaluation flow under multiple model partition, communication protocols and attacks and defences algorithms using datasets of different modality”. But it lacks a clear exposition of the workflow. What is the training and evaluation flow in VFLAIR and how does the workflow facilitate the benchmarking? \n\n- Following on the above point, the evaluation section is not organised systematically according to model partition, communication protocols, and attack and defence algorithms. The current evaluation section only amounts to a compilation of experimental results, which \nrequires a more structured, systematic and coherent organization. \n\nThere is only one paragraph in related work. There is no need to employ a bullet point at the beginning", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RA1P3zOR7B", "reviewer": "Reviewer_Gb2U", "summary": "This paper aims to develop a lightweight vertical federated learning (VFL) platform (VFLAIR) framework consisting of multiple model partitions, communication protocols, and attack and defense algorithms using datasets of different modalities. Under this platform, the unified evaluation metrics and benchmark defense performance with various attacks are introduced,  which sheds light on choosing defense techniques in practical deployment.\nAlthough this paper summarizes most of the VFL settings, some advanced model partition and communication protocol algorithms are missed. In this way, I suggest authors should add these missed algorithms. In addition, I think some evaluations of this platform should be developed to show how it is lightweight. Some recommendation tasks should be considered, such as Criteo, Avazu, etc.", "questions": "1. Please add some advanced model partition and communication protocol algorithms, such as quantization, federated graph neural networks, etc.\n2. Add some compassion and discussion on how this platform is lightweight.\n3. The authors apply too many image classification datasets in the experiments. Some recommendation tasks should be considered, such as Criteo, Avazu, etc.\n4. As discussed in limitations, the cryptographic techniques that are significant, are not included in this library. \n5. The communication and computation efficiencies are also very important. The metrics should include the evaluation of the communication and computation efficiencies.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "This paper can provide a VFL platform for researchers to evaluate the performance and efficiency of their proposed algorithms. This platform from five aspects, i.e., model partitions, communication protocols, attacks, defenses, and dataset modalities, which are very significant for VFL studies. The strengths of this paper are as follows: \n1. Most of the VFL settings, e.g., model partitions, communication protocols, attacks, defenses, and dataset modalities, are included in this platform.\n2. This platform provides unified evaluation metrics and benchmark defense performance with various attacks,  which can guide the selection of defense techniques in practical deployment.\n3. The analysis of experimental results is sufficient and insightful.", "weaknesses": "Some weaknesses are shown as follows:\n1. Although this paper summarizes most of the VFL settings, some advanced model partition and communication protocol algorithms are missed.  In this way, I suggest authors should add these missed algorithms, such as quantization, federated graph neural networks, etc.\n2. I think some evaluations of this platform should be developed to show how it is lightweight.\n3. VFL is usually adopted in recommendation tasks instead of image classification. The authors apply too many image classification datasets in the experiments.\n4. As discussed in limitations, the cryptographic techniques that are significant, are not included in this library. \n5. The communication and computation efficiencies are also very important. The metrics should include the evaluation of the communication and computation efficiencies.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)