VertiBench: Advancing Feature Distribution Diversity in Vertical Federated Learning Benchmarks
Authors: Zhaomin Wu
Junyi Hou
Bingsheng He
Venue: ICLR 2024 poster (2024)
Keywords: Vertical federated learning
benchmark
feature correlation
feature importance
Primary area: datasets and benchmarks
TL;DR: This paper presents VertiBench, a comprehensive benchmark for vertical federated learning, covering an extensive range of application scenarios.

Abstract: Vertical Federated Learning (VFL) is a crucial paradigm for training machine learning models on feature-partitioned, distributed data. However, due to privacy restrictions, few public real-world VFL datasets exist for algorithm evaluation, and these represent a limited array of feature distributions. Existing benchmarks often resort to synthetic datasets, derived from arbitrary feature splits from a global set, which only capture a subset of feature distributions, leading to inadequate algorithm performance assessment. This paper addresses these shortcomings by introducing two key factors affecting VFL performance - feature importance and feature correlation - and proposing associated evaluation metrics and dataset splitting methods. Additionally, we introduce a real VFL dataset to address the deficit in image-image VFL scenarios. Our comprehensive evaluation of cutting-edge VFL algorithms provides valuable insights for future research in the field.

VERTIBENCH: ADVANCING FEATURE DISTRIBUTION DIVERSITY IN VERTICAL FEDERATED LEARNING BENCHMARKS
Zhaomin Wu, Junyi Hou, Bingsheng He National University of Singapore
(zhaomin, junyi. h, hebs ) @comp nus edu.sg
ABSTRACT
Vertical Federated Learning (VFL) is a erucial paradigm for training machine learning models on Srature-partitionee distributed data. However, due to privacy restrictions, few public real-world VFL datasets exist for algorithm evaluation, and these represent limited array of feature distributions. Existing benchmarks often resort synthetic datasets, derived from arbitrary feature splits from global set, which only capture a subset of feature distributions, leading to inadequate algorithm performance assessment. This paper addresses these shortcomings by introducing two key factors affecting VFL performance feature importance and feature correlation and proposing associated evaluation metrics and dataset splitting methods. Additionally, we introduce a real VFL dataset to address the deficit in image-image VFL. scenarios. Our comprehensive evaluation of cutting- edge VFL algorithms provides valuable insights for future research in the field.
INTRODUCTION
Federated learning Konečný et al. 2016 is acknowledged for enabling model training on distributed data with enhanced privacy. In this study, we delve into the less explored vertical federated learn- ing (VFL), where each party has feature subset, aligning with general definition of federated learning (Li et al. 2021al that includes privacy-preserving collaborative learning like assisted learn- ing (Diao al. 2022) and split learning (Vepakomma et al. (2018). The VFI. application, depicted in Figure la involves an initial development phase using synthetic or real-world benchmarks, followed by deployment in actual federated environments upon validation.
Evaluating VFL algorithms is challenging due to the inherent confidentiality of VFL data (Liu et al. 2022p. The scope of party imbalance and correlation in existing real VFL datasets, termed the real scope, is limited. Datasets in the OARF benchmark (Hu et al.) (022), FedAds (Wei et al.) 2023), NUS-WIDE (Chua et al. 2009), and Vehicle (Duarte and Hu 2004), predominantly represent scenarios where parties are balanced and exhibit weak correlations, as depicted in Figure Tb
To address the constraints inherent in the real scope, many VFL. benchmarks (Hu et al.) 2022 et et al. | 2020; Caldas al. 2018) utilize synthetic datasets. This evaluation scope, termed uniform scope, represent the imbalance-corellatin scope under an equal distribution of features among parties, either randomly or manually. The uniform scope, though commonly adopted in VFL experiments (Diao et al. 2022 Castiglia et al., 2022). confines the evaluation to scenarios featuring balanced, strongly correlated parties according Figure lb Another critical limitation the misaligmment between the uniform scope and real scope, underscoring the imperative for a diverse and realistic VFI. benchmark.
Constructing a systematic synthetic VFL benchmark necessitates pinpointing the key factors affecting VFL algorithm performance. Existing synthetic benchmarks for non-i.i.d. horizontal federated learning (HFL), such as NIID-Bench (Li al. 2022a),, fall short for VFL due to inherent assumptions about feature space and instance significance. Specifically, while HFL benchmarks typically assume independent and uniformly significant instances, this does not hold in VFL where features exhibit intrinsic correlations and differing importances. Furthermore, HFL benchmarks posit that all parties share the same feature space, a premise misaligned with VFL's distributed feature paradigm. This delineates the unique analytical challenges inherent to synthetic VFL benchmarks.

---

10
Engineer
Data Owners
Inber pouty Comelation 18
VertiBench
Party
Party
Synthetic
Real
Evaluate
Vertical Federated
Deploy
Vertical Federated
Learning Algorithm
Learning Algorithm
13
10
(a) Pipeline of VFL
(b) Estimated scope of VFL datasets
Figure 1: Overview of existing VFL. piplines and datasets and the estimated scope of VFI. datasets
Given these limitations, our statistical analysis of supervised VFL tasks identifies party importance and correlation as two crucial factors influencing target probability distributions synthetic VFL datasets derived from the same global dataset. Accordingly, we propose VertiBench, comprehensive VFL benchmark featuring novel feature-spliting methods for synthetic dataset generation. VertiBench offers three primary benefits: (1) it generally encompasses the uniform scope; (2) it effectively emulates the real scope, as evidenced by comparable performance on Verüßench--yythhete datasets; and (3) it introduces the capability evaluate other scenarios that have not been explored in the previous studies, e.g. imbalanced feature split, broadening the scope VFL evaluation.
Our primary contributions include: (1) Synthetic dataset generation methods with varied party importance and correlation, capturing broad scope of VFL scenarios. (2) Novel real-world image- to-image VFL dataset satellite, (3) Techniques to evaluate the party importance and correlation of real-world VFL datasets, enabling feature split comparison with synthetic VFL datasets. (4) Comprehensive benchmarks of mainstream cutting-edge VFL algorithms, providing key insights. For example, we demonstrate the scalability of VFL algorithms, challenging prior assumptions about VFL scaling difficulties (Hu et al. 2022). and emphasize the challenges of communication efficiency in VFL datasets across varying imbalance levels. The VertiBench source code is available on GitHub (Wu et al. (023a)), with data splitting tools installable from PyPI (Wu et al. 2023b). The pre-split dataset is accessible in (Anonymized 2023)
2 EVALUATE VFL DATASETS
In this section, our objective is to investigate the primary factors influencing VFL performance when generating synthetic VFL datasets from a fixed global dataset. Additionally, we explore methods to efficiently estimate these factors, guiding the subsequent feature split.
2. FACTORS THAT AFFECT VFL. PERFORMANCE
Suppose there are K parties. Denote the data party Ps as random vector X₁ (1 < k < K). Denote the label as random variable y. supervised learning algorithm maximizes the likelihood function where hypothesis h represents models and parameters, i.e., T(yl X .... X1; h). These supervised learning algorithms estimate the probability mass function Eq. The proof of Proposition is provided in Appendix A
[
Proposition 1. The probability mass function can be written as
log Pyy|X....... X1) Ex=
log
P(yl X x1) log P(y)
In VFL, P(y) is the same for all the parties. The skewness among K parties is determined by K ratios of distributions. Interestingly, this ratio quantifies the divergence between two marginal probability distributions of one inclusive of X₁: and the other exclusive of X₁. Essentially, the ratio estimates the impact on the global distribution when the features of a single party are excluded. This can be interpreted as the importance of a given party. Proposition] Japplies regardless of the order of X,,..., X,. Shapley value, emphasizing feature independence, aids in precisely evaluating party importance in vertical federated learning, demonstrated in (Wang et al. 2019] Han et al.! 2021)
2

---

p 1 X1) is determined by the correlation between X and ,,,...,X4 In cases where the independence assumption underlying the Shapley value is invali- dated, assessing each party's ss impact on the global distribution becomes more accurate when based on feature correlation.
In another aspect, the ratio P X1)
We identify feature importance and correlation as pivotal factors influencing VFL algorithm per- formance. For datasets with nearly independent features, the low inter-party correlation makes correlation-based splits less meaningful, suggesting the superiority of importance-basee splits, Conversely, in datasets with highly correlated features, assessing individual feature impor- tance becomes impractical, making correlation-batt splits more suitable due to varying inter-party correlations.
feature
Importance and correlation are treated orthogonal evaluation factors applicable in distinct scenarios. While there may be an intrinsic link between them, our experiments indicate that focusing on one factor at time yields explainable results reflective of real-world performance. As discussed in Appendix H the interplay between importance and correlation can be complex. A joint optimization for both factors might be computationally intensive and less explainable, while providing limited additional insights. The subsequent sections will introduce our approach to evaluate these two factors and generating synthetic datasets based on each factor accordingly.
2.2 EVALUATE PARTY IMPORTANCE
To assess the importance for each party, we sum the importance of its features. While numerous methods to evaluate feature importance can be adopted in VertiBench, this study primarily focuses on two approaches: 1) Shapley Value: Feature importance is determined using Shapley values, efficiently estimated by evaluating the performance of a trained XGBoost (Chen and Guestrin (016) on random subsets. 2) Chapley-CMi (Han et al., (22TT) This approach, which does not rely specific models, estimates the importance of each feature based on the Shapley-CMI applied to the global dataset. Both methods yield consistent and reasonable estimates of party importance.
2.3 EVALUATE PARTY CORRELATION
The task of efficiently evaluating correlation among two groups of features is challenging despite well-studied individual feature correlation (Myers and Sirois) 2004: De Winter et al. 2016). The Shapley-Taylor index, proposed for evaluating correlation between feature sets (Sundararajan et al.) 2020), is computationally intensive (NP-hard), and unsuitable for nigh-dimensiioal datasets. The determinant of the correlation matrix (Wang and Zheng] 2014) efficiently estimates inter-party correlation but is over-sensitive to linearly correlated features, impeding its use in feature partitioning. A more refined metric the multi-way way correlation coefficient (mcor) (Taylor 2020), addresses this, but like the determinant, it struggles with unequal feature numbers across parties, a typical VFL scenario, due the assumption of a square correlation matrix.
Given the limitations of existing metrics TTaylor 2020 Wang and Zheng 2014), we propose a novel metric to examine the correlation when the parties involved possess unequal numbers of features. Our approach hinges on the use of the standard variance of the singular values of the correlation matrix. This serves as an efficient measure of the overall correlation between two parties. Since the feature- wise correlation is an orthogonal research area, we selected Spearman rank correlation (Zar) 2005) due its capability to handle non-linear correlation.
To elaborate further, we denote the column-wiss correlation matrix between two matrices, X; and X, as cor( X,, X,). As a result, we formally define the correlation between two entities, X; E Raxm and X, E Ruim;; in terms of their respective parties as Eq. 2
Pcor(X: x, x,) Vā / d-1 Σd_ (ar X,)) -) d min(m;; my)
In this equation, (.).) means the i-th singular value of matrix, while σ stands for their mean value. SatoossitionSssattttt.. that Pcor equivalent to mcor for inner-party marty correlation (see Appendix A for proof). Experiments detailed in Appendix [DII reveal that Pcor exhibits trends analogous to mcor (Taylor 2020) when assessing inter-party correlation between equal number of features.
3

---

Proposition 2. For any real matrix X. Peor(X, X)= mcor X, X)
The singular values of a correlation matrix, Pcor, represent the magnitudes of its ellipsoid's semi-axes, indicating the degree of dependence among features. The standard deviation of these singular values reflects the distribution of dependence across different axes. A notably large singular value in specific axis (Figure 2c) suggests high concentration dependence. For instance, there's 's only one nonzero singular value, it implies that all features are perfectly correlated with a single feature. Conversely, the singular values uniformly distributed such as Figure 2a (indicated by a small standard deviation), it denotes less concentrated feature correlations. Therefore, the standard deviation of singular values serves as a measure of the dataset's proximity to perfect correlation.
Proposition 3 states that Pcor, like mcor, spans a range from 0 to 1, even when assessing anterppatt correlation. Pcor value of signifies perfect correlation between X, and X2, while a value of indicates their independence.
Proposition 3. For any two real matrices X1 and X2. Pcor(X, X2) E [0,1]
is important to that the absolute value of Pcor alone does fully capture inter-party correlation. For instance, when X; and Xj are two parties both containing the same set of independent features, Pcor(X,, XXy) yields value of O, the same as the Pcor between two independent parties. Despite the same Pcor value, these scenarios intuitively differ in their levels of inter-party correlation. This discrepancy arises from overlooking the inner-party correlation of X; and Xj. Typically, parties with highly correlated features tend to exhibit higher Peor values with other parties.
To accurately measure the correlation between X; and Xj, we evaluate how the shift towards perfect correlation varies when X, is replaced by X,. This is captured by the relative change in Pcor, denoted as Pcor( X; X)) - Pcor(X,, X,, X:). In the perspective
variance analysis (Kruskal and Wallis 1952), J this difference quantifies the degree to which the standard deviation Pcor[X,, X3) is explained by inter-party factors, controlling the contribution of inner inner-party correlations. The overall inter-party correlation, denoted Icor, is described as the mean party-wise correlation across all distinct party pairs. Formally,
Icor II..... X = K(K-I) ER EjnjA (Pe X,) Peor X.)).
(3)
Independent
Partially Conrelated
Perfect Correlated
100 10
100 -18t -uoa
1
Cove abe Mutic
2.00 -1:00
comblton Mari
Comdation Marts
2.0 3:00 1.0)
(.00 1.00
5.00 1/00
Powmo Siga Vakes [L 000
Poav155 55. Eaaler Valuo [.00 2000 0.3
Peavi D0. Sngaer Vate (bux nox 0.d
(a) v. 1)
(b) y (0, 1), --= (c) U 1), y= 2r, x+1
Figure 2: Examples of Pcor values on different levels of correlation. U means uniform distribution. Arrow direction indicates right singular vector orientation, arrow scale represents singular values.
Icor exhibits notable properties both theoretically and empirically. Theoretically, as demonstrated in Theorem (see Appendix A for proof), optimizing Icor yields ideal feature splits in optimal scenarios. Specifically, in datasets comprising two independent but internally perfectly correlated feature sets, Icor reaches its minimum when each party exclusively possesses one feature set and attains its maximum when each party equally shares half of the features from both sets. Empirically, evaluate the link between inter inter-party party correlation and Icor in complex, real-world datasets (Appendix[)) These empirical observations align with theoretical insights, confirming Icor's capability in analyzing intricate data correlations.
4

---

Theorem 1. Consider global dataset X comprising two independent datasets D1, D2 E R each of the same dimension. Independence implies that for any feature dj from D and any Furthermore, assume within D1 and D2, all features are perfectly correlated, such that for all pairs of distinct features di aj
feature oj
(2) from D2, where i,j m,, the correlation Cor (a. (1) .a.²))
(1) (1) in D₁ and (2) ,aj D2, with i, [1, ml and + j. the correlations satisfy respectively. When the features of X are divided equally into two subsets, X, and X2, such that each subset contains m/2 features, the overall inter-party correlation Icor(X1, X2) satisfies
(2)
Cor (a. :
(1) (1) =1 and Cor (a ,aj )
Icor(X, X2) E -
m
7m(m-1)
The lower bound occurs if and only if X1 comprises all features of either D1 D2, with X2 containing the remaining features. The upper bound occurs if and only if X₁ holds m features from both D2 and 22, with X2 holding the remaining m features from D and D
SPLIT SYNTHETIC VFL DATASETS
This section aims develop algorithms split features according two key factors: importance and correlation. These algorithms should allow users to adjust the party importance and correlation of synthetic VFL datasets by simply modulating two parameters: a and (3. The intended mapping should meet two criteria: (1) The scope of and 3 should encompass broad spectrum of feature splits, inclusive both real splits and random splits. (2) When two global datasets bear similarities, synthetic VFL datasets derived from them using identical a and / parameters should yield similar VFL algorithm behaviors. We provide both theoretical and empirical validation for criteria (1) in this section, whereas criteria (2) is substantiated through experiments in Section 4.4
3.1 SPLIT BY PARTY IMPORTANCE
In light of the computational expense incurred by the Shapley value method, an alternative and more efficient strategy is necessary to perform feature splits based on importance. With all parties exhibiting symmetry in the context of X, varying the importance among parties essentially translates to varying the variance of the importance among them. Assuming each party P: possesses an importance factor O₂ > 0, we propose the implementation of the Dirichlet distribution parameterized by {o;) K for feature splitting. This approach ensures two beneficial properties post-split: (I) a larger a; guarantees higher expected importance for P.. and (2) smaller {a,) assures greater variance in the importance among parties.
More specifically, we propose feature splitting method based feature importance. After initializ- ing local datasets for each party, series of probabilities r1...., PK s.t. Σ r; is sampled from a Dirichlet distribution Dir(a ,ar). Each feature is randomly allocated to a party Ps. selected based on the probabilities rk. To accommodate algorithms that fail when faced with empty features, we can ensure each party is initially provided with random feature before the algorithm is set in motion. Detailed formalization of this algorithm can be found in Appendix
Theorem 2. Consider feature index set A {1, m.. and characteristic function 2 + IR such that v00 = O. Let Φ5 (v) denote the importance of the j-th feature on such that Σ Φbj) v(A). Assume that the indices in A randomly distributed to K parties with probabilities T1, ... TK ~ Dir(ai,.... "aaaa,,,.....,,. Let Zi be the sum of feature importance for party i. Then, we have Vi E [1, K] and E[2]] 04.
The proof of Theorem? can be found in Appendix A resembling the Dirichlet-mulinomiall mean proof but focusing on sum importance instead of feature counts. The metric of importance, dy(v), comprises the Shapley value and the recently proposed Shapley-CMI (Han et al. 2021). Theorem the importance
asserts that the expected cumulative importance E[Z,] of each party is proportional parameter a, The Dirichlet- based split method ensures that: (1) a larger value of a; leads to a higher expected value of ri, thus higher expected value of party importance, and (2) smaller value of
5

---

[e llz results in larger variance in r,, well more imbalanced importance among parties. Both properties are empirically validated in Appendix [D2 Hence, the proposed method naturally aligns with the requirements for feature importance. With 1. Dirichlet-split mirrors uniform distribution, incorporating random splits within the uniform scope. Even for manual equal splits lacking consistent criteria, large in Dirichlet-split encapsulate them by yielding nearly equal feature distribution among parties.
3.2
SPLIT BY PARTY CORRELATION
This correlation- based feature-split algorithm (Alg. 1 is designed to allocate features across multiple parties based on a given correlation parameter B. The algorithm's 's operation is premised on defined number of features for each party, represented as m.,.... ,,, m.. Commencing with the initialization of a column permutation matrix P to an identity matrix (line 1), the algorithm proceeds to define a score function, /(P: X). which represents the overall correlation Icor after the features permutated by P (line 2). Subsequently, the algorithm determines the range of the score function (lines 3-4). This forms the basis for calculating the target correlation (X: 3), which is a linear interpolation between the lower and upper bounds controlled by the correlation index 3 (line 5). Next, the algorithm locates the optimal permutation matrix P by solving an permutation- based optimization problem. Notably, we employ the Biased Random- Key Genetic Algorithm (BRKGA) (Gonçalves ves and Resende (2011) for this purpose. The final step the algorithm splits the features according to the derived optimal permutation and the pre-set number of features for each party (lines 6-7).
Algorithm 1: Feature Splitting by Correlation
Input:
Global dataset X E RTXm correlation index 3, number of features mi,.... mK
Output: Local dataasets X X .,
P t- I;
/* Initiate permutation matrix +/
[(P; x) = Icor(X, X ..... x s.L. XP, ..... Xx f split features of XP by mı,..., MKi
3 min! (X) minp [(P
Calculate lower bound */ /* Calculate upper bound +/
maa X maxp fPP
5 (XX;; B) + (1 -m) (X) + 38mar (X);
/+ Calculate target correlation
/* Find the permutation matrix +/ 7 X X + split features of XP* by mk....,k: return XI....,X, ...,,.KK
6 P <- arg minp [f(P:X) f'X:; B)1:
The efficiency of the optimization process, involving numerous Icor invocations, is crucial. For smaller datasets. Singular Value Decomposition (SVD) (Baker 2005) is used for direct singular value computation. However, for high-dimensional datasets, we employ truncated SVD (Hansen) 1990) estimates the largest top-d; singular values, assuming the remainder as zero for standard variance calculation. The ablation study of de included in Appendix G.6 Our experiments, detailed in Appendix D.2. confirm the efficacy of both split methods.
3.3
COMPARE FEATURE SPLIT ACROSS GLOBAL DATASETS
The metrics presented in Section facilitate meaningful comparisons of feature splits within the same global datasets but fall short when comparing across different datasets. To bridge this gap and enable comparison between real and synthetic VFL datasets, we introduce methods to map these metrics to two values: and B, where indicates party balance and B indicates party correlation. Consequently, this mapping enables a direct comparison between feature splits originating from real and synthetic VFL datasets, demonstrated in Figure lb
To estimate a, the importance of each party is calculated by Shapley values. These importance are then normalized and treated as Dirichlet parameters a; for each party P.. in line with Theo- rem 2 To approximate the scale of the Dirichlet parameters and align them with the generation of synthetic datasets, we find symmetric Dirichlet distribution Dir(a) that has the same variance as (ir ag) ). as given in Proposition This value reflects the variance of party importance. The proof is provided in Appendix

---

Proposition 4. Given a Dirichlet distribution Dir(ai.....,au ag) with mean variance symmetric Dirichlet distribution Dir(a) that has the same mean variance if a K-1-K"a
R
To estimate 3, we start by computing the potential minimum and maximum values of Icor by shuffling the features among parties, denoted Icormin, Icor Next, estimate the Icor of the actual dataset, Icorraal and derive the 3 value using / = min max Icot-ar- 0} )
It is important to note that in real- world scenarios, Icoreal might fall slightly outside the range of Icor Icormax due to the constraints of optimization algorithms. To rectify this, we clip the estimated 33 to ensure 3 E [0, 1|.
lcof 1o0f
4 EXPERIMENT
This section benchmarks cutting-edge VFL algorithms, with detailed review in Section 4.1 Experi- mental settings are outlined in Section 4.2 and results regarding VFL accuracy and synthetic-real correlation are in Sections 4.3 4.4 respectively. Further evaluations, such as real communication cost, scalability, training time, and real dataset performance, in Appendix G Each experiment elucidates results and provides relevant insights, highlighting (1) the cenffommancc-oommniccte tradeoff of NN-based and boosting-based based methods, (2) the performance similarity between synthetic and real VFL datasets under the same a, 3, and (3) the scalability potential of VFL algorithms.
4.1 REVIEW OF VFL. ALGORITHMS
This section reviews existing VFL algorithms, with a focus accuracy, efficiency, and communi- cation cost. VertiBench concentrates on common supervised learning tasks such as classification and regression within synchronized parties, summarized in Table 1 Notably, this benchmark ex-
cludes studies exploring other aspects (Jin et al., 2021: Qi et al. [022;] Jiang et al. 2022) and other tasks (Chang et al. 2020 et al.) 202 Chen and Zhang 2022, He al. 2022 Li et al. (022b).
Since most VFL algorithms presume exact inter-party data linking, we adopt this approach in Vert- iBench, despite recent contrary findings (Wu al. 2022a Nock et al. 202I) that this assumption may not be true. We refer to parties with and without labels as primary and secondary parties respectively.
Table 1: Summary of existing VFL algorithms
Category
Model'
Algorithm
Contribution
Reference
Data²
Feature"
Ensemble- based
AL
Accuracy
Xian 2020
Syn
Manual
Any
GAL
Accuracy
Diao el al.
2022
Syn
Manual
SplitNN
Accuracy
Vepakomma et
2018
Syn
N/A
NN
C-VFL
Communication
Castiglia et al.
2022
Syn
Manual
BlindFL
Efficiency
[Fu cl al. 20226
Syn
Manual
Split- based
FedOnce
Communication
Wu al.
2022b
Syn
Random
SecureBoost
Accuracy
Cheng et al
2021
Syn
Manual
Pivot
Accuracy
Wu et
2020
Syn
Manual
GBDT
FedTree
Accuracy, Efficiency
Ll al.
2023)
Syn
Random
VF2Boost
Efficiency
Fu ct al.
2021
Syn
Manual
Fed-Forest
Communication
(Liu al.
2020
Syn
Random
Abbreviations: NN neural network; GBDT gradient boosting decision trees; random forest; Any
crodel-agnostic
Dataset in experiments: Syn synthetic datasets partitioned from global datasets.
Datasets used in the experiments: Manual features manually split without specific reasons; Random
features randomly split without explanation; N/A VFL experiments conducted.
Most of the existing VFL methods can be categorized into ensemble-based and split-based. Ensemble- based methods have each party maintain a full model for local prediction and use collaborative ensemble techniques during training. Conversely, split-based methods delegate each party with portion of the model, representing different inference stages. comprehensive comparison is in Appendix B In this paper, we concentrate on the primary types of VFL, acknowledging that there are various subtypes as identified in (Liu et al. 2022). Exploring these subtypes in depth will be an objective of our future research efforts.
7

---

In our experiments, we evaluate various VFL algorithms, including split-NN- (e.g.. SplitNN, C- VFL, FedOnce), split- -aaii--BDDbaaeed (FedTree), and ensemble- based (GAL). For faimesss, evaluations exclude encryption or noise. Noting minor variances among split- babb-GeBDT methods such as FedTrre and SecureBoost, FedTrre is used as representative in our experiments.
4.2 EXPERIMENTAL SETTINGS
This subsection includes the datasets and training method. Detailed dataset specifications, environ- ments, and hyperparameter settings can be found in Appendix F
Datasets. Our experiments utilize 11 datasets: nine centralized ones (covtype ( BBlackard
1998), msd -Mahiex, (011). gisette (Guyon et al. 2008), realsim (Andrew
2015), epsilon (Guo-Xun et al.) 2008), letter (Slate) 11991), radar (Khosravi 2020),
MNTST (Deng 2012), CTFAR10 (Krizhevsky and Hinton 2009)). and two real-world VFI. datasets (NUS-WIDE CChua al. 2009). Vehi (Duarte and Hu 2004)), ) with detailed descriptions
available in Appendix F The msd dataset is used for regression tasks, while the others cater to
classification tasks. Each dataset is partitioned into 80% training and 20% testing instances except NUS -US-WIDE, MNIST, and CIFAR10 with pre-defined test set. The datasets' features are distributed among multiple parties (typically four), split based on party importance (a) or correlation (B). In the correlation- basedd split, each party is assigned an equal number of features.
Training. For classification tasks, we use accuracy as the evaluation metric, while regression tasks are evaluated using the Root Mean Square Error (RMSE). To ensure the reliability of our results, we conduct five runs for each algorithm, using seeds ranging from 0 to 4 to randomly split the datasets for each run, and then compute their mean metries and standard deviation. Detailed hyper-parameter settings for each algorithms are provided in Appendix
4.3 VFL ACCURACY
In this subsection, we assess the impact on the performance of VFL algorithms when varying a and 3. Our analysis includes all the three VFL categories in Table The performance summarized in Figure Jan detailed in Tablegin in Appendix G The result on mad dataset provides similar insights to others, thus only included in Table From our exploration, we can draw three key observations.
Split parameters and 3 significantly affect VFL algorithm performance, depending on the algorithm and dataset. SplitNN and FedTree show stable performance across various a and B settings. In contrast, C-VFL demonstrates notable performance fluctuations: up to 10% eps onn and 40% on letter with varying a. GAL performs better on imbalanced datasets (affected by a by 8% on letter and radar, 2-5% on others) and is minimally influenced by B. FedOnce, favoring balanced and highly correlated datasets, is affected by a (5-10% on letter, g sette, epsil lon) and by (1-3% on covtype, epst i 1 on). These findings highlight the need for comprehensive evaluations across range of a and 3 to determine VFL algorithms' robustness.
SplitNN often leads in accuracy across most datasets; however, the performance of split- GBDT-based and ensemble-based methods can vary significantly depending on the dataset. As anticipated, given its iterative transmission of substantial representations and gradients, SplitNN often outperforms other methods across a majority of datasets. Comparatively, the performance of FedTree and GAL is dependet. FedTree well- well-suited high figg--imensional, smaller datasets like g sette, but struggles with larger datasets like eps i 1 on and covtype. GAL, on the other hand, performs admirably with binary classification and regression tasks, though its performance drops significantly the number of classes increases, as observed on the covtype and letter dataset.
The compression of SplitNN renders them particularly affected by party imbalance. C-VFL, modelled after SplitNN, exhibits the least accuracy among tested baselines due to its compression approach. Moreover, C- C-VFL exhibits marked sensitivity to the imbalance level, a. Specifically, at = 0.1, its accuracy datasets like letter and epsilon scarcely surpasses random guessing. However, C-LFL thrives in highly imbalanced split of radar dataset. This data data-dependent behavior underscores an urgent need to refine compression techniques for VFL. tailored varying imbalances.
8

---

letter
rader
100
80
100
100
100
Accuracy
6D
80
40
90
70
50
1.0
10
1.0
10.0
10
10.0
1.0
10.0
1.0
30.0
coxtype
gisette
reslsim
epsiion
letter
radar
100
80
90
95.0
80
Mccu
70
90.0
40
20-
87.5
0.
00
SpIINN
FedOnce
Figure 3: Accuracy of VFL algorithms on different datasets varying imbalance and correlation
4.4 PERFORMANCE CORRELATION: VERTIBENCH SCOPE vs. REAL SCOPE
In assessing the performance correlation between VertiBench and real VFL datasets, use derived a and B values NUS-WIDE and Vehicl e (Section 3.3) to generate comparable synthetic datasets. To evaluate the relative performance of each algorithm, calculate the accuracy differences between Vehi cl ee-sytthetic and NUS SES--IIEE--S WIDE-syythhtic datasets for each algorithm and compare with real dataset accuracy differences, with further details in Appendix G.8
Our experiment reveals positive correlation between relative algorithm performance on synthetic datasets with matching a and 8, and their performance on VFL datasets. This indicates that, under the same or 3, higher mean accuracy on synthetic datasets typically implies better performance on real VFL datasets, thus affirming the relevance of serriBennh--yynthee datasets in approximating real VFL performance.
Cort. Carl: 0254
Corr. Coef: 0.387
FadTTee
Ceceorrryy difieennce (owal)
fucOnce
sccurac difteennc
-6
-10
10
Woon socuracy difforence synt (9))
Moon accuracy difforence 1d syni (%6)
Figure 4: Mean accuracy differences: synthetic datasets vs. real datasets
CONCLUSION
We introduce VertiBench, a refined benchmarking tool for Vertical Federated Learning (VFL), adept at generating variety of synthetic VFL datasets from single global dataset. The scope of VertiBench extends beyond the confines existing uniform and real scopes, shedding light on VFL scenarios previously unexplored. Our findings underscore performance variations under diverse data partitions, emphasizing the need to evaluate VFL algorithms across varied feature splits for enhanced insights into their real-world applicability.

---

REPRODUCIBILITY STATEMENT
The code for this study is accessible via a GitHub repository (Wu et al. 2023a accompanied by README md file that provides guidelines for environment setup and result reproduction. Compre- hensive proofs of all theoretical results are meticulously detailed in Appendix A Further, Appendix offers detailed description of dataset specifications and hyperparameter configurations.
ACKNOWLEDGEMENT
This research is supported by the National Research Foundation Singapore and DSO National Laboratories under the Al Singapore Programme (AISG Award No: AISG2-RP-2020-0185 Any opinions, findings and conclusions the authors and do not reflect the views of National Research Foundation. Singapore. This work is supported in part by AMD under the Heterogeneous Accelerated Compute Clusters (HACC) program.
recommendations expressed in this material are those of
REFERENCES
McCallum Andrew. Real vs. simulated, 2015, URL https::/www.csttttet
Anonymized, 2023.
URL https://drive http://www...ollleeoo/o//e////// 1Ti73Doy (tbDoyy><OORR//DFFZEFG lZ2Nf
Kirk Baker. Singular value decomposition tutorial. The Ohio State University, 24, 2005.
Bertin-Mahicux. Yearpredictionmssd, 2011. URL https::/www.cceeentuuuu.
relllm//ibssttooos//ttteeeee httpa///cee..a.eeeeee/e///aaaaaa
Jock Blackard. Covertype, 1998.
URL
tw/
DOI::
Sebastian Caldas, Sai Meher Karthik Duddu, Peter Wu, Tian Li, Jakub Konečný, H Brendan McMa- han, Virginia Smith, and Ameet Talwalkar. Leaf: A benchmark for federated settings. arXiv preprint arXiv:1812.01097 2018
Timothy J Castiglia, Anirban Das, Shiqiang Wang, and Stacy Patterson. Compressed VFL: cifemuicationnefffcieent
Communication efficient learning with vertically partitioned data. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Pro- ceedings of the 39th Internutional Conference Machine Learning, volume 162 of Pro- ceedings of Machine Learning Research, pages http://sstiiiaaaaaaaa.aaaaa2aaaa html Sktps://processt
PMLR, 17-23 Jul 2022. URL
mlr www.L..reessvvlee wttpp///wwww.///aaaaaaaaaaa////.
Qi Chang, Hui Qu, Yikai Zhang, Mert Sabuncu, Chao Chen. Tong Zhang, and Dimitris N. Metaxas. Synthetic learning: Learn from distributed asynchronized discriminator gan without sharing medical image data. In Proceedings of the IEFE/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2020.
Jiayi Chen and Aidong Zhang. Fedmsplit: Correlation-adaptive federated multi-task learning across multimodal split networks. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD '22, page 87 87-96, New York, NY. USA, 2022. Association for Computing Machinery. ISBN 9781450393850.
doi: 10. 00.1145/3534678.352955
URL
ibproxyl nus edu. stt...sg/....1....... httg//...115/553311 35.44445555548.8333388. 3539384
Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd aem sigkdd international conference on knowledge discovery and data mining, pages 785-794, 2016.
Kewei Cheng, Tao Fan, Yilun Jin, Yang Liu, Tianjian Chen, Dimitrios Papadopoulos, and Qiang Yang. Secureboost: A lossless federated learning framework. IEEE Intelligent Systems, 36(6): 87-98, 2021.
10

---

Tat-Seng Chua, Jinhui Tang, Richang Hong, Haojie Li, Zhiping Luo, and Yan-Tao Zheng. Nus-wide: A real-world web image database from national university singapore. In Proc. of ACM Conf. on Image and Video Retrieval (CIVR 09)). Santorini, Greece., 2009.
Creative Commons. Attribution 4.0 international, 2023a. URL htt.ttss://creetiiveee
arg//licesses/by/Ae
code
Creative Commons. Atributiion 4.0 international (ce by-nc 4.0), 2023b. URL
:ttt:::/ccratiivecommonss http:/ccccoossoog/////eees/////a
Julien Comebise, Ivan Oršolić, and Freddie Kalaitzis. Open high-resolution satellite imagery: The worldstrat dataset -with application to super-resolution. In Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Informa- tion Processing Systems, volume 35, pages 25979-25991.. Curran Associates, Inc., 2022. URL http::/proceednng.eerris....... wttptpps.ccpppppeee/e willss/apper/2000
and And_Beechmarks.
pdf
Joost CF De Winter, Samuel D Gosling, and Jeff Potter. Comparing the pearson and spearman correlation coefficients across distributions and sample sizes: A tutorial using simulations and empirical data. Psychological methods, 21 (3):273, 2016.
Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29(6):141-14) 2012.
Enmao Diao, Jie Ding, and Vahid Tarokh. Gal: Gradient assisted learning for de- centralized multi-organization collaborations. In S. Koyejo, S. Mohamed, A. Agar- wal, D. Belgrave, K. Cho, and A. Oh, editors, Advances Neural Informa- tion Processing Systems. volume 35, pages 11854-11868. Curran Associates, Inc..
2022. URL
fi (111e//6688811110222663311333333
Marco F Duarte and Yu Hen Hu. Vehicle classification in distributed sensor networks. Journal of Parallel and Distributed Computing, 64(7):826-838, 2004.
Fangcheng Fu, Yingxia Shao, Lele Yu, Jiawei Jiang, Huanran Xue, Yangyu Tao, and Bin Cui. V12boost: Very fast vertical federated gradient boosting for cross-enterprise learning. In Proceed- ings of the 2 International Conference on Management of Data, SIGMOD '21, page 563-576 New York, NY, USA, 2021. Association for Computing Machinery. ISBN 978 9784430383431. doi: 10.1 00011445/4440113445222
btt.ttts://doi-oo00 wtt.:/oooo-.r.....o.......... libproxyl nus edu sg/10 10
1145/3448015 3457241
Fangcheng Fu, Xupeng Miao, Jiawei Jiang, Huanran Xue, and Bin Cui. Towards communication-
efficient vertical federated learning training via cache-enabled local updates. Proc. VLDB Endow.
15(10):21112122) jun 2022a. ISSN 8097. doi: 10. 10.44778/5473055554711 //doi-org ibproxyl nus. www.nuuuuuuuuuuuuu.. sg 14778 33488/55533355 33443154
URL https:
Fangcheng Fu, Huanran Xue, Yong Cheng, Yangyu Tao, and Bin Cui. Blindfl: Vertical federated machine learning without peeking into your data. In Proceedings of the 2022 International
Conference on Management of Data, SIGMOD '22, page 1316-1330, New York, NY, USA, 2022b. Association for Computing Machinery. ISBN 9781450392495. doi: 10. 10.1145/3514221 1.22112777 URL http:https///ooor....iie wtt.oor.......pp....... nus edu wgwpb. wttp:........................... htt.apppww.ooooooo....oooo...... 35261777777
José Fernando Gonçalves and Mauricio GC Resende. Biased random- key genetic algorithms for combinatorial optimization. Journal of Heuristics, 17(5):487-525, 2011.
Bin Gu, Zhiyuan Dang, Xiang Li, and Heng Huang. Federated doubly stochastic kernel learning for vertically partitioned data. Proceedings of the 26th ACM SICKDD International Conference on Knowledge Discovery & Data Mining, KDD '20, page Association for Computing Machinery. ISBN 978 9781450379984, doi: 10.11445339444883440222
New York, NY. USA, 2020.
wtw.:ll.sss.uuu..... URL htt.h:ttp///ae / org. httpppowwww...a.eeeeeeeee.e...e.

---

Yuan Guo-Xun, Ho Chia-Hua, and Lin Chih-Jen. Epsilon, 2008. wwwp:///wwww.s.......c......... http:://ww..........ttttttaaaaaa httpa///eeeaaaaaaaaaaaaaaooaaaaa
httpouu:LLtttt//t/tw/m ntt
Isabelle Guyon, Steve Gunn, Asa Ben-Hur, and Gideon Dror.
Gisette, 2008.
URL
https://www.csie./ httpe//t////iiii//////ooo///////
gisette_scale.
Xiao Han, Leye Wang. and Junjie Wu. Data valuation for vertical federated learning: An information- theoretic approach. arXiv preprint (tx.iv:2/22.08:60 2021.
Per Christian Hansen. Truncated singular value decomposition solutions to discrete ill- ill-posed problems with ill-determined numerical rank. SIAM Journal on Scientific and Statistical Computing, 11(3): 503-518, 1990.
Stephen Hardy, Wilko Henecka, Hamish Ivey-Law, Richard Nock, Giorgio Patrini, Guillaume Smith, and Brian Thome. Private federated learning on vertically partitioned data via entity resolution and additively homomorphic encryption. arXiv preprint 1 1tf::::711.1.77.... 2017.
Chaoyang He, Songze Li, Jinhyun So, Xiao Zeng, Mi Zhang, Hongyi Wang, Xiaoyang Wang, Pranceth Vepakomma, Abhishek Singh, Hang Qiu, al. Fedml: A research library and benchmark for federated machine learning, arXiv preprint mxXiv:2007.33514 2020.
Yuanqin He, Yan Kang, Xinyuan Zhao, Jiahuan Luo, Lixin Fan, Yuxing Han, and Qiang Yang. A hybrid supe-sup learning framework for vertical federated learning. arXiv preprini arXiv:2208,088 08934, 2022.
Sixu Hu, Yuan Li, Xu Liu, Qinbin Li, Zhaomin Wu, and Bingsheng He. The oarf benchmark suite: Characterization and implications for federated learning systems. ACM Transactions on Intelligent Systems and Technology (TIST), 13(2) 2022.
Yaochen Hu, Di Niu, Jianming Yang, and Shengping Zhou. Fdml: A collaborative machine
learning framework for distributed features. In Proceedings of the 25th ACM SIGKDD Inter- national Conference on Knowledge Discovery & Data Mining, KDD '19, page 2232-2240, New York, NY. USA, 2019. Association for Computing Machinery. ISBN 678346006 doi: 10. R0011/////22000333306666 1145/3292500 3330765
1ibproxxbbxxxy nus. www.eed............ sg/10
Lingxiao Huang, Zhize Li, Jialin Sun, and Haoyu Zhao. Coresets for vertical federated learning: Regularized linear regression and k-means clustering. In S. Koyejo, S. Mo- hamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural In- formation Processing Systems, volume 35, pages 29566-29581. Curran Associates, Inc.. 2022. URL pttpp::/pproeeeeinggs.... . neurips. cc/paer files/pape::2222/
file/
Jiawei Jiang, Lukas Burkhalter, Fangcheng Fu, Bolin Ding, Bo Du, Anwar Hithnawi, Bo Li, and Ce Zhang. Vf-ps: How to select important participants vertical federated learning, efficiently and securely? In Advances in Neural Information Processing Systems, 2022.
Xiao Jin, Pin-Yo Yu Chen, Chia-Y Yi Hsu, Chia-Mu Yu, and Tianyi Chen. Cafe: Catastrophic data leakage in vertical federated learning. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wort- man Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 994- 1006. Curran Associates, Inc., 2021. Ut htt.:///ww.///e///oeeeeeeeeeeee. fi face://er22211111000000666661111
Iman Khosravi, UCI machine learning repository, 06 2020. /tL.httpp:://rrchheee....
Jakub Konečný, H Brendan McMahan, Felix X Yu, Peter Richtárik, Ananda Theertha Suresh, and
Dave Bacon. Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv:1610.00 05492, 2016.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical Report 0, University of Toronto. Toronto, Ontario, 2009. https:::: / wwwwww..o cs
Barronto.edu/keee
12

---

William H Kruskal and W Allen Wallis. Use of ranks in one-criterion variance analysis. Journal of the American statistical Association, 47(260):583 621, 1952.
Qinbin Li, Zeyi Wen, Zhaomin Wu, Sixu Hu, Naibo Wang. Yuan Li, Xu Liu, and Bingsheng He. A survey on federated learning systems: vision, hype and reality for data privacy and protection. IEEE Transactions on Knowledge and Data Engineering, 2021a.
Qinbin Li, Yiqun Diao, Quan Chen, and Bingsheng He. Federated learning on non-iid data silos: An experimental study. In 2022 IEEE 38th International Conference on Data Engineering (ICDE), pages 965-978. IEEE, 2022a.
Qinbin Li, Zhaomin Wu, Yanzheng Cai, Yuxuan Han, Ching Man Yung, Tianyuan Fu, and Bingsheng He. Fedtree: A federated learning system for trees. In Proceedings of Machine Learning and Systems, 2023.
Wenjie Li, Qiaolin Xia, Hao Cheng, Kouyin Xue, and Shu-Tao Xia. Vertical semi-federated learning for efficient online advertising. arXiv preprint arXiv:2209. 1522209955335, 15635, 2022b.
Zitao Li, Bolin Ding, Ce Zhang, Ninghui Li, and Jingren Zhou. Federated matrix factorization with privacy guarantee. Proc. VLDB Endow. 15(4):900-913, dec 2021b. ISSN 2150-8097. doi: (0.14778/35035883350333
URL /
nus edu. ag/
10 18714478/44 0018//5535888 3503598
Yang Liu, Yingting Liu, Zhijie Liu, Yuxuan Liang, Chuishi Meng, Junbo Zhang, and Yu Zheng. Federated forest. IEEE Transactions on Big Data, 8(3):833-844, 2020.
Yang Liu, Yan Kang, Tianyuan Zou, Yanhong Pu, Yuanqin He, Xiaozhou Ye, Ye Ouyang, Ya-Qin Zhang, and Qiang Yang. Vertical federated learning. arXiv preprint arXiv:2211.12814. 2022.
Leann Myers and Maria Sirois. Spearman correlation coefficients, differences between. Encyclope- dia of statistical sciences, 12, 2004.
Richard Nock, Stephen Hardy, Wilko Henecka, Hamish Ivey-Law, Jakub Nabaglo, Giorgio Patrini, Guillaume Smith, and Brian Thome. The impact of record linkage on learning from feature partitioned data. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 3216-8226 PMLR, 18-24 Jul 2021. URL ptttp:://proceedeee
mlr mlr.press/
1.39/nock2la.herr html
Tao Qi. Fangzhao Wu, Chuhan Wu, Lingjuan Lyu, Tong Xu, Hao Liao, Zhongliang Yang, Yongfeng Huang, and Xing Xie. Fairvfl: A fair vertical federated learning framework with contrastive adver- sarial learning. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 7852 7865. Curran Asso- ciates, Inc., 2022. URL wwR.hhtppss//prreeeiiis.srrssssp htt.://0000e0oooaaaaaaarraoeeeee
2022/files 2r
David Slate. Letter, 1991.
URL https://www.csie wtw.i....u.eeee.. wtw.:attuuuu...weeeee.e wtt..//see..tuuuedd....///1l11..
libsvmtools oo hotoosss/daastttte/e/ wwtissetsmmmutiie htt//mmliiicccc
/ wttpa///ettterrrccee http:///ll.///etete.ssseaee
Mukund Sundararajan, Kedar Dhamdhere, and Ashish Agarwal. The shapley taylor interaction index.
In International conference on machine learning, pages 9259-9268. PMLR, 2020.
Benjamin M Taylor. A multi-way correlation coefficient. arXiv preprint arXiv:2003.02561, 2020.
Praneeth Vepakomma, Otkrist Gupta, Tristan Swedish, and Ramesh Raskar. Split learning for health: Distributed deep learning without sharing raw patient data. arXiv preprint arXiv:1812.0055:
2018
Guan Wang. Charlie Xiaoqian Dang, and Ziye Zhou. Measure contribution of participants in federated learning. In 2019 IEEE international conference on big data (Big Data), pages 2597-2604. IEEE, 2019.
Jianji Wang and Nanning Zheng. Measures of correlation for multiple variables. arXiv preprint arXiv: AXiv:/401.482) 2014.
13

---

Penghui Wei, Hongjian Dou, Shaoguo Liu, Rongjun Tang, Li Liu, Liang Wang, and Bo Zheng. Fedads: benchmark for sanaacypreseevvin evr estimation with vertical federated learning. arXiv preprint S20.:::2300008222.. 2023.
Yuncheng Wu, Shaofeng Cai, Xiaokui Xiao, Gang Chen, and Beng Chin Ooi. Privacy preserving vertical federated learning for tree-based models. Proc. VLDB Endow. 13(12)209022033, jul 2020. ISSN 2150-8097 doi: 10.14778/34077790344778
URL /orp://///ooooo...oo/...
edu..edu.om Sc/ 1
14 1tt.//11..1778/3011111 3407811
Zhaomin Wu, Qinbin Li. and Bingsheng He. A coupled design of exploiting record S. Koyejo, Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Infor- mation Processing Systems, volume 35, pages 21087-21100. Curran Associates, Inc., wttpfffaaa/apaa//2222222 ngs www.nnurrrscom www.weip...ecpppppepeeeeeeeeeeee /
similarity for practical vertical federated learning.
2022a. URL
411e/44b744655055522
Zhaomin Wu, Qinbin Li, and Bingsheng He. Practical vertical federated learning with unsupervised representation learning. IEEE Transactions on Big Data, 2022b.
Zhaomin Wu, Junyi Hou, and Bingsheng He. Vertibench: Vertical federated learning benchmark, 2023a. URL UtLhttps::/ htt/ggttuuuuu com Ctwp:/ttaaaoomttnnnn//errrrr Bench
Zhaomin Wu, Junyi Hou, and Bingsheng He. Vertibench: Vertical federated learning benchmark (pypi), 2023b.
Xun Xian, Xinran Wang, Jie Ding, and Reza Ghanadan. Assisted learning: A framework for multi- organization learning. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 14580-14591. Curran Asso- ciates, Inc., 2020. URL ULL/acppee//ppooeeenss..eeuuuppp
cc/paper files/paper 1 2aa.ff/4422222eeoocoeeeeeee61113 Paper.
Jerrold H Zar. Spearman rank correlation. Encyclopedia of biostatistics, 7, 2005.
Qingsong Zhang, Bin Gu, Cheng Deng, Songxiang Gu, Liefeng Bo, Jian Pei, and Heng Huang. Asysqn: Faster vertical federated learning algorithms with better computation resource uti- lization. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery &; Data Mining, KDD '21, page 3917-3927, New York, NY, USA, 202 Association for Computing Machinery. ISBN 9781450383325. htttps:/dbiiorg 1eeg...bbnn wibprcxyl.......m.. nus 1t..............33...... 44444///3445444.444666555 3467169,
doi: 10.114573447548.4476666 URL
Qingsong Zhang, Bin Gu, Cheng Deng, and Heng Huang. Secure bilevel asynchronous vertical federated learning with backward updating. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 10896-10990 2021b.
14

---

Appendix
Table of Contents
A Proof
15
B Details of VFL Algorithms
20
B.1
Ensemble-based VFI. Algorithms
20
Splst-NN-based VFL Algorithms
21
B.3
Split-GBDT-based VFL Algorithms
21
C Split Method Details
22
D) Empirical Validation of Split Methods
23
D.1 Correlation Evaluation Metric
23
D.2 Feature-split Algorithm)
24
D.3 Time Efficiency of Split Methods
25
E Real Dataset Construction
26
F
Experimental Details
28
Additional Experiments
29
Communication Cost Details
29
G.2
Scalability
30
Training Time
................................
30
G.4
Performance on Satellite lie datasel
................................
34
Details of VFL Performancel
35
G.6
Ablation Study of Truncated Threshold SVD
35
G.7
Performance on MNIST and CIFARIO
35
G.8
Performance Correlation: Synthetic vs. Real VFL Datasets (Additional)
40
G.9
Variance of Performance)
40
10
Spearman Rank vs. Pearson
42
H
Discussion
42
Limitations
42
Social Impacts
43
A PROOF
Proposition The probability mass function can be written as
K
Pyy Xi) + log P(y)
log P Ko..,
X1) = log
Σ
15

---

Proof. According to the definition of conditional probability, this marginal distribution can be written
as
X X1)
Ply, X X1)
P(XK,-.. X1)
(X) (8) I (XXA B Xk. X1) P(X1) l p X X1)
(5)
P(X1ly)
P(X X 1,..., X1
P(v) P(X1)
P(X& Xx 1, X1
k=2
Denoting
P(Xxly, X X1)
6q log P
Cj log
(6)
P(X Xx Xx-1.... X1)
P(X,)
Adding logarithm on both sides, we have
K
log X,,
Σ
log o + log F(y)
(7)
Furthermore, we have
ck = P(XxlU. X Xp.
P(Xa X X1)
= P(X 8 X X1)
(8)
Ply X1)
P(y|Xx X,)
Combining 66 and (8) we have
P(y X x.,,,,, + log Ply)
log X X₂) =
log P(y,Xx-1...,,X) X1)
(9)
Proposition 2. For any real mairix X, Pcor(X, X)
mccor X. X)
Proof. In the context the correlation matrix of X, Pcor measures the standard deviation of singular values, while mcor (Taylor 2020) measures the standard deviation of eigenvalues. Given that the correlation matrix is symmetric positive semi-definite each eigenvalue corresponds to a singular value, resulting in Peor equating meor.
Proposition 3. For any two real matrices X1 and X2, Pcor(Xi, X2) E [0,1]
Proof. Denote the correlation matrix between X, and X2 as C, the singular values of C' as = (oi,...,oa] Then
1 (E(E3)) EZ(E))
Pcor(X,,Xx,
V Var(E))
(10)
Thus,
0 < Pcoro 1,,, X2) V (((29)
(11)
16

---

Given that every element in the correlation matrix is in range [-1,
E(Σ²) = C)/d s/d
(12)
Thus, Pcor X1, E [0,
Theorem 1. Consider global dataset X comprising two independent datasets D1, D2 E Rax each of the same dimension. Independence implies that for any feature 0f from D and any (2)
feature aj from D2, where i,j E [1, ml, the correlation Cor (a.") (2)) = 0. Furthermore,
assume within D and 22, all features are perfectly correlated, such that for all pairs of distinct (1) (1)
(2) (2)
features ai :0 in D1 and a, aj in D2. with i.j E [1, m) and + j. the correlations satisfy respectively. When the features of X are divided equally into two subsets, X1 and X2, such that each subset contains m features, the overall inter-party correlation Icor(X1, X2) satisfies
Cor (at (1) (1) =1 and Cor 03 (2)
Icor(X, X1 X²) E
m
0
mm(m-
The lower bound occurs if and only if X1 comprises all features of either D1 or D2, with X₂ containing the remaining features. The upper bound occurs if and only X, holds m features from both Dj and D2, with X2 holding the remaining m. features from D2 and D2-
Proof. Consider two sets of features, X, and X2, derived from datasets D2 and D2. Let X, contain The set X₂
features from D2 and features from D2, where u, [0, m] and comprises the remaining features, specifically v features from D and u features from D2. The inner-party correlation matrices Cn1 for X, and C22 for X2 are given by:
uux
0ux0
Luxe
0 14XM
Cn1
C22
(13)
Doxu
10x
Duxv
To calculate the Pcor(X,, X1) and Pcor (2,, X2), we need to determine the singular values of Cn and C22. As these are symmetric matrices, their singular values are the absolute values of their eigenvalues. We will illustrate the process using C11. with C22 following similar pattern.
The characteristic equation Ci1 is
Uuxa r
guxe
(14)
Vuxe
where U and VTX" are matrices defined as
1-x
UuxU Vex*
Equation 14 simplifies to det( U4x (eet(VVX)) The roots det(U = 0 and det (Vi = 0 are the eigenvalues of 1 uxx and 1xxe respectively. Since 1 1xxu and 10 are rank-1 matrices, they each have only one non-zero eigenvalue, u and v respectively. Therefore, C11 has two non-zero eigenvalues: and The Pcor for X, is
u2 2
Pcor( 1,
(15)
m 1
similar calculation for C22 yields:
22 2
Nor((X,X3))
(16)
1
17

---

Next, we consider the inter-party correlation matrices C12 and C21
1 1ux
0uxu]
1
0
C12
C21 1 1exu
C
(17)
O
nuxu
1 1xx
Since C21 = Ci2 they share the same singular values. A key principle in linear algebra states that the singular values of matrix remain invariant under rotation. This invariance arises because rotating a matrix is mathematically equivalent to multiplying it by an orthogonal matrix, a process that does alter its singular values. By rotating C12 and C21 90 degrees coumter-clockwise, we obtain symmetric matrices Cj and 22 whose singular values are the absolute values of their eigenvalues. As C22 is rank-2 matrix, has two non-zero eigenvalues.
The characteristic equation of C22 is
U 1
1uxe
0
(18)
Vexe
where 2 and Vi2 are defined as
0
aXa
... 0
0
X
0
0
X
0
U
VIXI
(19)
Since focused on non-zero eigenvalues (A * 0), U is invertible. According the property of block matrices, Eq. T8 is equivalent to
det det(U1z
det( (V12 - 1 uxa (U )ux = 0
(20)
Since det (utt(U12 (-A)", and
det(UT22) =
(21)
Given A + 0, Eq. is equivalent to
12
= 0
(22)
X
X2
1
0
(23)
0 u U X2
The roots of Eq. 23 are the eigenvalues of 1 1xx which rank-1 matrix. The only non-zero eigenvalue of u 1x is uv, leading to X we. Thus, the two non-zero eigenvalues of C are +yuv, and the Pcor of C12
2uv m-1
Pcor(X: x2)
(24)
Vm
Combining Eq. 15
16 and
24
the overall inter-party correlation (Icor) is derived as
Icor X, , x²)
(2 * Pcor[X,, X2) - Peor(X,, X,) Pcor(X2, X²))
2
1
2uv
u2 D2
(25)
Vm
m -
mml
m 1
(V2w- v u22 + u2
mm(m -
18

---

By the AM-HM inequality, Icor(X1, X2) I O, with equality if and only if For the lower bound, given u = m v, expressing Icor in terms of v gives
Icor(X1, 12) Vm(m- 1)
1
( (V2mu 20 /202 2mo + m²
(26)
Letting 2mz 2v (e E [0, ml), ( it holds [0, (0,m²// m22]_ Consider the function f(t) Vt vm2 f, the derivative of (tt ensures
ft) 2/8 + 1
2/m vm² -t Vt
(27)
2 2/1(m2 1)
Therefore, f(t) and [c, , are monotonically non-descreasing w.r.t. t. The lower bound of Icor reached when yielding
X2)
Icor X1, X2)
(28)
Vm(m- -
This condition holds if and only v 0 or m.
Theorem Consider a feature index set
f1.2...., m] and a characteristic function 2 - R such that v(0) = O. Let (5(v) denote the importance of the j-th feature on such that -T= x(j) = v(A). Assume that the indices in A are randomly distributed to K parties with probabilities Ti,..., TK ~ Dir(an.....ox. Let 7 be the sum of feature importance for party i. Then, we have Vi [1, K] and [[Zz] az-
Proof. For each feature j assigned to party i with probability Ti, we define the feature importance Yij as:
(u),
w.p. Ti
Yú = 0. w.p.
(29)
By leveraging the property of linearity of expectation, we find that:
m
E|Z
E( = Σ
(v) = Eirl Σ 6())
(30)
j=1
Ø=1
j=1
Given that Σイ Φ(j) v(A), we derive:
E[Z,] E(r:)((A)
(31)
Since the property of Dirichlet distribution asserts that a; EErr) holds
[[Z,] x (Au(AA
(32)
Moreover, since v(A) is constant, it follows that:
E 0:
(33)
Proposition Given a Dirichlet distribution Dir(a ag) with mean variance o, symmetric Dirichlet distribution Dir(a) that has the same mean variance a if K-1-K3a
Kag
Proof. Suppose we have variables XI...., following the Dirichlet distribution, denoted as Dir(a,....,.) a). Leveraging the inherent properties of the Dirichlet distribution, can formulate the variance VXr(X,) for all e [1, K] as
K
Var(X,) = (2(Ka+1)
(34)
19

---

The mean variance, denoted o, subsequently be articulated in terms of the expected variance, E| VVa((X)),, as
K
[[Var(X))] = (²
(35)
Recognizing that for a Dirichlet distribution o > 0 holds, transform the above equation to express a in terms of
K - 11-1-A 1-K'0
(36)
K30
B DETAILS OF VFL ALGORIHTM
In this section, we provide a detailed comparison of the existing VFL algorithms extension of Section 4.1 This section critically reviews current VFI. algorithms, with focus on accuracy, efficiency, and communication size. VertiBench concentrates on standard supervised learning tasks such as classification and regression within synchronized parties, summarized in Table T Notably, this benchmark excludes studies exploring different VFL aspects such as privacy (Jin al. 2021
fairness (Qi et al. 2022), data pricing (Jiang et al.] 2022), asynchronization (Zhang et al. 2021b
Hu al. 2019 Zhang al. 2021a), latency (Fu et al. (022a)). and other tasks like unsupervised
learning Chang et al., 2020). matrix factorization (Li et al. (021b), multi-task learning (Chen and Zhang 2022), and coreset construction (Huang al. 2022). While most VFL algorithms presume accurate inter-party data linking, we adopt this approach in VertiBench, despite recent contrary findings (Wu et al. 2022a Nock al. 202I) that this assumption may not be true.
The mainstream existing methods can be classified into two categories: ensemble-bee basedd and split- based. The distinguishing factor lies in the independent prediction capability of each party. Ensemble- based methods involve parties each maintaining a full model for local feature prediction, with collaborative ensemble methods during training, while split- based methods require each party to hold partial model forming different inference stages of the full model. Consequently, split-based partial models cannot perform independent inference. For split-based models, our focus is on advanced models such as neural networks (NNs) and gradient boosting decision trees (GBDTs) (Chnn and Guestrin 2016), though VertiBench can accommodate various models (Hardy et al., 2017 Gu et al. 2020). -past-NN-based models trained by transferring representations and gradients, while SpDitGBDTT-odee are trained by transferring gradients and histograms. While acknowledging the existence of various VFL algorithm subtypes (Li et al., (2022b), current study primarily addresses the major types of VFL algorithms. The exploration of other subtypes remains a subject for future research.
B. ENSEMBLE-BASED VFL ALGORITTMS
In the ensemble-based VFL algorithms detailed in Algorithm 2 each iteration t commences with the primary party P1 calculating the residual the prior global model Ft-1(.) (-) (line 3). This is followed by the communication of residuals rf to the secondary parties (P2.... ... Px). In the ensuing step (line 5), each secondary party trains local model [(t:: -) on its local data X, to predict the residuals, subsequently sending the model parameters 0 back to PI. P1 then aggregates these local models and updates the global model (=)()) (line 6). This process iterates until a convergence criterion is achieved.
Specifics of residual sharing and model aggregation depend on algorithm design. In AL, residuals are shared among parties, and models are aggregated through summation. Conversely, in GAL, pseudo residuals (i.c., gradients) are shared, and models are aggregated through a weighted summation. Furthermore, the aggregation weight in GAL can updated during the training process.
20

---

Algorithm 2: Outline of ensemble-based VFL algorithms
Input
:Number of iterations T'; number of parties K; data and labels of primary party
X1, learning rate 17²: data of secondary parties X Xx
Output
Models on K parties [(0
Algorithms :AL, GAL
C(P y)),)
AL
Desidual(F*-1) yX),y)
OF C(F (X),y)
GAL
f(0;: 11' i= [0: Xi)
,L
Ei=ı w (0;; Xi)
,GAL
Initialize 0g, ..... 0% initialize F + 0; 2 for t ,.... do
ri f- Resi du (X),y);
/* Pi: Compute residual x/
4
for i ,,..., k do
0: f arg ming C(f(0) x():
/+ Pk: Optimize local model */
F - p (X) + M
/x Pi Update ensemble model */
B.2 SPLIT-NN-BASED VFL ALGORITTMM
As described Algorithm 3 each iteration starts with the parties P. conducting forward-propageee on their local data X; to derive local representation Z (line 4). These representations are subse- quently forwarded to the primary party PT. Depending on the iteration, Pi then merges the local representations (line 6), further derives a global prediction Y with an aggregated model (line 7), updates the aggregated model parameters of (line 8), and broadcast the encoded aggregation model of to all parties (line 9). The parties P. then employ these encoded aggregation model to update their local models (line 11). This process is repeated until a specified criterion for stopping is met.
The specific methods of encoding, determining aggregation frequency, and merging are dependent on the algorithm design. In the process forward -ward--passaa encoding. SplitNN sends local representations di- rectly Party Pj for merging, while C-VFL compresses these representations before transmission. In contrast, BlindFL utilizes a source layer to encode local representations, ensuring privacy preservation. During backward-pass encoding. C-VFL transmits the top-k-compressed aggregation model. Both SplitNN and BlindFL initially compute the gradients with respect to Z and subsequently broadcast either the raw or source-layer encoded gradients all the parties. Regarding aggregation frequency, C-VFL aggregates every Q iterations to reduce communication cost, while both SplitNN and BlindFL aggregate at every iteration. For the merging process, SplitNN and C- use concatenation of local representations, while BlindFL. applies a secret-sharing summation with source-layer-encoded representations.
FedOnce (Wu et al., 2022b), a split-based VFL approach, distinguishes itself by its single-round communication protocol. Secondary parties initially engage in unsupervised learning, aiming to pre- dict noise and subsequently develop local representations. These representations then transferred to the primary party, which utilizes them training a SplitNN model on the primary dataset. This method notably reduces both the communication size and the synchronization overhead.
B.3 SPL.T-GGBDT-BAEER VFL ALGORITHMS
Outlined in Algorithm each iteration initiates with the primary party P1 encoding the gradient of residuals, yielding r' (Tine 3). Following this, all parties P; calculate local histograms H utilizing their individual local data X; and the encoded residuals r' (line 5). These local histograms are then transmitted to Pt for merging (line 6). In the next step, Pt trains a decision tree using the merged histogram H' and the encoded residuals r' (line 7). The selected split points from this tree are communicated to the secondary party that possesses the split feature, which stores these split points for potential future requests during inference (line 9). Finally, P1 updates the ensemble model Fr with the newly trained tree (line 10). This sequence of operations continues until a set stopping condition is fulfilled.
21

---

Algorithm 3: Outline of split-NN-based VFL. algorithms
Input
Number of iterations T'; number of parties K; data and labels of primary party
X1,y, y, learning rate 17": data of secondary parties X,, Xx; local epochs Q
Output
Models on K parties (....)
Algorithms :SplitNN, C-VFL, BlindFL
Z
SplitNN
Encode
EmbeddingCompress
C-VFL
SourceLayer(Z)
BlindFL
82: L(9' y; 01)
SpliiNN
Encode((1)
Top
.C-VFL
SourceLayer
C(y yy y;
BlindFL
oz
True
SplitNN, BlindFL
condition(() t mod Q =0 ,C-VFL
Merge k: 1=1 2 (oncatenate(Z) ..))
SplitNN, C-VFL
SecreethharingSume (Z")
BliinFLL
Initialize 09, ..... 0 2 for f == -=....,,77 do
for i 1, ..... k do L z + (f X,));
Ps: Get local representations */
condition(t) then
Z' M Z
1* Pi: Merge local representations */
g f(0); Z');
/* Pi:
Get global prediction */
8 + 8 1f az C(y',y)
/* Pi:
Update aggregated model x/
g f Encode((f) (0)):
/* Pi: Encode aggregated model
for i 1,....
h do 0g 0.- n/'g!
11
/x Ps: Update local model x/
Depending on the algorithm, the specific techniques for encoding, computing histograms, merging, and updating modes differ. Pivot views the instance set in each node as confidential information, implementing homomorphic encryption on it. Conversely, SecureBoost, FedTree, and VF2Boost treat the instance set public information and apply homomorphic encryption only specific set of instances. terms of merging, SecureBoost and FedTree perform homomorphic decryption directly to acquire actual values after aggregating encrypted histograms. VF2Boost further introduces efficiency measures such as Polynomial basedd histogram packing and reordered histogram accumulation, while Pivot utilizes multi-party computation supporting comparison maintain the secrecy of values at all times. In terms of the update mode, SecureBoost, Pivot, and FedTTree adopt sequential approach, whereas VF2Boost utilizes a pipeline processing for speedup. Additionally, is worth mentioning that SecureBoost employs a threshold for binary classification to enhance accuracy in the context of datasets with label imbalance.
C SPLIT METHOD DETAILS
In this section, we formally state out proposed importance- feature-split algorithm in Algo- rithm 5 After initializing local datasets for each party (line 1), series of probabilities Pı..... PK s.t. 1-1 P; 1 is sampled from Dirichlet distribution, parameterized by o OK (line 2). For each feature, it then proceeds to randomly select a party Pk, according to the probabilities ps. and assigns the respective feature to Ps (lines 3-5). In order address potential failures in algorithms when confronted with empty features, we can optionally initialize each party with a random feature prior to the commencement of the algorithm.
22

---

Algorithm 4: Outline of spli--GBDT-based VFL. algorithms
Input
Number of iterations T'; number of parties K; data and labels of primary party
11, learning rate 17": data of secondary parties X Xx
Output Models on K parties -). ((0k))
Algorithms SecureBoost, Pivot, FedTree, VF2Boost
HomomorphiicBcccryyy... HE =
cww.rrppppppccpppppppccc MPC
hntt:PPartyCOmurattiicccccco @ homomorphic multiplication, a is instance mask HH(r a)
SecureBoost, FedTree, VF2Boost Encode(Z) r 2 HE(a)
Pivot
Hist(X,r) HE(Histogram a. r))
SecureBoost, FedTre,, VF2Boost
Histogram X, r) & HE(()
Pivot
HD(HESum(HH H
k))
SecureBoost, FedTree
Merge H
(aaaaccdeHHSmmeeeoeeeeeeee MPPC x)))
k))))
VF2Boost
Pivot
mode sequence SecureBoost, Pivot, FedTree
pipeline VF2Boost
Initialize 09 pf..... & let federated model (-) t O;
for 1,..., in mode do
r' Encode OF C C(Ft-(FFFFF)) for i 1...., 1: do H +- (X,
( )),));
/+ Pi: Encode gradients +/
1* P.: Compute local histogram */
H' + Merge H of - arg ming: (( H"), for ........ k do
/* Pi: Merge local histograms */ /+ pi: Construct tree +/
7
0 +- Selected split points in H:;
/* P:: Update secondary parties */
10
Ft (H') + p (H- + [0] H P1: Update ensemble model x/
Algorithm 5: Feature Splitting by Importance
Input:
Global dataset X E Ruxmm importance factors "l,,..,KKK
Output: Local dataasets X XI....,X, Xx
X X 0;
/+ Initialize local dataset for P. +/
2 PI.... PK + D.r.. ax); /* Sample probabilities for K parties
*/
3 for j = 1,..., do
k: Random choice from [L,K]NN w.p. (p....,,Pk): /* Choose party */ /x Add feature X|:j] to P */
Dy + D2 U [X]:,]}
return X,...., Xx
D EMPIRICAL VALIDATION OF SPLIT METHODS
To rigorously evaluate the practical performance of our proposed correlation evaluation metric and the correlation-based feature-split algorithm, we conduct a series of systematic experiments.
D.1 CORRELATION EVALUATION METRIC
In order to validate the efficacy of the correlation evaluation metric, Pcor, create two synthetic datasets, X, (m, features) and X (my features), using the sklearn library. Initially, party P holds X1, and Pa holds X², Over the course of the experiment, X, to Pz. each time exchanging for a feature of X2. This process continues until all features of X end up Pz, while the total number of features remains constant during the whole process.
gradually transfer features from
23

---

Our observations, presented in Figure Sa and 5b reveal the following: (1) Peor behaves similarly to meor when evaluating inner-party correlation and shows similar trend to mcor TTalor 2020) when assessing inter-pane siter-patty correlation. (2) Both Pcor and mcor exhibit the lowest inter-party correlation and the highest inner-party correlation at the extremities of the x-axis. This suggests that the datasets X₁ and X2 are managed by distinct parties and exhibit complete independence. This pattern is also reflected in Figure 5c when Pcor is applied to parties of different dimensions. These observations validate the appropriateness of Pcor as a measure for evaluating inter-party correlation, even when the number of features between the two parties varies.
Party Py: number thatines from 9
Party F. number features from Xs
Perty number of featuree from
0.4
0.4
Conelation (Peor)
(ndor
Conelatton indea
0.3
0.2
82-
022
meart-1
Poor -1
0.1
Poot1:
mear2-2
Poor2-2
Pror2-2
Tearl-2
Poort.2
Prr11-2
0.0
Party
Prrmmberr Featurea
Party
namber features Rrmm
Party Pi:number of feetures frem
(a)
mcor: mi 172
10
(b) Pcor: m1 172
10
(c) Pcor: my = 10, m2 = 20
Figure 5: The trend of correlation and metrics when exchanging features between two parties. mcor: multi-way correlation Taylor 2020); Pcor(i)-()): Pcor(X,, X,); mcor(i)-(i): mc Xy).
D.2 FRATURE-SPIIT ALGORITHM
Importance-based Feature Split. We rigorously investigate the correlation hetween {o:} and party importance, using both Shapley value and Shapley-CMI for feature importance assessment. For property (I), in a two-party VFL experiment, we fix a2 = and vary from 10-2 2 to 10a. For each Q2 we compute the scaled a; as as Σg Oj and run 1,000 trials estimate the party importance's expected value. The subsequent relationship between scaled a; and the expected party importance is illustrated in Figure and 6b| demonstrating clear proportionality.
For property (2), four parties are divided under symmetric Dirichlet distribution (where Vi, a) with a values ranging from 10-2 to 103. For each a, the standard variance party importance is estimated over 1,000 trials, visualized in Figure (6c and [dd] These plots signify negative relationship between a = [a; 1 1 and the standard variance of party importance, extending the scenario of random split leading to consistent party imbalance levels.
24
02 d.
standardd
Randor Spl:
2
02
246
237 1271
18
107
(a) a, vs. Shapley
(b) ar vs. Shapley-CMI
(c) a vs. Shapley std.
(d) a vs. Shapley- std.
Figure 6: Relationship between a and party importance: (b) Proportionality of scaled as to party P,'s importance; (c)(d) Negative correlation between a and standard variance (std.) of party importance. The red dotted line indicates the importance variance of random split.
Correlation-basee Feature Split. We turn our attention towards validating the efficacy of our proposed correlation-based feature-split algorithm. Three synthetic datasets, each encompassing 10 features, are independently generated using the sklearn library. These datasets are subsequently concatenated along the feature axis, yielding a global dataset with 30 features. This dataset, with features shuffled, is then split into three local datasets, each containing 10 features, deploying our proposed algorithm with B values at 0, 0.5, and 1.0.
24

---

Specifically, split synthetic datasets with various 3 values using Algorithm (as shown in Figure 7a 7b and 7c and compare this to random split (Figure 7 Our findings reveal that as increases, inter-party correlation correspondingly escalates. Additionally, depicted in Figure 8 when we perform splits on a concatenated feature-shuffled real VFL dataset Vehicle (8 0), VertiBench, when parameterized with the same 83 exactly reconstruct the real feature split of Vehicle dataset. This indicates that VertiBench scope aligns well with the real scope which corresponds scenario with small inter-party correlation further depicted in Figure 9 In contrast, uniformly random split (uniform scope) produces a distinct correlation matrix.
Conveaation
0.62)
1.0
Corviatioo rast
Corvlation -001
2.0
restro (foo 6:03)
Nert
Noto
Tatt
Tutly
26
26
56
10
Puty
24
17
(a) corr-spppi corr-split:/
(b) corr- splr-split: 3 0.5 (c) corr-split: 1.0
(d) uniform-sple
Figure 7: Absolute correlation matrix of the global dataset with party boundaries indicated by red lines. Icor means inter-party correlation. (a).(b)(cc) correlation-belle split; (d) uniform split.
1.4
1.0
1.0
0.2
9.5
0.5
0.6
0.6
0.6
32
50
0.4
9.4
0.4
02
02
100
Puty
100
100
100
100
Ptroy
(a) Real: Vehicle dataset
(b) Synthetic: spilt with same &
(c) Synthetic: uniform split
Figure 8: Absolute correlation matrix of real Vehicle dataset and synthetic datasets generated from feature-shufflee Vehicle VertiBench with the same B vs. uniform split
Correlation metrix
Correlation matrix
1.0
Party
Hely
Party
210
284
50
Pert
0.4
413
Pety
Fett
20
639
100
50
Rerty
100
NUS-WIDD correlation
(b) Vehicle correlation
(c) Satellite correlation
Figure 9: Absolute correlation matrix of real VFL datasets with party boundaries marked in red lines
D.3 TIME EFFICIENCY OF SPLIT METHODS
Table provides a summary of the estimated time requirements for our proposed split methods, with the I/O time for loading and saving datasets excluded. Notably, the importance-battt split method demonstrates significant efficiency, typically completing within a minute. In contrast, the correlation- baased split method requires longer processing time, due to the need resolve three
25

---

optimization problems. This time cost is especially pronounced on dineesiienni datasets, such as zeals a im, because the singular value decomposition (SVD) used in the correlation basdd split algorithm is dependent on the number of features. Despite these differences in time consumption, both split methods prove capable of handling large datasets, accommodating instances up to 581k and features up to 20k, within reasonable time frame.
Table Estimated time cost splitting methods (in seconds)
Performance importance based split
Performance of correlation- split
Dataset
0.1
10
100
3 0
B 0.3
B 0.6
B
covtype
0.21
0.24
0.26
0.27
64.15
55.77
53.19
69.34
msd
0.29
0.27
0.29
0.30
85.39
68.88
64.76
62.01
gisette
0.23
0.29
0.25
0.27
6743.26
6388.56
5640.81
6733.69
realsim
30.11
29.80
30.60
26.68
13681.97
13381.56
11341.31
10057.42
letter
0.00
0.00
0,00
0.00
43.15
46.01
41.50
38.72
epsilon 12.67
12.30
10.36
12.28
4853.63
4395.76
4105.53
3621.99
radar
0.14
0.15
0.16
0.15
308.61
274.06
236.73
299.12
To further evaluate the performance, specifically arranged experiments to evaluate the computa- tional intensity the correlation based splitting method. We conducted tests on synthetic datasets of varying dimensions sizes generated by sk learn Using Pearson-based splitting method single RTX 3090 GPU, the results are detailed in Figure TO
Scalability experiments indicate the efficiency of the proposed split method. For the importance- based split method, it avoids the computational demands of Shapley value calculations while retaining related properties. This method completed within 30 seconds across all tested datasets, ensuring scalability for large datasets. The correlation-base method effectively handles datasets with up to 10k features, aligning with the dimensional needs of common VFL datasets such as reals sim and CIFAR10. Meanwhile, the number of parties hardly affect the efficiency under a fixed global dataset.
240
4020
200
Tiree 000 (50
10.00
160
1200
1000
1100
130
10'
3000
101
Number samotes
Number features
Number partes
(a) Sample size impact (#fcatures=100)
Feature size impact (#samples=1000)
(c) Party size impact (#samples=Hparties=100)
Figure 10: Scalability of correlation-basee split algorithm #samples and #features
E REAL DATASET CONSTRUCTION
In this section, we outline the construction process of Satellite, which was adapted from the WorldStrat dataset (Cornebise et al. 2022), originally intended for high-resolution imagery analysis. The Satell eelelliee encompasses Point of Interest (POI) data, each associated with one or more Areas of Interest (AOI). Every AOI incorporates unique location identifier, land type, and 16 low-resolution, 13-channel images, each taken during a satellite visit to the location.
During the data cleaning phase, scrutinize the dataset thoroughly, identifying and removing 67 incomplete data records that have an insufficient number of rewolustttion images. Furthermore, given the inconsistent widths and heights of images across different locations, standardize the size
26

---

of all images 158x158 square via bicubic interpolation. Additionally, the pixel values of each image are scaled to integer values within the range of [0, 255].
Satellite forms practical VFL scenario for location identification based satellite imagery. Each AOI, with its unique location identifier, is captured by satellite visits. Assuming each visit is carried out by distinct satellite organization, these organizations aim to collectively train model to classify the land type of the location without sharing original images. Satell fatellite llite encompasses four land types labels, namely Amnesty POI (4.8%), ASMSpotter (8.9%), Landcover (61.3%), and UNHCR (25.0%), making the task 4-class classification problem of 3,927 locations.
As depicted in Figure [T] the consistency in capturing the same location 16 times presents variations in image quality. These discrepancies arise due to the changing weather and lighting conditions experienced during each satellite visit. Hence, each of these 16 satellite visits can equivalently be considered as 16 separate parties. All 13-channel images in the satellite dataset that correspond to the description of Sentinel-2 Bands as shown in Table
Party
Party
Party
Party 4
Party
Party
Party 7
Party 8
Party
Party 10
Party 11
Party 12
Party 13
Party 14
Party 15
Party 16
Figure 11: Preview of an area of Satellite dataset at channel-1 with land type Amnesty
Table 3: Channel information of satellite dataset
Channel
Band
Description
Band
Acrosol (443 nm)
2
Band 2
Blue (490 nm)
3
Band 3
Green (560 nm)
4
Band 4
Red (665 nm)
Band 5
Vegetation Red Edge (705 nm)
6
Band 6
Vegetation Red Edge (740 nm)
7
Band 7
Vegetation Red Edge (783 nm)
8
Band 8
NIR (842 nm)
Band 8A
Narrow NIR (865 nm)
10
Band 9
Water Vapor (940 nm)
11
Band 10
SWTR Cirrus (1375 nm)
12
Band 11
SWTR (1610 nm)
13
Band 12
SWIR (2190 nm)
License. Our use of the WorldStrat dataset was restricted the labels and Sentinel-2 imagery, falling under the CC BY 4.0 (Commons) 2023a) license, while excluding high-resolution imagery that falls under the CC BY-NC 4.0 (Commons 2023b) license. Therefore, we have released the Satellis llite dataset under the CC BY 4.0 license.
File description and maintenance plan. We aim to create dedicated website for federated learn- ing datasets to host Satellite and future VFL datasets. Although the website is currently under construction, have made Satellite available via public Google Drive link (Anonymized 2023) for review purposes. The provided ZIP file comprises 32 CSV files, corresponding to training and testing datasets split ratio of 8:2. Bach training and testing file contains 3, 142 and 785
27

---

flattened images from party, respectively. The code of VertiBench (Wu al. 2023a also contains a Satellite loader for demonstration.
F EXPERIMENTAL DETAILS
Datasets. The datasets employed in our experiments exhibit a range of dimensions (from 16 20,95 20,958), instance numbers (from 15k to 581k), and tasks, which include binary classification, multi- class classification, and regression. Detailed information about these datasets and the corresponding licenses are presented in Table 4 and Table
Table 4: Detailed information centralized datasets (N/A means unspecified)
Dataset
Task #samples #features #classes
License
covtype
Blackard
1998
cls
581,012
54
BSD'
msd
Bertin-Mahieux
201
reg
463,715
90
1
CC BY-NC-AA 2.0
gisette
Guyon ct al.
2008
cls
6,000
5,000
CC CCBY4.0 4.0
realsim [Andrew
2015
cls
72,309
20,958
2 N/A
epsilon
(Guo-Xun et
2008
cls
400,000
2,000
2 N/A
letter
1991
cls 15,000
16
N/A
radar Khosravi 2020
cls 325,834
N/A
MNIST Deng 2012)
cls 70,000
784
10 CC BY-SA 4.0 DEED
CIFARIO KKnzhevsky and Hinton 2009
cls 60,000
1.024
10
N/A
tto.ps://ppessuucee
/
Cctaa///eoaaaeoooooooocooooo///r code
Table 5: Detailed information of real VFL datasets
Dataset
Task #parties
#samples
#features
#classes
NUS-WIDE (Chua et al.
2009
cls
5
107. 107,859/161, 7,8,
64/144/73/128/225
Vehicle DDuaree and Hu
2004
cls
2
63,058/15.765
50/50
Satellite (proposed)
cls
16
3,927x16
16x 158
The #samples is written in the form of train/test.
The #features for each parties are divided by The licenses of NUS-WIDE and Vehicle are unspecified. The license Satellite is CC-BY-4.0
Hyperparameters. For models based on split-GBDT, such SecureBoost, FedTree, and Pivot, our experiments are conducted with the following hyperparameters: learning_rate- 1, num_trees=50 max_bin=32, and max_dept h=6. 6= Due to the constraints of dataset sizes in their codes, Pivot is evaluated exclusively on two datasets: the letter dataset under the default setting of MAX GLOBAL SPLIT GLOBAL_SPLIT SPLLISSTTIII NUM-6, COO and on the gisette dataset with MAX_ ELOBAL_SPLITT SPLIT SSLISSPPII-SESSSSS N0M-500, 000. The latter alteration was necessitated by segmentation fault encountered under the default setting.
With regard to Dlit-NN-based models, specifically SplitNN and C-VFL, each local model is trained by two-layer multi-layer perceptron (MLP) with each hidden layer containing 100 units. The corresponding aggregated model is single-layer MIP with 200 hidden units. The learning rate, 3 3 x 10 10-3), is contingent on the specific algorithm and dataset. The number of iterations is fixed at 50 for SplitNN and 200 for C-VFL, with the latter setting aimed at ensuring model convergence. We also test C-VFL using four quantization buckets, a single vector quantization dimension, and top-k compressor recommended in the default setting. The number of local rounds Q in C-VFL is set to 10. In the evaluation communication cost, the split parameter is set to 0.1 since feature split hardly affects the communication size.
chosen from the set (10-1, 10-3
Finally, for the ensemble-base based model, GAL, we utilize a beteiing_rateeee D1, local 2ecal_eppch=20, gl oball __poch=20, and batch_si ze=512, 2, with the assist mode set
28

---

to stack. In the GAL framework, each party employs MLP model consisting of two hidden layers, each containing 100 hidden units.
Environments. The hardware configuration used for C-VFL, GAL, SplitNN,, and FedTree consists 2x AMD EPYC 7543 32-Core Processors, 4x A100 GPUs, and 503.4 GB of RAM, running on Python 3.10.11 with PyTorch 2.0.0, Linux -generre,,, Ubuntu 22.04.2 LTS. For FATE frame- work, we are using federatedal /standa oneoooeeea Docker image, running with Python 3.8.13 on Docker 23.0.2. Pivot is compiled from source using CMake 3.19.7, g++ 9.5.0, libboost 1.71.0, libscapi with git commit hash 1f70a88, and runs slurm cluster with AMD EPYC 7V13 64-Core Processor with the same number of cores as 2x AMD EPYC 7543 used for other algorithms. We conducted real distributed experiments cost, the machines are equipped with AMD EPYC 7V13 64-Core Processors, 503Gi RAM, 1GbE NICs (Intel I350), and varying GPUs (AMD Instinct MI210 for P1 and P2. AMD Instinct MI100 for P3 and Pa). The distributed experimental environment consisted of Python 3.9.12 and Py PyTorc 2.0. 1+rocm55.4.2 We monitored network traffic from the network adapter before and after the ex- periments using ifconfig to measure the actual data size transferred between parties. We also recorded the network traffic size during idle server periods as background noise. Measurements were taken for 1 minute with 1-minute gap between each measurement, totaling 5 times. The mean received data was 56.76 KB (std +1.08 KB), and the mean sent data was 46.50 KB (std #135 KB)
four distinct machines evaluate the communication
Distributed Implementations. FedTree inherently supports distributed deployment through gRPC, a feature we utilize. In contrast, SplitNN, C-VFL, and GAL lack native distributed support. To address this, we implemented distributed support for these algorithms using torch distr distributed. buted. For the sparse matrix representation in C-VFL,, we employ torch. cectase_coo_ttnne
License. The licenses pertinent to the datasets and algorithms utilized in VertiBench are documented in Table and Table respectively. We ensure adherence to these licenses as VertiBench neither redistributes the codes and data nor utilizes them for any commercial purpose.
Table 6: License usage for each VFL algorithm.
Algorithm
License
Share
Commercial use
Redistribute
GAL DDiao al.
2022)
MIT' MIT'
C-VFL. Castiglia et al.
2022
SecureBoost (Cheng et 2021
Apache 2.0
/
Pivol
Wu al. 2020
BY-NC-ND 4.0
FedTree Li et al.
2023
Apache 2.0
/
Note: AL. SplitNN, BlindEL and VF2Boost not specify speciif their licensing terms.
httpss//ccastteeoooooso///////// www.anndd/////eeeeeee nd/4.0/
G ADDITIONAL. EXPERIMENTS
G.1 COMMUNICATION COST DETAILS
In this subsection, we evaluate the total and maximum incoming/outgoing communication costs of various VFI algorithms over 50 epochs. This assessment is performed on four physically distributed machines, with results displayed Figures [2, [ We observe real network adapter communi- cation for both four and two-party settings with ifconfig. Additionally, thcoretically estimated communication sizes are compared in Figure 14] and real-time performance in relation to increasing communication costs for four VFL algorithms presented in Figure 15 The results yield following four observations.
Gradient-boostin algorithms, such as GAL and FedTree, tend to have small communication sizes when contrasted with NN-based methods like SplitNN and C-VFL. Yet, FedTree expe- riences marked communication overhead with high-dimensional datasets. C-VFL, due to its
29

---

different pipeline from SplitNN, fails to reduce real communication cost, especially as it broadcasts compressed representations from all parties too all other parties. The large communication expenses in NN-based algorithms from the recurrent transmission of gradients and representations, contributing to SplitNN's superior accuracy. The high communication costs of Fed'Tree on high-dimensional datasets might be linked to the transmission of feature histograms and node information across parties.
Generally, GAL is more efficient than FedTree, though FedTree sometimes has less incoming communication cost on low-dimensional datasets. The incoming communication cost includes residuals for GAL and histograms for FedTree. Residual size is dependent on the number of instances, while histogram size relates to the feature count. Hence, FedTTree shows small incoming communication cost on low-dimensional datasets but more on high- dighndiiensional datasets, However, FedTrree's 's outgoing communication cost is much larger than GAL., as transmits both gradients and node information, resulting in its large overall communication overhead.
C-VFL struggles to reduce real communication costs via representation compression sent to the server, and it neither enhance the backward server-elient communication. Initially, C- VFL's incoming communication costs align closely with those of SplitNN. This is attributable to the method of estimating the compression ratio of representations by counting the ratio of non- zero elements. Though these representations appear sparse. they cannot be effectively transmitted without added overheads. Specifically, the coordinate format (COO) sparse matrix, utilized in our experiments, demands even greater communication due to the need for additional indices, As a result, such compressions offer limited advantage in practical scenarios. Moreover, the consistent outgoing communication costs indicate that transmitting compressed aggregated models in C-VFL or broadcasting compressed representations does not display efficiency over transmitting uncompressed gradients of the cut layer. This revelation hints at avenues for potential refinement to minimize SplitNN's backward communication expenses.
The real-time performance per communication round aligns with observations made for a fixed epoch. This largely attributed to the dominance of communication per iteration over the number of iterations. GAL achieve similar accuracy levels with minimal communication overhead, a scenario where SplitNN and C-VFL might struggle to complete even single iteration. However, when more communication is allowed, SplitNN typically delivers the most superior performance.
The estimated communication costs generally align with actual observations, with notable exceptions in C-VFL and FedTree when handling high-dimensional datasets. In the case of C-VFL, the estimation overlooks the overhead associated with transmitting sparse matrices. For FedTree high-dimensional data, the reported communication cost likely arises from unaccounted transmission information related to nodes and histograms. Such discrepancies underscore the need for comprehensive testing on real distributed systems for VFL algorithms.
G.2 SCALABILITY
In this section, we examine the scalability of various VFL algorithms on two high-dimensional datasets, depicted in Figure 16 The datasets, split by importance with a = 1. consist of a varying number of parties, ranging from 2 to 2048. Our results demonstrate that SplitNN and FedTree are scalable to thousands of parties without any significant drop in accuracy. This is attributable to FedTree's 's lossless design and SplitNN's robust structure. However, both GAL and C-VFL show substantial performance declines with an increase in party numbers.
An intriguing observation is that C-VFL's accuracy nearly matches SplitNN's when the number of parties reaches 2048 on the gisette dataset. This is likely because the average number of features per party reduces in this scenario, causing the compression mechanism potentially fail, and thus, C-VFL reverts to SplitNN's performance.
G.3 TRAINING TIME
The training duration for VFL algorithms is consolidated in Table It should be noted that FedTree and SecureBoost are executed without the of encryption or noise. Conversely, we retain the default privacy setting for Pivot as it does not offer a non-encryption alternative. Three observations can be gleaned from the table.
30

---

Total communication cost of VFL aligorithms
RedThee
Communication Size (M88)
10
10a
10
covtype
msd
gisette
realsim
epsiion
letter
radar
Max incoming communication cest VFL algorithms
SQIRNN
CVRL
GAL
RodiToee
Communication Size (XS)
10
10a
covtype
méd
realsim
epsilon
letter
radar
Max outgoing communication cost of VFL algorithms
10
SpirhN
CVTL
col
Communication Size (MB)
10a
10a
covtype
msd
gisette
realsim
epslion
letter
radar
Figure 12: Real total, incoming and outgoing communication size across parties of VFL algorithms (50 global iterations)
31

---

Total communication cest of VFL aligorithms
10
RedTree
Communication Size (M8))
10a
covtype
mid
gisette
realsim
epsiion
letter
radar
Max incoming communication cest of VFL algorithms
10
SpIRNN
CARL
GAL
RodiToce
Communication Size (XS)
10a
10a
covtype
realsim
epsilon
letter
radar
Max outgoing communication cost of VFL algorithms
SpHtNV CUTL
GAi
Recilbee
Communication Size (M1)
100
10a
covtype
msd
gisette
realsim
epsiion
letter
radar
Figure 13: Real total, incoming and outgoing communication size across parties of VFL algorithms (two parties, 50 global iterations)
32

---

Total communication cost of VFL algorithms
10a
Communicatin Size (Md)
10
10a
107
SplRNN
10
GAL
Securelloast
covtype
msd
gisette
realsim
epsilon
letter
radar
Figure 14: Estimated total communication size of VFL algorithms (50 global iterations)
covlype
100
60
40
Accuracy
2500
3000
10000
12500
15000
200
600
800
1000
500
1500
2000
epsilon
letter
Feder
100
80
40
SpiiNNN
GAL
20
CAVFL
FodTree
1000
3000
4000
200
250
500
750
1000
1290
1500
Communication cost (MiB)
msd
SplRNN
0.8
GAL
C-VILL
FedTree
0.6
RMSE
0.2
1000
2000
3000
4000
5000
Communication cost (MIB)
Figure 15: Test performance as the increasing communication cost (MiB)
33

---

gisette
realsim
1.0
0.9
Accuracy
Accuracy
0.9
0.8
SpitNN
0.8
SpiRNN
0.7
CAVLL
CAVFL
Featree
0.7
Fedtree
128
512
2048
32
128
512
#parties
#parties
Figure 16: Scalability tests for VFI. algorithms on gisette and reals sim dataset
Firstly, we observe a considerable overhead associated with the encryption processes of Pivot. Pivot, which employs both homomorphic encryption and secure multi-party computation to ensure stringent privacy, endures training time that is up to 10 times longer than FedTree. This limitation renders such strict privacy measures impractical for real-world applications that employ large datasets. This observation underscores the necessity for further exploration into the efficiency-priiie -rricccyyp- trade-off in VFL.
Secondly, when comparing non-encryption methods, find that split-based algorithms (SplitNN, FedTTree) generally outperform ensemble-based algorithms (GAL) in terms of efficiency. This is primarily because split-based algorithms require each party train partial model, whereas ensemble- based algorithms mandate that each party train an entire model for ensemble purposes. This design characteristic also contributes to the lower communication costs associated with ensemble-based algorithms, as demonstrated in Figure 12
Lastly, we note that SplitNN demonstrates higher efficiency than FedTree on high-dimensional small datasets, yet demands more training time on low-dimensional large datasets. This discrepancy arises because FedTTree computes fixed-size histogram for each feature, which alleviates the impact of a large number of instances but is sensitive to the number of features. Conversely, SplitNN trains data in batches, rendering it sensitive to the number of instances. This observation emphasizes the importance of carefully selecting VFL algorithm based on the properties of the dataset the application.
Table 7: Estimated training time of VFL algorithms (4-party, in hours)
VFL Algorithm
Dataset
SplitNN
GAL
FedTree
SecureBoost
C-VFL
Pivot
covtype
0.19
2.18
0.076
14.84
1.45
msd
0.17
2.01
0.037
0.28
1.29
gisette
0.01
0.03
0.052
0.56
0,02
25805.52
realsim
0.06
1.40
0.32
1.16
0.48
epsilon
0.15
1.79
0.33
7.99
1.11
letter
0.06
0.47
2.69
0.04
6750.76
radar 0.12
1.31
0.21
10.06
0.83
A segmentation fault arises Pivot when datasets encompass more than 60,000 samples. Conse- quently, only the gisette and letter datasets have their training times estimated.
G.4 PERFORMANCE ON SATELLITE DATASET
In Table 8 present the single-pane party and VFL performance results the Satellite dataset. For an equitable comparison, each single party trains concatenated MLP, formed by linking a SplitNN's local model with its aggregated model, under the same hyperparameters. Our results indicate that
34

---

VFL can yield approximately 10% accuracy improvement local training, thus affirming the practical utility of the Satellite dataset for vertical federated learning applications.
Table Accuracy of single party training (Solo) and VFL with SplitNN on Satellite dataset
Method
Accuracy (mean +: standard deviation)
Party P. (Solo)
0.7368 = 0.0086
Party P1 (Solo)
0.7310 0.0054
Party P3 (Solo)
0.7320 + 0.0055
Party Pa (Sola)
0.7289 + 0,0115
Party P4 (Solo)
0.7251 + 0,0080
Party Pt. (Solo)
0.7167 + 0.0097
Party P% (Solo)
0.7353 + 0.0045
Party Py (Solo)
0.7256 + 0.0139
Party Ps (Solo)
0,7335 + 0.0126
Party Pg (Solo)
0.7154 = 0.0049
Party Pio (Solo)
0.7264 0.0172
Party Pu (Solo)
0.7169 0.0106
Party P12 (Solo)
0.7182 1 0.0075
Party Pia (Solo)
0.7118 + 0.0084
Party P14 (Solo)
0.7060 = 0.0119
Party Pia (Solo)
0.7292 0.0069
Party Pis (SplitNN)
0.8117 0.0035
G.5 DETAILS OF VFL PERFORMANCE
In this subsection, we present the detailed information (including msd) in Table Besides the previous observations, we make two additional observations the mad regression dataset. First, compared among algorithms, split-based algorithms still have leading performance. Second, compared among split parameters, observe that both and / have slight effect the performance on msd.
G.6 ABLATION STUDY OF TRUNCATED THRESHOLD OF SVD
In this subsection, we assess the Pcor of each dataset using VertiBench, varying the truncated threshold d,. We quantify both the speedup defined as the execution time ratio between exact and truncated SVD and the relative error, as outlined in Eq. 37
Relative Error Approximate Pcor Exact Pcor
(37)
Exact Pcor
From the presented data, it is evident that high high-ddmeensionl datasets such as gisette, realsim epsilon, and radar benefit from notable speedup with minimal error. In contrast, for lower- dimensional datasets like covtype, msd, and letter, the speedup is marginal and accompanied by a significant relative error. Consequently, in our experiments, we employ exact SVD for datasets comprising fewer than 100 features. For those exceeding 100 features, utilize approximate SVD with de = 400.
G.7 PERFORMANCE ON MNIST AND CIFAR10
In this subsection, we extend VertiBench to accommodate image datasets, with experiments conducted on MNTST and CTFAR10 0 revealing novel insights.
Image Dataset Splitting. Unlike tabular datasets, image datasets hold crucial positional information, implying that feature order is pertinent. Adapting this characteristic. flatten the image and perform the split using VertiBench, akin to tabular datasets, all while preserving positional metadata. Post-splitting, the features reconfigured their original image positions, and any absent features
35

---

Table 9: Accuracy/RMSE of VFL algorithms on different datasets varying imbalance and correlation
Performance of importance-based split
Performance of correlation-bused split
Dataset Method
0.1
o=
==
3
0.6
3=
SplitNN 91.2+0.4%
92.1=0.2%
92.1=0.3%
92.1=0.1%
92.0=0.2%
92.1=0.2%
92.3=0.2%
92.1=0.1 %
GAL
66.1=2.9%
60.2+1.1%
62.1+4.1%
61.5+3.9%
63.1+2.3%
62.9:2.4%
63.0+2.4%
64.0=1.6%
covtype FedTree
77.9=0.1%
77.8+0.2%
17.8
17.8
77.8+0.2%
77.8+0.2%
77.8+0.2%
77.9+0.2%
C-VFL
46.924.1%
14.8=19.0%
26.9425.3%
38.3+18.6%
26.6418.2%
36.3+17.4%
39.2=15.4%
46.626.7%
FedOnce
68.945.9%
70.2#2.6%
74.4=3.1%
75.0=1.5%
71.6+3.5%
73.421.7%
75.9+0.8%
73.8+2.5%
SplitNN
0.1010:0.0
0.1020:0.0
0.1010:0.00
0.1010:0.0
0.1015#0.0
0.101520.0
0.101320.0
0.101020.0
GAL
0.1220=0.0
0.1220=0.0
0.1220=0.0
0.1220#0.0
0.1222+0.0
0.1222+0.0
0.1222=0.0
0.1222+0.0
msd
FedTree
0.1040#0.0
0.1040=0.0
0. 0.1040=0.0
0.1040#0.0
0.1035+0.0
0.1035#0.0
0.1035=0.0
0.1035+0.0
C-VFL
0.1270=0.0
0,1270=0.0
0.1270=0.0
0.1270=0.0
0.1910#0.0
0.1708#0.0
0,1878=0.0
0.1830=0.0
FedOnce
0.1110=0.00
0,1103+0.00
0.1080=0.0
0.1079x0.0
0.1100=0.0
0.1097#0.0
0. 0.199000.0
0.1092+0.0
SplitNN 96.8+0.4%
96.8+0.3%
96.9+0.3%
96. 06.9=0.254 .%%
97.0=0.4%
96.9+0.6%
96.9+0.3%
96.7+003%
GAL 94.4+1.7%
95. 75.6%%
96.2+0.6%
96. 1=0.6%
95 95.4=0.55% 5%%
95 35.5+0.5%%
95 6+0.5%
95 05.6%
gisette
FedTree 97.0+0.3%
97.1=0.5%
97.1=0.5%
97.1=0.3%
97.0=0.4%
96.9+0.5%
97.0=0.3%
97.1=0.4%
C-VFL
91.0=11.5%
97.8+3.4%
1.8%
94.0=4.0%
95.021.9%
95.7#1.2%
95.0%1.2%
94.622.0%
FedOnee
91.0+9.8%
96.4+0.6%
96.3+005%
96. 06.2%0.5%%
95.6+0.4%
95 95.6%0.5%%
96.0=0.7%
95.7
SplitNN 97.0=0.0% 97.110.0%
97.0=0.11%
97.1.0.1 %
96.9+0.2%
97.0+0.2%
97.0=0.1%%
96.9#0.2%
GAL 93.5+2.2% 92.420.7%
12..4%
96.420.1%%
96.5+0.2% 56.520
96.520.2% 96.5+0.2%
realsim FedTree
80.1%
85.2+0.2%
150.%%
85.1
85.2+0.3%
85.2+0.2%
85, 1+0.2%
C-VFL
89.6=17.4%
88.6211..74
89.5+17.4%
87.8+16.11 1%
95.942.19 1%
96.5+0.8%
94.9++.6%
3.7%%
FedOnce
94.1+3.0%
1
95.6+0.3%
95.2+0.2%
94.9+0.3%
94.9+0.3%
95. 1
04.2%
SplitNN 86.3+0.1%
86.2+0.1% %
86.3=0.0%
86.2=0.0%
85,9=0.1% 1
86,0=0.11 %
86.1=0.19 %
86,2+0.1 %
GAL 88.220.6% %
85.9+1.5%
85.7+0.4%
86.3+0.2%
35.840445%
85.820.4%
85.8+0.49)
85.8+0.4%
epsilon
FedTrre 77.2+0.1 1% 50.1#0.0%
77.3+0.1%
77.2+0.0%
77.2+0.1%
77.240.11 1%
77..240 1%
77.3+0.0%
77. 77.220.19%
C-VFL
53.5+4.9%
67.7420.0%
73 73.847.599 61.6+16.4%
65.0#11.1%
71.3%13.5%
69.3%%4.8%
FedOnce
76.6=12.8%
79.5+3.0%
81.743.0%
80.7=1.7%
79.7
80.1+1.6%
78.342.0%
77.7+1.0%
SplitNN 95.5+0.3%
96.0=0.3%
96.0=t03%% 96.1=0.2%
95.0=0.399
94.7+0.4%
95.0=0.3% 94.7=0.4%
GAL 57.9s4.1%
11.413.1%
52.0a4.3%
19.1122.%
48.9&1.994
18.9%1.9%%
1.90 18.9&1.8%
letter
FedTree 91.7+0.3%
91.9+0.4%
92.0+0.3%
92.0+0.3%
91.8+0.3%
91. 91.940.4%
91.9+00.%
92.0+0.3%
C-VFL
5.0=1.0%
33.%%..3%
37.1
43.824.5%
69.424.2%
70.4+2.4%
70.6+3.7%
68. 155.7%
FedOnce
50.3+9.6%
55.6+339%%
6..1
60.0=1.1%
60.242.1%
59.813.6%
59.1
60.4+2.2%
SplitNN 99.740.0%
99.8+0.0%
99.8+0.0% 99.8+0.0%
99.8+0.0%
99.8+0.0%
99.8+0.0%
99.8+0.0% 0
GAL 90.0+3.6%
95. 95.9+2.5%
97.5=0.4%
97.9=0.19. 1%
96.5+0.11 1%
96.5+0.1%%
96.5+0.1 1%
96.5+0.11 %
rudar
FedTree 99.3+0.0%
99.3+0.0%
09.3+0.0%
99 3+0.0%
99.340.047
99.3+0.0%
99.3+0.0%
99.2+0.0%
C-VFL
59.9247.8%
43. 13..9%999% 53.4238.5%
37.3+15.7%
35.5+18.6%%
44.5=17.9%
53.2+19.8%
40.0=15.0%
FedOnce
97.7+1.6%
98,4+0.2%
98.440.2%
98 4+0.1%
18.1
98.140.1%%
98.1
98.3+0.19
36

---

150
125
covtype
Spccdup Broor
0.5
1250
15:0
gisette
Error
Spcoduu
15,00
realsim
Enor
Spcodup
0.3
Speeeuus
1030
Spaeenu
20000
00
750
0.2
25
0.1
0.3
5300
02
0.0
epsilon
letter
radar
Speecup
250
Speedup Eror
150
Speedup
Emrer
125
Enor
130
0.06
0.3
Erron
100
75
0.02
0.3
0.3
0.0
mid
150
Spoodup Enor
Spereuuc
300
0.3
50
0.0
102
Figure 17: Effect of de on relative error and speedup
are replaced with the background color black for MNIST (Figure 20 and white for CIFAR10 (Figure 19.. 19...
Visualization Insights. The processed images post-split elucidate the effects and B. Analyzing by party importance, a, observe that smaller values result in certain parties retaining recog- nizable image details (c.g.. party 2), while others contain little meaningful information. Conversely, larger distributes image data more equitably across parties. Addressing party correlation, B, especially in the MNIST context, is observed that a smaller 3 ensures each party is allocated distinct parts of an image. For instance, with 0.0, party 3 predominantly possesses the left portion of the digit "8", while party obtains its right portion. In contrast, a larger 3 induces high inter-party correlation, with all parties vaguely representing the digit. This trend is less pronounced in CIFAR10 due to its narrower Icor range.
Experiments. In our experimentation on image datasets, utilize the ResNet18 architecture Figure 22 presents the outcomes and suggests three new observations:
GAL consistently has leading performance, particularly on CIFAR10. The performance of GAL can be attributed to the relatively large number of parameters of ResNet18, In contrast, the use of split-NN- excessive parameters leading to overfitting.
algorithms, which concatenate models, may introduce
On imbalanced VFL datasets, the efficacy of GAL is reduced. This decrease in per- formance might be due to less-informative parties introducing noise during the boosting process.
C-VFL demonstrates a close performance to SplitNN in image classification tasks, which potentially rooted in the sparsity of image representations.
37

---

cifer20 2.1.1 sampleno
cffar20 a=10 seedi-0 sample
Thai
Party?
Partyd
Combined
Party
Taln
Tain Combinne
lest
lest
Rest Party
Tost Party
Combined Image
lest Harty
lest Party
Party
Iest Party
Combéned Image
ctfar10
seed--
cttaro a-100.0
sample-0
Train
Combined
Combéred
Teal
Cambénall Imruge
Harty
Harty
kad Cambenad Imagu
Figure 18: Visualization of CIFARIC split by different
elfar10 8-0.0 socci sample-0
citar10 8-0.3 socei- sample-0
Fain Party
Fain Party
Bain Harts
Fain Parte
Corbénned Imrgge
Fain Party
Tain Party
Fain Parte
Tain
Comiénse Image
Harty
Narty
Narty
Imagge
Hany
Party
Party
Image
cifer1o Smb.e seedmó semple=t
cifar30 $-1.0 seediO sample-0
Thain Party
Tain Party
Trin Party
Trin Party
Combined Image
Tain Party
Tain Party
Tran Party
Tain Parzy
Combined Image
That Party
Thet Party
Test Party
Test Party
Combined Image
Tee Party
Party
Test Party
Test Party
Combined Image
Figure 19: Visualization of CTFAR1 split by different
soedi-0 sample-0
mnist sample-0
Party
Party
Image
Party
Paryy
3
3
8
8
sample=0 farty
minist
sample-O
Party
Party
Vain
Image
Party
Yuin
Imagu
3
3
$
Figure 20: Visualization of MNI split by different
38

---

mnist
mnist
seed=00
Trair
Combéned
Combéred
3
3
leal
Marty
leal Harly
Combinad Image
Harty
kcll Harty
koll Harty
Combénad Image
8
8
mnist sead=0
mnist
send=0
Train
Yein
Combined
Combéned
3
3
Test
Party
Teak Marty
Combinee Image
Party
Teak Party
Nost
Combéned Image
8
Figure 21: Visualization of MNIST split by different
CIFRR1
0.525
0.590
0.650
0.990
0.000
0.980
0.600
0.900
0.575
0.955
0.550
0.970
0.50
0.970
0.965
0.500
0300
0.960
0.0
0.8
1.0
SpittNN
GAL
C-VLL
Tederee
FedOnce
Figure 22: correlation(3)
The accuracy of MNIST and CIFAR10 using ResNet18 varying imbalance(a) and
39

---

G.8 PERFORMANCE CORRELATION: SYNTHETIC VS. REAL VFL DATASETS (ADDITIONAL)
In this subsection, we extend the experiments of Section 4.4 and showcase the detailed performance on each dataset real and synthetic datasets in Figure 23 Our experiments yield primary observations, suggesting the alignment of VeritBench scope and the real scope. Firstly, the relative per- formance ranking of VFL algorithms remains consistent across real and synthetic datasets. Secondly, for algorithms sensitive to data splits like VFL. the trends observed on synthetic datasets generally align with real ones. For instance, C-VFL. exhibits superior performance on the balanced two-party Vehicl clee e dataset compared to the imbalanced five-party NUS-M IIDE. Similarly, it fares better on balanced two-party synthetic data than on imbalanced five-party datasets, pattern consistent across synthetic datasets derived from both covtype and letter.
1.0
covtype
1.0
letter
AccuO
0.4
1-10-44
INLS MD0
SASMIDE
1-30-44
8.00-14
JAfical
MASAIEE:
oisette
0.6
0.8
Acturary
0.2
0.2
NEWEERSS
K-2 Neekel
Kx2
Znd
NESEEERS
6+2.00
2-2
3-2
9-0 e-14
Neta
[MUS/00
radar
1.0
med
0.6
0,6
ROYSE
0.0
Ce5 240
0xg SANics
0.0
0404
RI5-WEE- ANd
Ked 2eo
Des 240
Dxt 2-5.4
AuSATEE:
SpItNN
Figure 23: Performance of VFL algorithms real synthetic datasets with consistent a, 3
G.9 VARIANCE OF PERFORMANCE
In our experiment, we examined the performance variance of VFL. algorithms under a consistent or 3, revealing that VFI. accuracy affected by the randomness of feature partitioning. Figures 24 investigates the convergence of both mean and variance in algorithm performance as the number of test repetitions increased. Figure 25 illustrate the performance distributions under the same a or 3.
Figure 24 reveals that both mean and variance stabilize after 5-10 iterations, showcasing distinct performance variance across different algorithms. Notably, C-VFL displays the highest variance, while SplitNN and FedTree demonstrate consistently low variance, underscoring the algorithm- dependent nature of performance variance. Additionally, Figure 25 illustratss that most algorithms exhibit concentrated distribution of results.
40

---

1220
20
wewe--ddde-.. sdlld addedd 66663
SpitNN
GAL
CV
FooTree
FooOnce
Figure 24: Standard deviation of accuracy as number of seeds tested
SpiRNN
GAL
C-VLL
FedTree
FedOnce
10
12
10
54
85
50
55
900
91
92
50
55
60
Accurscy 1961
letter when 10.0
SplitNN
GAI
C-FFI
FedTree
FedOrce
10
8
2
95.5
(b) Letter when 1.0
SpliINN
GAL
C-VFL
FedTree
FedOnce
12
10
8 75
93.775
39.800
59.825
36
97
59.20
99:25 93.30
58.00 38.25 $8.50
Accurscy (56)
(c) radar when 10.0
SpliNN
GAL
C-VFL
FedTree
FedOnce
(d) radar when 3 = 1.0
Figure 25: Accuracy distribution of VFL algorithms
41

---

G.10 SPEARMAN RANK VS. PEARSON
In comparing Spearman rank and Pearson correlation coefficients, we find Spearman rank effective for non-linear correlations, while Pearson is suitable for linear correlations. Figure 26 illustrates the performance differences on datasets split using these metrics.
For linearly correlated datasets like gi sette, Pearson's 's coefficient is preferable. It accurately captures the enhanced performance of C-VFL on highly correlated splits, nuance Spearman rank- based splitting misses. Conversely, for non-linearly correlated datasets such as radar, Spearman rank-based correlation is more effective, highlighting variations in C-VFL performance that Pearson- based splits overlook. Therefore, the choice of correlation coefficient should align with the dataset's correlation characteristics.
giselte
leter
rodar
100
96
80
Acqurecy
94
70-
60
40
88
50-
20
0.6
0.6
SpItVN
GAL
CVLL
Federee
FodOnce
(a) Pearson
gisette
letter
radar
90
Accuracy
96
80
85
60
50
0.6
1.0
0.5
1.0
8
SpItNN
-4-
GAL
C/FLL
*
Foderee
FodOnce
(b) Spearman rank
Figure 26: Accuracy of VFI. algorithms on different datasets varying correlation metrics
H DISCUSSION
This section discusses the limitations of VertiBench, shedding light on several areas where improve- ment is needed. Additionally, we engage in a discussion surrounding potential negative social impacts and personal privacy issues related to VertiBench.
H.1 LIMITATIONS
In this subsection, we outline the limitations of VertiBench, focusing on three primary aspects.
Scalability of correlation-based split. The correlation- based split method that we propose may face efficacy and efficiency challenges when applied to large number of parties. As the number of parties increases, the potential feature splits proliferate exponentially. This complexity presents significant obstacle for optimization methods such BRKGA (Gonçalves and Resende, (011)). making it challenging to locate the minimum and maximum Icor, as well as the optimal split that corresponds to the given B. This situation underscores the necessity for more advanced permutation-
42

---

based optimization algorithms that can enable the inrellttion-assee split method to scale out to greater number of parties.
Relationship between importance and correlation. Within VertiBench, we regard importance and correlation as two orthogonal factors impacting the feature split. However, this viewpoint might overlook the potential correlation that could exist between these two factors. For instance, in cases of highly imbalanced feature split, parties might demonstrate low inter-party correlation. As result, a comprehensive benchmarking framework that simultaneously considers both importance and correlation is desired to provide more rigorous evaluation of VFI. algorithms.
Designing a benchmark that captures the intricate relationship between feature importance and correlation presents significant challenges, as certain levels of correlation and importance may be mutually exclusive. Designing a co-optimization algorithm for both feature importance and correlation while maintaining efficiency and explainability presents significant challenge. Balancing the demands of these metrics could lead to compromises in both computational efficiency and the explainability of the resulting data splits. Developing approach that effectively integrates these factors without sacrificing these key aspects remains a complex, yet crucial task in advancing VFL. benchmarking methodologies.
Evaluation of privacy. Although VertiBench assesses performance, efficiency, and communication cost, does not provide quantitative evaluation of privacy. The high performance observed with SplitNN could potentially come the cost of privacy, while the markedly high overhead of Pivot might be attributed to its robust privacy requirements. The task of quantitatively evaluating the privacy of different VFL algorithms and models remains open problem, which we aim tackle in future work.
H.2 SOCIAL IMPACTS
Negatíve social impact. While VertiBench primarily focuses analyzing and comparing existing methodologies, and hence is less likely to cause additional negative social impact, the potential for biased interpretation of our experimental results could inadvertently mislead future research or applications. Specifically, we emphasize that the superior performance of non-encrypted methods such SplitNN and GAL does not necessarily indicate that they are fit for immediate deployment in real-world VFL applications. The privacy concerns arising from the transfer of residuals or representations require further investigations. A quantitative benchmark on privacy critical prerequisite to deploying VFL approaches in real-world applications, which we plan to explore in the future research.
43


Reviews:
[{"review_id": "HtzsUGtVSf", "reviewer": "Reviewer_MyAZ", "summary": "Because existing vertical federated learning methods either use randomly feature-partitioned dataset or a few public real-world datasets, this paper focuses on understanding the existing datasets’ characteristics (through dataset evaluation) and on enable wider feature distribution (through synthetic data construction). Specifically, it focuses on two factors — feature importance and feature correlation — and proposes evaluation metrics ($\\alpha$ and $\\beta$) and algorithms on how to split datasets according to these metrics. Given the proposed data construction methods, the authors compare existing VFL algorithms over the newly generated benchmarks and provide insights about their performance and communication efficiency.", "questions": "- Can the authors provide further discussion of the scalability of their two partitioning methods? For example, how scalable is the computation of Shapley values used in the experiments (run-time as a function of the number of features)? How many partition dimensions can the optimization method BRKGA efficiently solve?\n- Minor: Should the probablity $p_k$ (three lines above Theorem 1) be $r_k$?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The paper focuses on an important yet often less-explored aspect of dataset evaluation and construction.\n- The authors have provided algorithms to construct synthetic feature partitions covering a wider range of feature distribution than existing benchmarks.\n- The insights obtained from VertiBench by comparing different split learning algorithms over different $\\alpha$ and $\\beta$ values are interesting.", "weaknesses": "- **The specific form of the correlation of two party correlation $\\textrm{Pcor}$**. It’s not clear to me why the metric Pcor is defined as roughly speaking (upto an additional $\\frac{1}{\\sqrt{d}}$ constant) the standard deviation of the correlation matrix’s singular values. Consider a case where $m_i = m_j$ and the two parties have exactly the same set of features $X_i = X_j$ where the column features of $X_i$ are pairwise independent. In this case, the correlation matrix is the identity matrix $\\textrm{cor}(X_i, X_j) = I_{m_i \\times m_i}$ with singular values all being $1$. Then under the metric Pcor, these two parties' features would have 0 “correlation” because the singular values have zero variance while in reality there is indeed a somewhat strong notion of correlation (their features are identical). I would appreciate if the authors can provide an explanation of the reasoning in choosing this new metric and whether my suggested scenario would indeed be measured as having 0 “correlation” using pcor.\n\n- **Exploring the performance variations of feature partitioning under the same $\\alpha$ and $\\beta$**. For party balance using $(\\alpha_1, \\ldots, \\alpha_K)$, the feature partitioning procedure (algorithm 5) has internal randomness, thus making two runs of the same algorithm produce feature partitions that are likely different. In this case, I would expect to see some results on the performance distribution of the same split learning method over these different feature splits and understand how much variation there is. Similarly, I’m not sure if the argmin step (line 6) of Algorithm 1 could also yield multiple different solutions. If so, I think the authors should also explore the performances variation over these different partitions. Taking one step further, I wonder if there are algorithm ranking differences for the same $\\alpha$ and $\\beta$ but different feature partition realizations. If there are such ranking differences, then the authors should provide more detailed suggestions on how to use their benchmark (how many such splits should be used/averaged for such comparison). If there aren’t, I suggest the authors provide official splits after performing the sampling/optimization themselves so that future papers using these evaluation benchmarks would have a consistent comparable reporting.\n\n- __Proposition 1__ can be trivially proved through taking logarithm of both sides of a telescoping product equality: $\\frac{\\mathcal{P}(y | X_K, \\ldots, X_1)}{\\mathcal{P}(y)} = \\prod_{k=1}^K \\frac{\\mathcal{P}(y | X_k, \\ldots, X_1)}{\\mathcal{P}(y | X_{k-1}, \\ldots, X_1)}$. However it’s not clear to me whether this likelihood ratio quantity really has deep connections to Shapley or the feature correlations, or is simply used as a heuristic quantity to motivate the story of feature balance and correlation.\n\n- **Figure 2, 3, 4.** I appreciate the authors including Figure 2, 3, 4 in the paper. However, I believe they are basic sanity checks of the correctness of Theorem 1 and the correctness of the implementation of Algorithm 1, and do not provide additional new insights. I believe they should instead be put into the Appendix.\n\n- __Communication efficiency__ Section 4.4 seems detached from the rest of the paper, as they do not explore in the axes of party balance ($\\alpha$) and party correlation ($\\beta$) explored in the rest of the paper.\n\n- Minor typos: In Equation 1, the index should be $k$ instead of $i$. In Algorithm 5 (line 4), the $\\cup$ should be $\\cap$.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ipzbyiCqoG", "reviewer": "Reviewer_oXEA", "summary": "The paper introduces \"VERTIBENCH,\" a novel contribution aimed at advancing the state of benchmarking in the domain of Vertical Federated Learning (VFL). VFL is a crucial paradigm for machine learning on feature-partitioned, distributed data. However, the lack of public real-world VFL datasets, and the limited diversity of feature distributions in existing benchmarks, hinders algorithm evaluation. To address these limitations, the paper introduces two essential factors in VFL performance evaluation: feature importance and feature correlation. It proposes associated evaluation metrics and dataset splitting methods to enrich benchmark datasets and provide a more comprehensive assessment of VFL algorithms. The paper presents a thorough evaluation of state-of-the-art VFL algorithms, offering valuable insights for future research in this field.", "questions": "Please see weaknesses.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper effectively addresses a critical limitation in the field of VFL benchmarking. The lack of real-world datasets and the limited diversity of feature distributions in existing benchmarks have been significant challenges. By introducing novel factors and associated evaluation metrics, the paper enriches benchmark datasets and provides a more robust framework for evaluating VFL algorithms.\n\n\n2. The paper conducts a comprehensive evaluation of state-of-the-art VFL algorithms. This empirical assessment provides valuable insights into the performance of these algorithms and helps researchers and practitioners in making informed decisions regarding algorithm selection and development.\n\n3. The proposed benchmarking framework and insights provided by the paper contribute to the advancement of research in the field of VFL. By addressing the need for more diverse and real-world benchmark datasets, the paper lays the foundation for future investigations and developments in this area.", "weaknesses": "1. The paper highlights, in Figure 1(b), the relatively small scope of existing VFL datasets. This raises questions about the alignment of the proposed VFL dataset synthesis approach with real-world scenarios. Rather than expanding the scope to what may be nearly impossible in practice, it might be more appropriate to delve deeper into the real-world scope. While I recognize that Section 4.5 includes experiments to demonstrate the relationship between synthetic and real-world VFL datasets, further clarification is needed on how to interpret the findings from Figure 7 to better understand the implications of the results.\n\n2. Section 2.3 discusses various party correlation metrics, with the paper employing Spearman rank correlation in Algorithm 1. It would be beneficial to explore whether Spearman rank correlation can be replaced by the other mentioned correlations and, if possible, provide a comparison of their performance. This comparison would enhance the understanding of the selected correlation metric's effectiveness.\n\n3. Figure 5 indicates that three out of four VFL baselines are minimally impacted by alpha and beta. This raises questions about the rationale for partitioning the dataset based on feature importance and party correlation. It prompts further consideration of whether the proposed VFL dataset synthesis approach may overemphasize expanding the scope to an extent that may not reflect practical scenarios. Clarifying the reasoning behind these observations is crucial.\n\n4. The paper discusses the communication efficiency of VFL algorithms, which seems to be minimally impacted by data distribution. Given this, it might be more appropriate to allocate more space to extending the experiments in Section 4.5. Moving additional explanations and results, which may be included in the appendix, to the main paper, would provide a more comprehensive understanding of the proposed method and its real-world applicability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZGMmr199AC", "reviewer": "Reviewer_BWae", "summary": "This paper tackles the problem of inadequate feature diversity in VFL benchmarking. Motivated from a probabilistic perspective, this paper factors the feature diversity into party importance and party correlation.\n\nBased on this factorization, this paper proposes corresponding metrics for real VFL datasets (*Shapley-Based metric for party importance, SVD-based Icor metric for party correlation*) and a tailored splitting method for synthesized VFL datasets (*Dirichlet-based importance splitting and permutation-based correlation splitting*). By further mapping the two evaluation metrics into splitting parameters $\\alpha$ and $\\beta$, the paper provides a unified solution to enable cross-scope comparison between real and synthesized datasets.\n\nEmpirical experiments are conducted on 2 real VFL datasets and 7 centralized datasets. The results validate the effectiveness of the designed metrics and split method, and provide additional observations to better understand 4 representative VFL methods.", "questions": "I am willing to increase my score if the authors adequately address the questions or clarify my misunderstandings.\n\n**Major**\n\nSee in the Weakness part.\n\n**Minor**\n- **Unclarities**:\n  - It is confusing when reading \"VertiBench, when parameterized with the same $\\beta$, exactly reconstruct the real feature split of Vehicle dataset.\" in Section 3.2 and comparing Figure 4 (a) vs (b), since you do not introduce how we can get $\\alpha$ and $\\beta$ for real VFL datasets. There should be a kind hint to remind readers to refer to Section 3.3 for details.\n  - There are 2 real VFL datasets and 7 centralized datasets used in the experiment. However, the introduction of the two real VFL datasets is missing, it should be clearly described in section 4.2. This confused me a lot at the beginning.\n  - The citation for Shapley-CMI is likely incorrect in line 5 of Section 2.2; the given citation is about the \"Shapley Taylor Interaction Index\". However, the correct one is given in Line 3 after Theorem 1.\n- **Citation Absence or Overclaim**: As is commonly expected of benchmark papers, they are usually anticipated to provide a comprehensive literature review of related methods in this domain and clarify their connection with existing benchmark papers. However, the discussion of a recent benchmark paper, [1]FedAds, is omitted, and some splitNN-based methods (e.g., [2]FedHSSL, [3]JPL, and others listed in Table 3 of work[1]) as well as other types of tree-based methods (e.g., [4]Federated Forest) are omitted in discussion. Although it is not necessary to evaluate all these methods, a comprehensive literature review is usually beneficial for a benchmark paper to be solid. From this perspective, it would be more appropriate to restrict it as a benchmark **only for major VFL method types**, since the paper excludes some sub-type methods.\n  - [1] FedAds: A Benchmark for Privacy-Preserving CVR Estimation with Vertical Federated Learning\n  - [2] A Hybrid Self-Supervised Learning Framework for Vertical Federated Learning\n  - [3] Vertical Semi-Federated Learning for Efficient Online Advertising\n  - [4] Federated Forest", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- **Tackling an important problem**: Feature diversity is a fundamental property that highly affects the performance of VFL. Evaluating VFL algorithms across different levels of feature distribution contributes to understanding the distribution feasibility of a proposed method. In HFL, using the Dirichlet distribution to simulate various Non-IID levels has already become a consensus and common tool in HFL algorithm evaluation. It's happy to see this paper provides an effective tool for VFL to achieve a similar purpose. This would be helpful in forming a more fair and comprehensive standard in benchmarking VFL methods.\n- **Effectiveness of the Method is Mostly Supported**: The basic purposes of Party Importance and Party Correlation are well-achieved as shown in Figures 2, 3, and 4. The splitting method shows ideal results as expected. Besides, the importance splitting method has a good theoretical guarantee. It shows promise to become a commonly utilized tool in future VFL research, thereby making a significant impact.\n- **Sufficient Benchmark of Major VFL Method Type**: This is the first work comparing three different classes of VFL methods (AL-based, NN-based, and Tree-based), providing insights from a broader view. The experiments are conducted on sufficient datasets with various settings. The appendix provides very detailed additional results, and the splitting method used in the CIFAR10 and MNIST datasets is much more reasonable than that in previous works (left-right image half split).", "weaknesses": "- **Unclear Independent Assumption**: From the beginning (the end of Section 2.1), the reason for omitting the correlation between Party Importance and Party Correlation is not well-described. A rough or intuitive relationship would be better to explain such a choice (e.g., the independent assumption is more concise and sufficient for evaluating most cases, or explain why it is so hard to capture their correlation). Since there are two metrics representing feature diversity, it can be confusing to determine which one to use, which one is better suited in specific cases, and whether it is possible and reasonable to evaluate a specific method in a 2D grid of $\\alpha$ and $\\beta$. \n- **Seemingly Case-Restricted Significance**: As shown in Figure 5, only C-VFL reflects a significant response to the variation of feature diversity, and all other methods perform consistently robust on all datasets. It seems like there are some specific reasons tailored to C-VFL that cause this observation, and further, **it makes me doubt whether it is meaningful to conduct such feature diversity evaluation for non-compression-based VFL methods**. This may largely reduce the potential impact of this work. Besides, as shown in the right part of Figure 7, the performance rank of GAL observed on synthetic datasets is different from real ones and shows significantly low performance. The consistency level is not as strong as claimed by words.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GWbNEmJewv", "reviewer": "Reviewer_6Ygd", "summary": "This research paper proposes improvements to vertical federated learning (VFL) benchmarks by addressing the shortcomings of existing benchmarks. The paper introduces feature importance and correlation as two crucial factors that could potentially influence the performance of VFL algorithms. It also proposes evaluation metrics and dataset splitting methods to improve VFL performance. The study evaluates cutting-edge VFL algorithms, including ensemble-based, Split-NN-based, and Split-GBDT-based algorithms, and provides insights gained from the evaluation. Overall, the paper provides valuable insights for future research in the field of VFL.", "questions": "Is it possible to use one metric for reflecting both importance and correlation? Or say how should a user trade-off between these two factors in comparing different VFL algorithms?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1.\tThis paper is well-written, so I can pick the core ideas up effortlessly.\n2.\tThe focused problem has broad interests, as VFL has attracted much attention from both academia and industry, but there are rarely practical or realistic datasets. This paper improves existing VFL benchmarks by addressing the shortcomings of inadequate evaluation of existing VFL methods. \n3.\tThis paper introduces feature importance and correlation as crucial factors that could influence VFL algorithm performance. Meanwhile, it proposes evaluation metrics and dataset splitting methods to improve VFL performance.\n4.\tAdditionally, the evaluation of cutting-edge VFL algorithms provides valuable insights for future research in the field.", "weaknesses": "1.\tThe notation needs to be clarified. In Equation 1, the summation is over index $i$, but the added terms do not contain $i$. This forbids me to understand the meaning of this decomposition of the left-hand side’s log probability.\n2.\tFrom another point of view, as synthetic datasets can be split in different ways, focusing on either importance or correlations, real-world datasets are still in more demanded. Thus, offering a more realistic VFL dataset in a related benchmark would be better.\n3.\tThe proposed correlation-based split method may face scalability challenges when applied to a large number of parties, which could limit its practicality in certain scenarios. Could you provide more discussion about this?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)