Vertical Federated Learning with Missing Features During Training and Inference
Authors: Pedro Valdeira
Shiqiang Wang
Yuejie Chi
Venue: ICLR 2025 Poster (2025)
Keywords: Vertical Federated Learning
missing features
Primary area: optimization
TL;DR: We propose a flexible vertical federated learning method for efficient training and inference in the presence of missing features.

Abstract: Vertical federated learning trains models from feature-partitioned datasets across multiple clients, who collaborate without sharing their local data. Standard approaches assume that all feature partitions are available during both training and inference. Yet, in practice, this assumption rarely holds, as for many samples only a subset of the clients observe their partition. However, not utilizing incomplete samples during training harms generalization, and not supporting them during inference limits the utility of the model. Moreover, if any client leaves the federation after training, its partition becomes unavailable, rendering the learned model unusable. Missing feature blocks are therefore a key challenge limiting the applicability of vertical federated learning in real-world scenarios. To address this, we propose LASER-VFL, a vertical federated learning method for efficient training and inference of split neural network-based models that is capable of handling arbitrary sets of partitions. Our approach is simple yet effective, relying on the sharing of model parameters and on task-sampling to train a family of predictors. We show that LASER-VFL achieves a $\mathcal{O}({1}/{\sqrt{T}})$ convergence rate for nonconvex objectives and, under the Polyak-Łojasiewicz inequality, it achieves linear convergence to a neighborhood of the optimum. Numerical experiments show improved performance of LASER-VFL over the baselines. Remarkably, this is the case even in the absence of missing features. For example, for CIFAR-100, we see an improvement in accuracy of $19.3$\% when each of four feature blocks is observed with a probability of 0.5 and of $9.5$\% when all features are observed. The code for this work is available at https://github.com/Valdeira/LASER-VFL.

Published as a conference paper at ICLR 2025
VERTICAL FEDERATED LEARNING WITH MISSING FEATURES DURING TRAINING AND INFERENCE
Pedro Valdeira
Shiqiang Wang IBM Research wangshiq@us. ibm com
Yuejie Chi
Carnegic Mellon University & Instituto Superior Técnico pva 1 dei ra emm.edu
Carnegie Mellon University yue jiechieehee wiechhicmmu... moniemmu.... eduu
ABSTRACT
Vertical federated learning trains models from feature-partitionet datasets across multiple clients, who collaborate without sharing their local data. Standard ap- proaches assume that all feature partitions are available during both training and inference. Yet, in practice, this assumption rarely holds, as for many samples only subset of the clients observe their partition. However, not utilizing incomplete samples during training harms generalization, and not supporting them during inference limits the utility of the model. Moreover, if any client leaves the federa- tion after training, its partition becomes unavailable, rendering the learned model unusable. Missing feature blocks are therefore key challenge limiting the appli- cability of vertical federated learning in real-world scenarios. To address this, we propose LASER-VEL, a vertical federated learning method for efficient training and inference of split neural network-based models that is capable of handling arbitrary sets of partitions. Our approach simple yet effective, relying on the sharing of model parameters and on task-sampling to train a family of predictors. We show that LASER-VER VFL achieves O(IVT) convergence rate for nonconvex objectives and, under the gence to a neighborhood of the optimum. Numerical experiments show improved performance of LASER-VFL over the baselines. Remarkably, this is the case even in the absence of missing features. For example, for CIFAR-100, we see an im- provement in accuracy of 19.3% when each of four feature blocks is observed with a probability of 0.5 and of 9.5% when all features are observed. The code for this work is available at /ttp::://gttttte htt.:ttww.......e.eeeeeeaeeeeeee www.oo//aaaieiaaaa LASERE VFL
inequality, it achieves linear conver-
INTRODUCTION
In federated learning (FL). set of clients collaborates to jointly train a model using their local data without sharing (Kairouz et al., 2021). In horizontal FL, data distributed by samples, meaning each client holds a different set of samples but shares the same feature space. In contrast, vertical FL (VFL.) involves data distributed by features, where each client holds different parts of the feature space for overlapping sets of samples. Whether an application is horizontal or vertical FL is dictated by how the data arises, as it is not possible to redistribute the data. This work focuses on VFL.
In VFL (Liu et al. 2024), the global dataset D = f......,,.. x.VV }. where N is the number of samples, partitionee across clients := {I...., K), J. Each client & E K. typically holds a local dataset Dr = {z) (....................... x J. where x is the block of features of sample observed by client k. We have that x=' = (zf.... x.k).. Unlike horizontal FL, in VFL, different clients collect local datasets with distinct types of information (features). Such setups e.g.. online retail company and social media platform holding different features on shared users eteecally involve entities from different sectors, reducing competition and increasing the incentive to collaborate. To train VFL models without sharing the local datasets, split neural networks (Ceballos et al. 2020 are often considered.
In split neural networks, each client has representation model fE. parameterized by Of.- The representations extracted by the clients are then used as input to a fusion model g, parameterized by Oy. This fusion model can be one of the clients or at a server. Thus, to learn the parameters 0x = (0,...... fix: 0g) of the resulting predictor h, we can solve the following problem:
(I) with l denoting loss function and y" the label of sample n, which we assume to be held by the same entity (client or server) the fusion model. We illustrate this family of models in Figure
minek N/ I {(h(="; ((h(=";0c)..)") where b(x";0k) = 9 (ff )z)))

---

Published as a conference paper at ICLR 2025
fusion model
blocks of Reatures
Paiu
Method
0.0
0.5
MIMIC-IV
(Fi-scorex 100)
repros, models
Local
76.9
73.0
Standard VEL
91.6
51.8
Dulaset Availability:
Standard VFL
Local model
Our method
LASER-VEL tours)
92.0
82.0
Usage:
Usage:
Usage:
observed
train & test
train & test
train & lest
client
missing
wasted
wasted
train
(a) Split neural network.
(b) Datasct availability and usage.
(c) Results preview.
we show the availability of a dataset with K = 3 blocks of features, each with 0.5 probability of being observed, and its usage and waste by three key methods: standard VFL, a local approach, and our method. Standard VFL trains single predictor, while the local approach and our method train different predictors different clients (we show the data usage for client 1). In Figure Ic we present preview of our results: mortality prediction task using the MIMIC-IV dataset, when the probability of each block of features missing, Pmiss. is in {0.0, 0.5}, for both data and test data.
Figure 1: In Figure la we illustrate a split neural network. In Figure Ib]
Generalization and availability in standard VFL. We see in ①1 that, even for a single sample n, predictor h depends on all the blocks of features, {ær : VkJ. Thus, for both training and inference, (1) requires the observations of all the clients to be available T Building on our example of an online retailer and social media company sharing users, each company is also likely to have unique users. This applies to both training and test data. Further, if any client drops from the federation during inference, its block will permanently stop being observed. In both the case of nonshared users and of clients leaving the federation, standard VFL predictors become unusable. Figure [bb illustrates this phenomenon: for both training and inference, if each of K clients observes its block of features with an (independent) probability of 0.5, only 0.53 = 12.5% of the original data is usable. Thus, restricting training to fully-observed samples hinders generalization, while restricting inference to such samples limits the availability and utility of the model.
Dealing with missing features. To handle missing features in training data, some approaches pand the dataset, filling the missing features before collaborative training (Kang et al., (0226), while others use their partition of partiall--observer samples for local representation learning but exclude them from collaboration (He et al. 2024). These methods can outperform standard VFL when par- tially observed samples are present, but they add new training stages and auxiliary modules, making them more complex. Moreover, they train a joint predictor that requires the collaboration of all clients during inference. On the other hand, to improve robustness against missing blocks in- ference, each client can train a local predictor using only its own features, avoiding collaboration altogether. This also addresses missing features in the training data. Alternatively, collaborative methods can employ techniques such as knowledge distillation (Huang et al. 2023) and data aug- mentation (Gao et al., 2024) to train local predictors, also adding complexity with multiple stages. However, while robust to missing features, these local predictors cannot utilize additional feature blocks if available at test time, resulting in wasted data and reduced predictive power. We illustrate this in Figure 1b.
Therefore, there is a gap in the literature when comes to leveraging all the available data without either dropping incomplete samples or ignoring existing features. This raises the following question:
Can design an efficient VFL method that is robust to missing features at both training and inference with provable convergence guarantees, without wasting data?
In this work, we answer this question in the affirmative. We propose LASER-VEL (Leveraging All Samples Efficiently for Real-world VFL), a novel method that enables both training and inference using any and all available blocks of features. To the best of our knowledge, this is the first method to achieve this. Our approach avoids the multi-stage pipelines that are common in this area, providing
'This contrasts with horizontal Fl., where the local data of one client suffices to estimate mini-batch gradi- ents and to perform inference.
2

---

Published as a conference paper at ICLR 2025
simple yet effective solution that leverages the sharing of model parameters alongside task- sampling mechanism. By fully utilizing all data (see Figure lb). our method leads to significant performance improvements, demonstrated in Figure Icl Remarkably, standard VFL when all samples are fully observed. We attribute this to dropout-like regularization effect introduced by task sampling.
approach even surpasses
Our contributions. The main contributions of this work are follows.
We propose LASER MASER-VEL, VVF,, a simple, hyperparamette--free, and efficient VFL method that is flex- ible to varying sets of feature blocks during training and inference. To the best of our knowledge, this is the first method to achieve such flexibility without wasting either training or test data.
We show that LASER-VEL converges at ((/VT) rate for nonconvex objectives. Furthermore, (PI.) inequality, achieves linear convergence a neighhorhood
under the Polyak-F.ojasiewe around the optimum.
Numerical experiments show that LASER VFL consistently outperforms baselines across multi- ple datasets and varying data availability patterns. demonstrates superior robustness to missing features and, notably, when all features are available, it still outperforms even standard VFL.
Related work. VFL shares challenges with horizontal FL., such as communication efficiency (Liu et al. 2022, Valdeira et al. (025) and privacy preservation (Yu et al., (2024), but also faces unique obstacles, such as missing feature blocks. During training, these unavailable partitions render the observed blocks of other clients unusable, and at test time, they can prevent inference altogether. Therefore, most VFL literature assumes that all features are available for both training and infer- ence-and an often unrealistic assumption that has hindered broader adoption of VFL (Liu al. 2024).
To address the problem of missing features during VFL training. some works use nonoverlapping, or nonaligned, samples (that is, samples with missing feature blocks) to improve generalization. In particular, Feng (2022) and He et al. 2024) apply self-supervised learning to leverage nonaligned samples locally for better representation learning, while using overlapping samples for collaborative training of a joint predictor. Alternatively, Kang et al.] (2022b), Yang et al. (2022). and (un et al. (2023) employ remi-supervised learning to take advantage of nonaligned samples. Although these methods enable VFL to utilize data that conventional approaches would discard, they still train a joint predictor and thus require all features to be available for inference, which remains limitation.
A recent line of research leverages information from the entire federation to train local predictors. This can achieved via transfer learning, as in the works by Feng Yu (2020); Kang et al. (2022)) for overlapping training samples and in the works by/Liu et al. (2020); et al. (2022) for handling missing features. Knowledge distillation is another approach, as used by Ren et al. ) Li et al. (2023b) for overlapping samples and by Li et al. (2023a); Huang al. 2023) which leverage nonaligned samples when training local predictors (only the latter considers scenarios with more than two clients). Xiao et al. (2024) recently proposed using distributed generative adversarial network for collaborative training on nonoverlapping data and synthetic data generation. These methods yield local predictors that outperform naive local approaches, but fail to utilize valuable information when other feature blocks are available during inference.
A few recent works enable varying number of clients to collaborate during inference. Sun et al. (2024) employ party-wise dropout during training to mitigate performance drops from missing fea- ture blocks at inference; Gao et al. (202)) introduce multi-stage approach with complementary knowledge distillation, enabling inference with different subsets of clients for binary classification tasks; and Ganguli et al. (2024) extend Sun et al. (2024) to deal with communication failures during inference in cross-device VFL. However, all of these methods consider the case of fully-observed training data and they all lack convergence guarantees.
In contrast to prior work, LASER-VFL can handle any subset of feature blocks being present during both training and inference without wasting data, while requiring only minor modifications to the standard VFL approach. It differs from conventional VFL solely in its use of parameter-sharing and task-sampling mechanisms, without the need for additional stages.
2 DEFINITIONS AND PRELIMINARIES
Our global dataset D is drawn from an input space X' which is partitioned into K feature spaces X Xi .... x Xx such that Dr. C Xr, where, recall, D3 is the local dataset of client k. Ideally,
3

---

Published as a conference paper at ICLR 2025
would train a general, unconstrained predictor h: X + V/ where 3' is the label space, by solving ming & En= (0)(==")),)0 However, VFL brings the additional constraint that this must be achieved without sharing the local datasets. That is, for all k, the local dataset Dx must remain at client k. As mentioned in Section | standard VFL methods approximate h by h: X + V as defined in ( allowing for collaborative training without sharing the local datasets, but they cannot handle missing features during training nor inference.
Another way to train predictors without sharing local data is learn local predictors. In particular, each client & E K can learn the parameters 0% of predictor h& A2 + y to approximate h:
h 0x) gb (fr (x²;; Oz,); mm.) where O2 :0:=000 (0 On.))
The representation models fa Xx + Eg, where & is the representation space of client k are as in ( yet the fusion models 9k & + R differ from & x ..... x.. &K + R in ( and hy differs from h. This approach is useful in that hy allows client k perform inference (and be trained) independently of other clients, but it does not make use of the features observed by clients j * k.
More generally, to achieve robustness to missing blocks in the test data while avoiding wasting other features, we wish to be able to perform inference based on any possible subset of blocks, (P())) {Ø), where (KC) denotes the power set of K. Standard VFL allows us to obtain a predictor for the blocks K. and the local approach provides us with predictors for the singletons {{i}: K. in the power set. However, none of the other subsets of K. is covered by either approach.
A naive way to achieve this would be to train predictor for each set I E P(K)) (Ø) in decoupled manner. That is, we could train each of the following predictors {hi: ΠREI Ys + )} independently using the collaborative training approach of standard VFL.:
(1z(2)7 0x) == (I((((( ( & E 0z) I (KC) ((3)};
(3)
where 0z = (T) : k E 0 and 2 (af: : E I). The fusion models {gz) can either be at one of the clients at the server. The set of predictors in (3) includes the local predictors } in (2) and the standard VFL predictor h in ( but also all the other nonempty sets in the power set ge [K) This approach addresses the issue of limited flexibility and robustness in prior methods, which struggle with varying numbers of available or participating clients during inference. However, by requiring the independent training of 2 distinct predictors, it introduces new challenge: the number of models would grow exponentially with the number of clients, K. Consequently, the associated memory, computation, and communication costs would also increase exponentially.
Further, if the fusion models in (3) are all held by a single entity, this setup introduces a dependency of all clients on that entity. To enhance robustness against clients dropping from the federation, we would like each client to be able to ensure inference whenever it has access to its corresponding block of the sample. That is. we want each client & E K. train a predictor for every nonempty subset of K. that includes k. We define this family of subsets as (k) = (I E (K)) k E Z). This allows client i to make predictions even if the information from blocks observed by other clients is not available and thus cannot be leveraged to improve performance.
In the next section, we present our method, LASER- AALE,-VFL, which enables the efficient training of a family of predictors that can handle scenarios where features from an arbitrary set of clients are missing, We have included a table of notation and diagram illustrating our method in Appendix
3 OUR METHOD
Key idea. The main challenge, when striving for each client to be able to perform inference from any subset of blocks that includes its own, is to cireumvent the exponential complexity arising from the combinatorial explosion of possible combinations of blocks. To address this, in LASER-VEL. we share model parameters across the predictors leveraging different subsets of blocks and train them so that the representation models allow for good performance across the different combinations of blocks. Further, we train the fusion model each client to handle any combination of representations that includes its own. To train the predictors on an exponential number of combinations of missing blocks while avoiding an exponential computational complexity, we employ a sampling mechanism during training, which allows us to essentially estimate an exponential combination of objectives with a subexponential complexity.
The notation in differs slightly from and 2 which use a simpler, more specific formulation.
4

---

Published as a conference paper at ICLR 2025
3.1 A TRACTABLE FAMILY OF PREDICTORS SHARING MODEL PARAMETERS
In our approach, each client & E K. resorts to a set of predictors { I (K)) Each predictor
huz: liiz X; + y has the same target, but different domain. Therefore, we say that each predictor performs a distinct task. We define the predictors of client k as follows:
hKz (az; (I.D)) = 9h
1 I
f((o;;; 0z,:: IE TEBKKK
(4)
IEI
where (I. (00 i I) 0g.) and [I] denotes the cardinality of set I.
Sharing model parameters. Note that, although we can see as (P((K)] 2 different predictors, the predictors are made up of different combinations of only K representation models and fusion models. That is, we only require parameters 0 == (O...... 0z ........ 0gx) ). In particular, in contrast to (3), where the predictors used different representation models for each task, in H we share the representation models across predictors. Similarly to the representation models, have single fusion model per client.
is also important to note that each fusion model gl ((f. Ox:): E I); 0g.) takes the specific form of gk [2]]
1 Eiez f 0;:): By employing a nonparameterized aggregation mecha- nism on the extracted representations whose output does depend on the number of aggregated representations, the same fusion model can handle different sets of representations of different sizes. In particular, we opt for an average (rather than, say, a sum) because neural networks perform better when their inputs have similar distributions (loffe & Szegedy) 2015). Note that the representation model can be adjusted so that using averaging instead of concatenation as the aggregation mecha- nism does not reduce the flexibility of the overall model, but simply shifts it from the fusion model to the representation models, as explained in Appendix A.2
With this approach, we have avoided an exponential memory complexity by sharing the weights across an exponential number of predictors such that we have representation models and K fusion models. In the next subsection, we will go over the efficient training of this family of predictors.
3.2 EFFICIENT TRAINING VIA TASK-SAMPLING
The family of predictors introduced in Section 3.1 can efficiently perform inference for an expo- nential number of tasks. We will now discuss how to train this family of predictors while avoiding exponential computation and communication complexity during the training process. The key to this approach lies in carefully sampling tasks each gradient step of model training.
Our optimization problem. As noted in Section D we want to address missing features not only during inference, but also during training. Let KC; be the set of observed feature blocks of sample n, assume that K., == (KT- Vn) follows random distribution. Recalling (4), denote the loss é of sample n, with label y", for the predictor of client using blocks I E P (1CB of x. as follows:
Aaz (0) (her (z): (R.TT) .8").
For sample n and client k; the (weighted) prediction loss across all observed subsets of blocks that include is denoted by:
(0) = ΣIE [
(0).
where superscript o indicates that this loss concerns the observed dataset. The normalization by [Z] avoids assigning a larger weight to tasks concerning a larger number of blocks, since the set I is used for prediction at [Z] different clients, whose losses are combined in the loss of sample n:
2 (0)
(0).
(5)
Now, to train the predictors in 4 we consider the following optimization problem: N
min {C()) = E (0)}
where
()) = N n=1
Σ ()),
(6)
where the expectation is over the set of observed feature blocks, K...
5

---

Published as a conference paper at ICLR 2025
Algorithm 1: LASER-VVL. Training Input: initial point 0º training data D.
for f O,..., do
Clients IC sample the same mini-batch B with K!! = K. An E B via a shared seed.
for k E k(d parallel do
Broadcast f and receive {f : : K..", k).
Sample task-defining set of blocks Sky ~ U 11-0((PP0 (k(! for =1, 1....,,
Compute s; (f'), as in [). and backpropagate over fusion model gi-
Send the derivative of sg (0") with respect to each & E x-(() and receive theirs.
Sum the received gradients and backpropagate over fk-
Update the weights 0 0 7/V L (0"), where LB (0*) is in
Note that, if different blocks features have different probabilities of being observed, we can also normalize the loss with respect to the different probabilities, which can be computed during the entity alignment stage VFL (see below), before taking the expectation in (5). omit this here for simplicity. We assume that the data is missing completely at random (Zhou et al. 2024).
Looking (5), we see that, while we are interested in tackling K' 2K different tasks, which falls in the realm of multi-objective optimization and multi-task learning (Caruana 1997), we opt for combining them into a single objective, rather than employing a specialized multi-task optimizer. This corresponds weighted form of unitary scalarization (Kurin al. 2022).
Our optimization method. To solve (6), we employ to gradient- brase method, summarized in Algorithm which we now describe, At iteration f, with the current iterate 0f, we select mini-batch indices 8 C [N]. where [N] == (l....,N, ],, such that the set of observed feature blocks for each sample n Bt, denoted by 1Cg, is identical. We denote this common set by x=!!) that is, K!! KC
for all n E B¹, and define (0) = B EnEB (0).. Then, each client k E xC(!) broadcasts its representation
fi = Ss (xf 0%,) where o = {xf Vn E B'}
to the other clients in x this stage, each client & E KC! holds the representations correspond- ing to the set of observed feature blocks of the current mini-batch, {f i i xc
To train the model on all possible combinations of feature blocks without incurring exponential computational complexity at each iteration, each client E xC(!) samples K == | tasks from the 2 tasks in P& (K((")) the subset of the power set of K:5 whose elements contain k: These sampled tasks estimate Lg. (0') without computing predictions for an exponential number of tasks (note that K3 S K). We use this estimate to update the model parameters.
More precisely, the task-defining feature blocks sampled by client k at step are given by St == where set Sh. contains feature blocks. Each set St is sampled from uniform
distribution over a subset of (x((t))) that contains sets of size i. That is:
S SE
where ((C!!) (I (KC))) -
This task-sampling mechanism allows us to estimate the loss with linear complexity, instead of the exponential cost of exact computation. More precisely,
the following loss estimate:
Eg (0') = ExeKço sg (0))
where St = (St (Si..... ( and each Eg (0'), computed at client k. is given by:
Kt af
Laksy, (0') where af = 1 1 i-1
K -1
2 Lg (0')
(8)
B
WEB

---

Published as a conference paper at ICLR 2025
Algorithm 2: LASER-VFL. Inference
Input: test sample æ', trained parameters 0"
The blocks of features K% of sample x' are observed.
for k E Ky in parallel do
Broadcast Sa (2 (7.) and receive fi(a''; 0J))) for all E X% \ {k}.
Client & predicts the label using hyx; (zRt ((.K)))
The weighting af counteracts the probability of each task being sampled, allowing to obtain an unbiased estimate of the observed loss. Now, we use this to minimize (5) by doing the following update:
0++ 0- LB
After completing the training described in Algorithm [ we can perform inference according to Algorithm 2 In this inference algorithm, each client k E 1% that observes partition of the test sample æ' shares its corresponding representation of x'. Then, each such client & uses h(,.g)) to predict the label.
Block availability-awar entity alignment. Entity alignment is a process the precedes training in VFL where the samples of the different local datasets matched so that their features correctly aligned, thus allowing for collaborative model training (Liu et al. 2024). The identification of the available blocks for each sample can be performed during this prelude VFL training, allowing for a batch selection procedure that ensures that each batch contains samples with the same observed blocks, which we use in our method.
Extensions to our method. It is important to highlight that our method can also be easily em- ployed in situations where different clients hold different labels, as it naturally fits multi-task learn- ing problems. Further, it can easily be applied to setups where any subset of K. holds the labels, sufficing to drop fusion models from the clients which do not hold the labels or have them tackle unsupervised self- sulf-super learning tasks instead.
4 CONVERGENCE GUARANTEES
In this section, we present convergence guarantees for our method. Let us start by stating the as- sumptions used in results.
Assumption (L-smoothness and finite infimum). Function L θ + IR is differentiable and there exists a positive constant such that:
0,, O2 O: (11C (L)(02) < L| 02l1 The inequality above holds if C is L'smmooth. We assume and define L = infe [( ~00.
(A1)
Assumption (Unbiasedness), The mini-batch gradient estimate is unbiased. That is, we have that:
v O: EE [VCg( K.] (L),
(A2)
where the expectation with respect to the mini-batch B. Assumption 3 (Bounded variance). There exist positive constants and 0, such that, for all O ④ and le E K,, we have that:
E
v C3 (0) - (0)II
Ko| Sof and E V v VC (0) Ko. B
(A3)
where CiSx. (
EneB Cix (0) and the (conditional) expectations are with respect to mini- batch and with respect the tasks sampled each client St. respectively.
We use the following set in our results:
= [B So
B',''
....,,,, 1 st- 1
S
Let us start by presenting lemma which resort in our proof of Theorem

---

Published as a conference paper at ICLR 2025
Lemma (Unbiased update vector). If the mini-bated gradient estimate is unbiased A2 then, for all t 0:
E v s=0")
=k,,
(9)
where the expectation is with respect to mini-batch B and the sampled of tasks S Theorem (Main result). Let {0*} } be sequence generated by Algorithm 7
if C is L-smooth and
has finite infunum } the mini-batch estimate of the gradient unbiased
A2 and our update
vector has bounded variance A33) we then have that, for E (0, 1/L]:
+
E S T/T
2A
n (o + + K-KKo -
(10)
t=0
where A = -(00) L- and the expectation is with respect to (B)), (S'), and K...
we take the stepsize to 1) =
(o
we get from 10p that K k.,))
LT
T
1
8AL (o K k.))))
VL(0')|2
thus achieving O(1VT) convergence rate. Note the effect of the variance induced by the task- sampling mechanism, which grows with K.
We further establish linear convergence to neighborhood around the optimum under the Polyak-t.ojasiewicz inequality (Polyak 1963).
Assumption (PL inequality). We assume that there exists a positive constant js such that
vO e O: ||V[(0)) 2 2n(((()) L-))
(A4)
We use the expected suboptimality 8 == EC( L* where the expectation is with respect to {B'), {S*), and K.,, as our Lyapunov function in the following result.
Theorem 2 (Linear convergence). Let {0'} be a sequence generated by Algorithm 7 if L is L- smooth and has finite infimum (AT), the mini-batch estimate of the gradient is unbiased (A2). our update vector has a bounded variance (A3), and the PL inequality (A4) holds for L, we then have Shat,for 1/L]:
nL
8T 1 (1 - pm) +
(0r + K-K-o o
24
follows from S that jar) E (0, 1). Therefore, we have linear convergence to O (oī K o²) neighborhood the global optimum.
We defer the proofs of Lemma Theorem and Theorem to Appendix C
EXPERIMENTS
In this section, we describe our numerical experiments and analyze their results, comparing the empirical performance of LASER-VFI, againsss baseline approaches. We begin with brief overview of the baseline methods and tasks considered, followed by a the presentation and discussion of the experimental results.
We compare our method to the following baselines:
Local: as in (2) the local approach leverages its block for training and inference whenever is observed, but ignores the remaining blocks.
Standard VFL (Liu et al. 2022): as in [ the standard VFL model can only be trained on and used for inference for fully-observed samples. When the model is unavailable for prediction, a random prediction is made.
VF WFedTrans (Huang et al. 2023): VPedTrans introduces a multi-stage VFL training pipeline collaboratively train local predictors. (We give details about VFedTrans below.)
8

---

Published as a conference paper at ICLR 2025
Table Test metrics for K clients across different datasets and varying probabilities of missing blocks during training and inference, averaged over five seeds (# standard deviation).
Training Pain
0.5
Inference
0.0
0.1
0.5
0.6
0.1
0.5
0.0
0.
0.5
HAPT faceuracy, %)
Local
82.1
6.
81.5
50.7
80.5
Standard VET.
01.6++03 0.9
67.
16.0 +.0++4. 4.1
90.0 1,4
66.0 28
15.8
+ 3.9
84.6+22
622
15.3
3.5
VredTrans
0.
823=001
0.3
02
8.8
82.
822 2
82
0.8
Insemble
0.9
9..3
0.
0.5
88.1
Combi nstorial
92.5++0.2
02.4400.
88.5
1.7
90.0
90.0
1.4
87.3
1.6
8422
84.7
80.0
P PlugVP1
91.1 1.4
88.5 + 1.6
75.1 =
0.9
88.6
78.3 4.8
87.7 1.6
87.2 1.5
82.8
1.6
LASER-VEE VFL (ours)
Credit
soorex 100)
Local
3.1
3.1
37.5
3.2
S7.6 3.0
2.9
37,5
1155:85 3.2
3.5
36.2 = 3.6
36.0 3.6
Standard
45.7412.
16.8 2.5
31.
4.
0.7
31.
32.4 2:44
32.5
L3
30.3 0.5
VFedTrana
31.7
37.6
37.2
39.2
35.7 0.3
35.6
35.6
Ensemble
42,1=100
42.1 1.0
40.1 0.8
11413
00.
39.2 1.1
424
41.8
LT
#0.1111
Combinatorial
2.9
42.7 2.5
41.7 155
44.4 L.0
2.2
41.7
32.8 3.4
36.3
0.6
37.6 2.1
PlugVEL
32.7
30.8
31.9
38.6
37.8
LASER-VEL (ours)
46.5 + 2.8
45.0 + 2.5
43.7
1.2
43.1
4/2
41.9 +: 4.0
41.3
1.9
41.5 + 4.2
40.9
4.0
41.4
MIMIC-IV (F Scoreccee 100)
0.5
74.9
76.0
75.0
76.0+0033
T3.0 0.5
7300
73.0
Standard VPL
9.2
77.0 13
52.5 2.
90.9 0.3
76.8 13
52.3
81. 0.1
70.3 0.8
518+18
Ensemble
0.3
83.9 0.900:: 0.3
71.9 L6
84
0..
78.1
0.3
818 0.5
Combi nstorial
0.3
87.2
0.4
86.7 1.4
0.4
80.5
806
78.900
PlugVPL
91.2 0.5
89.0
79.6 0262288
90.6
0:2
8
7.8
86.5 0.7
85.3
0.9
779428
LASER-VEL (ours)
92.0 L: 0.2
91.2 1 0.3
1.0 85.5
91.1 1 0.5
90.2 0.3
84.7 22 0.9
87.7 1 0.2
$6.9
0.2
82.0 1 L.
CIFAR- caceuracy.
(6)
Local
76.1
76.2 +
75.6
75.7
71.1
71.3+
Standard VEL
89.7 0.2
60.5 12.1
11.6 3.5
87.0
58.8 11.4
11.5
3.4
54.9 10.0
38.6 9.1
109
Ensemble
8.2 0.2
0.77
0.9
86.1 0.3
85.0 0.5
77.8
8.24
810 0.
73.8 0.6
Combinatorial
88.7
82.4
54.9
58.6
PlugVF1
89.4 0.2
88.1 0.8
81.8
1.7
88.1
86.9 0.6
81.2
1.2 83,3
76.5+21
76.6 1.5
124+10
LASER-VP1 (ours)
91.5 0.1
0.4
83,8
1.5
91.2 0.2
90.2 0.1
87.4
0.6
L6
CIFAR-100 (ncsuracy,
Local
30.3 0.1
0..3+001
0.1
4:2
19.2
49.1 0.3
0.7
413=06 0.6
41.300
Standard VEL
6.47
4.5
2.4
57.2
36.6
233
15.81..
10.5 466
144 LO
Ensemble
6.3
6..2+-1.2
5.3
60.8
1 1.0
51.0
52.6 0.8
30.4-077
+..5008
Combinetorial
6..4
640000
58.6
572422
67 18
56.3 0.7
164468
19.5
316
FlugVEL
66.0 0.3
61.1112
2.1
61.2111
0.7
53.1
1.9
11.6 2.3
43.9 L77
16
LASER-VEL, (ours)
0.1
71.0 0.7
61.3
70.4 0.3
0.8
59.8
61.4
39,9
$1.9
Ensemble: we train local predictors in Local. During inference, the clients share their predictions, and joint prediction is selected by majority vote, with ties broken at random.
as in (3: during training, each batch is used by all the models corresponding to subsets of the observed blocks. During inference, we use the predictor corresponding to the set of observed blocks. We go over the scalability problems of this approach below.
P PLugVFL (Sun al. 2024); extend P PlugVFL to handle missing features during train- ing by replacing missing representations with zeros. This method matches the work proposed
by (Ganguli et al. 2024) for scenarios where all clients have reliable links to one another.
Our experiments focus on the following tasks:
HAPT Reyees al.
2016): human activity recognition dataset.
Credit Yeh & Lien 2009): a dataset for predicting credit card payment defaults.
MIMIC-IV (Johnson al. 2020): a time series dataset concerning healthcare data. We focus on the task of predicting in-hospital mortality from ICU data corresponding to patients admitted with chronic kidney disease. We resort the data processing pipeline of Gupta et al. (2022).
CIFAR-10 CIFAR-100 (1ri)000KKizeevskk (Krizhevsky et 2009): two popular image datasets.
For all datasets, we split the samples into K 4 feature blocks. For example, for the CIFAR datasets, each image is partitioned into four quadrants, each assigned to a different client. Given our interest in ensuring that all clients perform well at inference time, we average the metrics across clients. In Appendix D we provide detailed descriptions of how compute accuracy and Fi-score.
Experiments on HAPT and Credit show that VFedTrans does not significantly outperform Local despite being considerably more complex and slower. The training pipeline of VFedTrans in- cludes three stages: federated representation learning, local representation distillation, and training local model. The first stages are repeated K 1 times for each client pair. This makes VF WedTTrans time-intensive to run and hyperparameter tune, as seen in Appendix [ For this rea- son, limit our experiments with VFedTrans these simpler tasks.

---

Published as a conference paper at ICLR 2025
Performance for different missing data probabilities. In Table local predictors (Local and VFedTrans) are robust to missing blocks during both training and inference, yet they fall behind when all the blocks are observed. In contrast, Standard VFL performs well when all the blocks are observed, yet its performance degrades very quickly in the presence of missing blocks. The Ensemble method improves upon Standa rd VFL significcnl in the presence of missing blocks, as it leverages the robustness its local predictors. Yet, when all the blocks are observed, although Ensemble improves over the local predictors significantly, it still cannot match the performance of Standard VFL. We see that the Comb inatoriaiii approach outperforms most baselines when there is little no missing training data (Pmiss {0.0, 0.1)). Yet, for a larger amount of missing training data (Pmiss 0.5), its performance degrades significantly. This degradation is due to the fact that each mini-batch is only used train predictors that use ob- served feature blocks, harming the generalization of predictors that use larger subset of the features.
that the methods with
This also leads to an interesting phenomenon: when Pmiss 0.5 for training data, Combing Combinatorial performs better for higher probabilities of missing testing data. This is because missing test data dictates the use of predictors trained on smaller, more frequently observed feature sub- sets. PlugVFL often outperforms most of the other base- lines, though not consistently. Specifically, it is at times out- performed by Combinat
100
Test accuracy
60-
40
LASER-VPL Combinatoria
when the missing prob- ability of the training data is low, and by Ensemble l when it high. LASER-VFL. consistently outperforms the baselines across the different probabilities of missing blocks during training and inference. (In fact, in Table [ LASER-VE is never outperformed by more than standard deviation.) In particular, even when the samples are fully- observed, LASER-VEL outperforms Standard VFL and Combinatorian We believe this is due to the regulariza- tion effect of the task-sampling mechanism in our method, which effectively behaves as a form of dropout.
Local
Standard VRL
Emsemble
PugVFL
0
Number clients
800
LASER-IFL
700
Comminatorial
Local
600
Standard VRL
Runtimm (min)
Ensemble
500
PlugVTL
200
300-
Scalability. Figure 2 examines the scalability of the dif- ferent methods with respect to the number of clients, K. In the top plot, we see that LASER-VFL performs the best for all K. It is followed by Combinatorial. Ensemble, and PlugVFL. However, the performance drops faster as K increases, since
200
35.8 x
100
9
Number clients
of Combinatorin each exact combination of feature blocks is observed less frequently. In contrast, Ensemble and PlugVFL, like LASER-VFLL train each representation model on all batches for which its feature block is observed. In the bot- tom plot, we observe the expected scalability issues of Combinatori ial, caused by its exponential complexity.
Figure 2: Performance and runtime across different numbers of clients (CIFAR-10, Pmiss 0.1 for train- ing and inference). We did not run Combi natorial for 8 clients due to resource constraints.
Nonuniform missing features. Table 3 in Appendix E presents an experiment where the prob- ability of each feature block i not being observed is independent and identically distributed as Puniss (k:) ~ (eta 2.0). As in the previous experiments, LASER-VFL. outperforms the baselines.
CONCLUSIONS
In this work, we introduced LASER-VEL novel method for efficient training and inference in vertical federated learning that is robust to missing blocks of features without wasting data. This is achieved by carefully sharing model parameters and by employing a task-sampling mechanism that allows us to efficiently estimate a loss whose computation would otherwise lead to an expo- nential complexity. We provide convergence guarantees for LASER -ALER-VFL and present numerical experiments demonstrating its improved performance, not only in the presence of missing features but, remarkably, even when all samples are fully observed. For future work, it would be interesting to apply LASER-VEE -FL to multi-task learning and investigate how it behaves when different clients tackle different tasks scenario that naturally fits our setup.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work supported in part by the Fundação para a Ciência e Tecnologia through the Carnegie Mellon Portugal Program under grant SFRH//D//50733/202 and by the grants U.S. National Sci- ence Foundation CCF-2007911 and ECCS-2318441.
REFERENCES
Rich Caruana. Multitask learning. Machine learning, 28:41-75, 1997.
Iker Ceballos, Vivek Sharma, Eduardo Mugica, Abhishek Singh, Alberto Roman, Praneeth Vepakomma, and Ramesh Raskar. SplitNN-driven vertical partitioning. arXiv:2008.04137, 2020.
Di Chai, Leye Wang, Junxue Zhang, Liu Yang, Shuowei Cai, Kai Chen, and Qiang Yang. Practical lossless federated singular vector decomposition over billion-scale data, 2022. URL https:
//arxii..rr//bb////11 08925
Siwei Feng. Vertical federated learning-based feature selection with non-overlapping sample uti- lization. Expert Systems with Applications, 208: I 18097, 2022.
Siwei Feng and Han Yu. Multi multi-participant multi-class vertical federated learning. arXiv preprint arXiv:2001.11154, 2020.
Siwei Feng, Boyang Li, Han Yu, Yang Liu, and Qiang Yang. Semi-supervised federated heteroge- neous transfer learning. Knowledge-Bosse -assed Systems, 252: 109384, 2022.
Surojit Ganguli, Zeyu Zhou, Christopher G. Brinton, and David I. Inouye. Fault tolerant server- less vfl over dynamic device environment, 2024. URL
16638
Dashan Gao, Sheng Wan, Lixin Fan, Xin Yao, and Qiang Yang. Complementary knowledge distilla- tion for robust and of the AAAI Conference on Artificial Intelligence, volume 38, pp. 19832-19839, 2024.
model serving in vertical federated learning. In Proceedings
Mchak Gupta, Brennan Gallamoza, Nicolas Cutrona, Pranjal Dhakal, Raphael Poulain, and Rahma- tollah Beheshti. An Extensive Data Processing Pipeline for MIMIC-IV. In Proceedings of the 2nd Machine Learning for Health symposium, volume 193 of Proceedings of Machine Learning Research, pp. 311 1-35- PMLR, 28 Nov 2022. v1 v193/ 111933/upt222aaaaam html
mll. p.esss/
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, 2016.
Yuanqin He, Yan Kang, Xinyuan Zhao, Jiahuan Luo, Lixin Fan, Yuxing Han, and Qiang Yang- A hybrid self-supervised learning framework for vertical federated learning. IEEE Transactions on Big Data, 2024.
Hochreiter. Long short-term memory. Neural Computation MIT-Press, 1997.
Chung-ju Huang, Leye Wang, and Xiao Han. Vertical federated knowledge transfer via representa- tion distillation for healthcare collaboration networks. In Proceedings of the ACM Web Conference 2023, pp. 4188 4199, 2023.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift, 2015. URL htt.:ps//aaxvv..orr/ab///////
Alistair Johnson, Lucas Bulgarelli, Tom Pollard, Steven Horng, Leo Anthony Celi, and Roger Mark. Mimic-iv. PhysioNet. Available online at: https://physionnt. wgg/comtettnimmcci//.oocccm August 23, 202/), l). pp. 49-55, 2020.
11

---

Published as a conference paper at ICLR 2025
Peter Kairouz, H. Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, Rafael G. L. D' D'Oliveira, Hubert Eichner, Salim El Rouayheb. David Evans, Josh Gardner, Zachary Garrett, Adrià Gascón, Badih Ghazi, Phillip B. Gibbons, Marco Gruteser, 7aid Harchaoui, Chaoyang He, Lie He, Zhouyuan Huo, Ben Hutchinson, Justin Hsu, Martin Jaggi, Tara Javidi, Gauri Mikhail Khodak, Jakub Konečný, Aleksandra Korolova, Farinaz Koushanfar, Sanmi Koyejo, Tancrède Lepoint, Yang Liu, Prateck Mittal, Mehryar Mohri, Richard Nock, Ayfer Özgür, Rasmus Pagh, Mariana Raykova, Hang Qi. Daniel Ramage, Ramesh Raskar, Dawn Song, Weikang Song, Sebastian U. Stich, Ziteng Sun, Ananda Theertha Suresh, Florian Tramèr, Praneeth Vepakomma, Jianyu Wang. Li Xiong. Zheng Xu, Qiang Yang, Felix X. Yu, Han Yu, and Sen Zhao. Advances and open problems in federated learning, 2021.
Joshi,
Yan Kang, Yuanqin He, Jiahuan Luo, Tao Fan, Yang Liu, and Qiang Yang. Privacy-preserving fed- erated adversarial domain adaptation over feature groups for interpretability. IEEE Transactions on Big Data, 2022a.
Yan Kang, Yang Liu, and Xinle Liang. Fedevt: Semi-supervised vertical federated learning with cross-view training. ACM Transactions on Intelligent Systems and Technology (TIST), 13(4): 1-16, 2022b.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Vitaly Kurin, Alessandro De Palma, Ily Kostrikov, Shimon Whiteson, and Pawan K Mudigonda. In defense of the unitary scalarization for deep multi-task learning. Advances in Neural Information Processing Systems, 35:12169-12183, 2022.
Wenjie Li, Qiaolin Xia, Hao Cheng, Kouyin Xue, and Shu-Tao Xia. Vertical semi-federated learning for efficient online advertising. In Proceedings of the International Workshop on Trustworthy Federated Learning in Conjunction with IJCAI 2023 (FI-IJCAI 2023a. URL https://
arxiv. org/ abs 2209. 1565
Wenjie Li, Qiaolin Xia, Junfeng Deng, Hao Cheng. Jiangming Liu, Kouying Xue, Yong Cheng, and Shu Xia. Vfed-ssd: Towards practical vertical federated advertising, 2023b. URL https:
//arxiv. owg. /205 15987
Yang Liu, Yan Kang, Chaoping Xing. Tianjian Chen, and Qiang Yang. A secure federated transfer learning framework. IEEE Intelligent Systems, 35(4):70-82, July 2020. ISSN 1941-1294, doi: 10. 202.:iss.220029885222 URL http::ttp//tt doi. wow.ooor......... 111.111111190 /MIS. 2020. 202..2220..22222222 2988525
Yang Liu, Xinwei Zhang, Yan Kang, Liping Li, Tianjian Chen, Mingyi Hong, and Qiang Yang. FedBCD: A communication-effiicenn collaborative learning framework for distributed features. JEEE Transactions on Signal Processing, 70:4277-4290, 2022.
Yang Liu, Yan Kang. Tianyuan Zou, Yanhong Pu, Yuanqin He, Xiaozhou Ye, Ye Ouyang, Ya-Qin Zhang, and Qiang Yang. Vertical federated learning: Concepts, advances, and challenges. JEEE Transactions Knowledge and Data Engineering, 2024.
Boris T Polyak. Gradient methods for the minimisation of functionals. USSR Computational Math- ematics and Mathematical Physics, 3(4)8864-878, 1963.
Zhenghang Ren, Liu Yang, and Kai Chen. Improving availability of vertical federated learning: Relaxing inference on overllaprpppngg data. ACM Transactions on Intelligent Systems and Tech- nology (TIST), 13(4):1-20, 2022.
Jorge-L Reyes-Ortiz, Luca Oneto, Albert Samà, Xavier Parra, and Davide Anguita. Transition-aware human activity recognition using smartphones. Neurocomputing, 171:754-767, 2016.
Jingwei Sun, Ziyue Xu, Dong Yang, Vishwesh Nath, Wenqi Li, Can Zhao, Daguang Xu, Yiran Chen, and Holger R Roth. Communication--fficieee vertical federated learning with limited overlapping samples. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5203-5212, 2023.
12

---

Published as a conference paper at ICLR 2025
Jingwei Sun, Zhixu Du, Anna Dai, Saleh Baghersalimi, Alireza Amirshahi, Qilin Zheng, David Atienza, and Yiran Chen. PlugVFL: Robust and IP-protecting vertical federated learning against unexpected quitting of parties, 2024. URL https://openzeviers wtwp://wwwee/ooourmmmeo
id=TzcuXQQOaR
Pedro Valdeira, João Xavier, Cláudia Soares, and Yuejie Chi. Communication-effccienn vertical federated learning via compressed error feedback. IEEE Transactions on Signal Processing, pp-
1-15, 2025. doi: 10.1 2..11009/SPP222555500655
Yunpeng Xiao, Xufeng Li, Tun Li, Rong Wang, Yucai Pang, and Guoyin Wang. A distributed generative adversarial network for data augmentation under vertical federated learning. IEEE Transactions Big Data, 2024.
Yitao Yang, Xiucai Ye, and Tetsuya Sakurai. Multi-viww federated learning with data collaboration. In Proceedings of the 2022 14th International Conference on Machine Learning and Computing, pp. 178 778--13,, 2022.
I-Cheng Yeh and Che-hui Lien. The comparisons of data mining techniques for the predictive accuracy of probability of default of credit card clients. Expert systems with applications, 36(2): 2473-2480, 2009.
Lei Yu, Meng Han, Yiming Li, Changting Lin, Yao Zhang, Mingyang Zhang, Yan Liu, Haiqin Weng, Yuseok Jeon, Ka-Ho Chow, et al. A survey of privacy threats and defense in vertical federated learning: From model life cycle perspective. arXiv preprint PrXiv:2402.03685 2024.
Youran Zhou, Sunil Aryal, and Mohamed Reda Bouadjenek. Review for handling missing data with special missing mechanism. arXiv preprint arXiv:2404.04900, 2024.
A ADDITIONAL DISCUSSIONS
A.1 A MORE DETAILED COMPARISON WITH CLOSEST RELATED WORKS
As mentioned in Section a few recent works have addressed the problem of missing features during VFL training. In particular, a line of work uses all samples that are not fully observed lo- cally to train the representation models via self-supervised learning (Feng) 2022 He et al. 2024). This is followed by collaborative training of joint predictor for the whole federation using the samples without missing features. Another line of work utilizes a similar framework but resort- ing to semi-supervised learning instead of self-supervised learning in order to take advantage of nonaligned samples (Kang et al.] 2002bb] Yang et al. 2022] Sun et al. 2023). In both cases, these works are lacking in some areas that improved by our method: 1) they require the existence of fully observed samples; 2) they do not allow for missing features at inference time; and 3) they do not leverage collaborative training when more than one, but less than the total number of clients observed their feature blocks.
Another approach leverages information across the entire federation to train local predictors, thereby enabling inference in the presence of missing features. [Fnn & Yu (202)): Kang et al. (002aa) propose following standard, collaborative vertical FL with transfer learning method allowing each client to have its own predictor. These works assume that the training samples are fully observed.
Liu et al. (2020)
al. (2022) follow similar direction but they also consider the presence of missing features. Instead of utilizing transfer learning to train local predictors. Ren et al. (2022); Li
al. (2023b) use knowledge distillation under the assumption of fully observed training samples, while Li et al. (2023a): Huang et al. (2023) also employ knowledge distillation, but on top of this they further consider missing features during training. Finally, Xiao et al. (2024) recently proposed using distributed generative adversarial network for collaborative training on nonoverlapping data, leveraging synthetic data generation. In all of these works, the output of training are local predictors which, despite being able to perform inference when other clients in the federation do not observe their blocks, are also not able to leverage other feature blocks when they are observed. This failure to utilize valuable information during inference leads to reduced predictive power.
13

---

Published as a conference paper at ICLR 2025
Only few, very recent works allow for inference from varying number of feature blocks in ver- tical FL. PlugVFL (Sun et al. 2024) employs party-wise dropout during training to mitigate per- formance drops from missing feature blocks at inference; Gao et al. (2024) introduce
stalti-stage approach with complementary knowledge distillation, enabling inference with different client sub- sets; and Ganguli et al. (0224) deal with the related task of handling communication failure during inference in cross-device VFL. These methods make progress towards inference in vertical FL in the presence of missing feature blocks, however, they do not consider missing features in the training data. Further, none of these methods provides convergence guarantees.
In contrast to the prior art, our LASER-VEL method can handle any missingness pattern in the feature blocks during both training and inference simultaneously without wasting any data. Fur- ther, LASER-VFI. achieves this without requiring complex pipeline with additional stages and hyperparameters, requiring only minor modifications to the standard VFL approach.
A.2 ON THE USE OF AVERAGING AS THE AGGREGATION MECHANISM
As mentioned in Section 3.1 using averaging instead of concatenation as the aggregation mech- anism in the fusion model does reduce the flexibility of the overall approach, provided the representation models are adjusted accordingly. We illustrate this with simple example employing aggregation by sum (which differs only by parameter scaling factor).
Consider the scenario with two clients, each extracting representation, U1 E Rd and V2 E R2 When aggregation is by concatenation, the first layer of the fusion model can be written W Rd (d) +d2), with W = [W] W2]. and we define = [e] oo] Hence, the output of the first layer Wo Wi + W2#2-
In contrast, a naive aggregation by sum would restricts us to Wi (w) + v2) and impose dj = d2- However, if we include an extra linear layer in the representation models for each client (namely Wi in the first and W2 in the second), then the outputs become W and W2v2. This way, even with aggregation by summation, the fusion model can recover Winn Vi + W202.
Thus, by adjusting the representation models appropriately, we can indeed achieve the same flexi- bility as in the concatenation- based approach, while using aggregation by sum or average.
B LASER-VFL DIAGRAM AND TABLE OF NOTATION
D2o0 32200
2.
..D
client 4 does observe its partition and it therefore does not participate in this round
St
22
21
repres.
repres.
repres.
local data
local data
local data
elient
client 2
client
elient 4
Figure Diagram illustrating a forward pass LASER-VEL.
14

---

Published as a conference paper at ICLR 2025
Table Table of Notation
Symbol
Description
K
Number of clients
N
Number samples
xn"
Sample n
Block k of sample
D
Global dataset
Dx
Local dataset
Parameters of our model
0c
Parameters of the standard VFL model
ff
Parameters of the representation model at client A:
0g.
Parameters of the fusion model at client A:
Loss function
C
Objective function
Predictor of the standard VFL model
Predictor at client using feature blocks J
(SS)
Power set of S
(S)
Subsets in the power set of S containing j
(S)
Subsets in the power set of S of size i containing
fx.
Representation model at client
9k
Fusion model at client k
hg
Predictor at client
Standard VFL fusion model
Standard VFL predictor
K
Set of observed features blocks of sample n
x'
Test sample
C PROOFS
C.I PRELIMINARIES
The following quadratic upper bound follows from the L-smoothness of L Al
L
V 0,, 02 E O 5 (0))2) + (L) - 02) + / 110 02 12.
(1I)
Let X denote a random variable and let be a convex function, Jensen's inequality states that
F(B(X)) < E(F(X)).
(12)
We define the following shorthand notation for conditional expectations used our proofs:
E, [| = E [: K.),,
E, [ = E Ko: Ft
Ext E [ KT BB.
15

---

Published as a conference paper at ICLR 2025
C.2 PROOF OF LEMMA 1
From the law of total expectation, we have that
Ex [Cres. (0')
Ex
Ex
(()ss((0))
(13)
Focusing on the inner expectation, we have from the definition of s::s: in and that
E++ (S (0)) =
Σ
a
Ex+
(14)
B
nEB
kEX
i=1
Now, from the fact that Shi ~ U(P (XC) }), we have that
Ex+ Funsy (0))
Σ
P(S I) (0')
IEPL(KE
Σ
Likz (0')
(
IE(P
K3 1
Σ
(0"z
So, we have from 14 and from the fact that a¹ ( /t, as defined in that
k
E
(BBS:(()) (00)
1
Σ
(
1Bc
kEK!!)
(ES
Σ
Σ
(0t)
[
nEB' kEX!! (E
So, from 130 and A2 it follows that
Eg
( (0*) E, [Cg (0') = L
Therefore, have from the dominated convergence theorem, which allows interchange the expectation and the gradient operators, that
E2 VL (0*) = VE, (00)) VC)
C.3 PROOF OF THEOREM II
Let us show the convergence of our method. It follows from the quadratic upper bound in 11 which ensues from the L-smoothness assumption (AT). that
L (0 ('( T (0++1 0²) 10 0911
Using the fact that 0-+1 = 0¹ - v L (0t), we get that:
22
- < - ((' T VL (0') +
Lo (0')
2
We now take the conditional expectation E, [:]. defined in Appendix
C.2
arriving that:
22L R,
F2 [[] (0') - E, VLBIS: +
2
16

---

Published as a conference paper at ICLR 2025
Now, under the unbiasedness assumption A2
use Lemma 1 to arrive at:
n
Ee [[ - (" =- }|
(15)
From Lemma D we also have that the following equation holds
Ex -
v E V2 -
(16)
Further, we have that
E. (CBrs: VL E, V s: V VLg (0') VC (0)) vC
E, Vi (0')) V V (0') 2 + E, VL (0') - vL)(0']
+ 2E, Eel (VLxss: - VC (0)))))(((()))))))))))))))))))))
(VEg - vC
Recalling that Ex+ (0
= V VLgg (0'), as seen in Appendix C.2
we get
Eg Ex+ (VL (0*)
v Vigg (Gg) T (v. (0') (O)(
]| 0.
Further, it follows from the bounded variance assumption in A3 that
E, Vgg (0)) (C) s oه
and
2
E2
V LBss:
L. E,
Σ KEK
Larsy (0') V (0")
S K 0².
Therefore, arrive
E, vi VLL So
Using the inequality above in (6) we get that
E, Visss [1] <|7"0 K
+ 0z
which we can use in 15 to arrive at
LL
| + lLL
E, - < < +
(o.
2
2
=- (1- 1 (
nLL
(0) K
Therefore, for stepsize 7) 1/) we have that
E, (0(++)) (0) < 1| 1 (of
Taking the unconditional expectation (over K,, and F(+|) on both sides of this inequality, we get:
1LL
E (( - c E ||<
+ 2
K. K
Now, using the law of total expectation, we get that
r²²L
E E 11 (((())) |
F++||| < " E
|
(o) + K -0,)
2
17

---

Published as a conference paper at ICLR 2025
Therefore, using the fact that L(0) E[C) and Jensen's inequality 12 have
n (o + K o.))
EC(e*+1) - E
- E VC") |]+
"7 2
Using the dominated convergence theorem, we can interchange the expectation and the gradient operators, arriving at the following descent lemma in expectation:
r 2
EC( EC(0') - - E V
(o + K k..))
(17)
Taking the average of the inequality above for & {0, 10.
1}, we get that:
EC(0T) -(00)
" Σ
n (o k o.)
S
T
27
t=
Lastly, rearranging the terms and using the existence and definition of C* A1
we have that
T-1
2(C(0)) - L*) + + (n + K -0.)
Σ
B||
n/T
thus arriving at the result in (10).
C.4 PROOF OF THEOREM 2
Following the same steps as in the proof of Theorem expectation (1).
1 we can arrive the same descent lemma in
T2L
EC( EC(O') < - - [[ +
Rearranging the terms and subtracting the infimum on both sides, we get that
L (o) k
EL(O++1) L' <
L-
Now, using the definition of our Lyapunov function, 8' EE((0") - L*, we arrive at the following inequality:
g++1 S 8t -. VC())) + 1
2 (o K k
2
Now, from the PL inequality (44). we have that:
g++1 8 pm (C(0)) C*] + T2L (0 + K -
Therefore, again using the definition of our Lyapunov function, we arrive at
L² (ot + k
g++1 (1 um) 5"
Recursing the inequality above, we get that
72 8T < (1- +
2 00 + K
o3 Σ
(1 um)
t=0
Finally, using the sum of geometric series, we arrive at
nL
8T (1 pm) T
T
(o) + 1o k
2μ
which corresponds to the result that we out prove.
18

---

Published as a conference paper at ICLR 2025
D EXPERIMENT DETAILS
Notes on VFedTrans. For the VFedTrans (Huang et al. 2023) pipeline, we use FedsVD CChai al. 2022) for federated representation learning and an autoencoder for local representation learn- ing, as these are the best performing methods in Huang et al.] (2023). For the local predictor, we use multilayer perceptron, as Huang et al. (023). As seen Table [ the performance of "FedTrans is nearly identical to that of Local. However, given that requires a multi-stage pipeline with operations for each pair of clients, its runtime is much longer. In particular, for the Credit dataset, for Pmiss = 0.0 for training and inference, running VFedTrans takes 2843.0 while running Local takes only 127.8 + 3.0s.
Notes on PLugVEL. We also note that the original P PlugVFL paper (Sun et al. 2024) 1) does not address missing training data and 2) only considers settings with two clients. In our experiments, we extended P PlugVEL to handle more generic scenarios. To address missing training data fairly, we replaced representations for missing feature blocks with zeros at the fusion model, similarly to the dropout mechanisms in the original paper, but now, for missing data, these representations are zeroed throughout training rather than per iteration. To generalize PlugVFL for more than two clients, assigned a dropout probability to each passive party (clients not holding the fusion model).
Notes on MIMIC-IV. We use ICU data of MIMIC-IV v1.0 (Johnson et al. 2020) and follow the data processing pipeline in Gupta et al. (2022). We focus the task of predicting in-hospital mortality for patients admitted with chronic kidney disease. We do feature selection with diagno- sis data and the selected features of chart events (labs and vitals) used in the "Proof of concept experiments" Gupta et al. (2022) as input features.
Computing metrics. As mentioned in the main text the paper, we want our metrics to capture that fact that we want all clients to perform well at inference time. Therefore we consider the average metrics across the clients. More precisely, we compute the metrics for Table T as follows:
For the test accuracy, let i" (k) denote the prediction of client & for sample n, we compute:
N
accuracy = 100 X
(0) 5²")
N n=1
K
REA
For the Fi-score, we compute:
K
1
P Rs
R Rsy
P -c-oe
K
Ps
where Pa and RA are the precision and recall of client k. with respect to its predictor and (only) the samples it observed.
Models used. For the HAPT and Credit datasets, we trained simple multilayer perceptrons: for the MIMIC-IV dataset, we trained an LSTM (Hochreiter) 1997); ); and, for the CIFAR-10 and CIFAR-100 experiments, we use ResNets18 (He et al. 2016) models.
E ADDITIONAL EXPERIMENTS
In this appendix, we present additional experimental results.
Nonuniform missing features. Table 3 we present an experiment where the probability of each feature block & not being observed, Pmiss k)), is sampled independent and identically distributed
following Panss(AA (k) ~ Beta(2.0, (2.0, 2.0). Note that, for X ~ Beta(2.0,2.0)) we have that E[X] = 0.5. comparison with Pmiss (k) 0.5 for all k. For the column corresponding to the case where both the probability of missing training data and inference data are nonuniform, these probabilities are sampled independently for training and inference.
This motivates
19

---

Published as a conference paper at ICLR 2025
As in the previous experiments, we that LASER-VEL outperforms the baselines. Further, not only does the nonuniform nature of Pmiss (k) not harm performance, but in fact our LASER-VFL method seems to do slightly better when the feature blocks have missing training data with proba- bility Pmiis (k) ~ (eta 2.0) rather than when we simply have Pmiss (k) 0.5, which corresponds the expected value of the random variable Pmiss
Table 3: Test accuracy for different probabilities of missing blocks during training and inference (including nonuniform) for CIFAR-10 10 with K = clients, averaged over five seeds (# standard deviation).
Training Priss (8)
Bcta(2.0,2.0)
0.5
Inference (k)
Beta(2.0,2.0)
0.5
CIFAR-10 10 (aceuracy. %)
Local
72.7 + 1.9
72.8
72.6 2.2
7344 1.0
71.2
71.1 0.4
71.3 0.7
Standard VFL
11.6
510 15.7
13.9 5.4
13.5
519 10.0
38.6 9.1
2.2
Ensemble
$3.9
82.7
75.3 2.6
75.4
8..4
81.0
0.8
73.8 03.8400 0.6
Combinatorial
67.8
70.2
73.4=
75.9
54.9
58.6 7.7
68.4
#lugveL
80.1
77.8-0.4
76.5
1.5
72.4
LASER-VEL (ours)
88.8 1:
88.2 1.6
81.
81.7 1.0
87.4
86.4 0.6
79.4 1144110
20


Reviews:
[{"review_id": "xvZqj3DK9O", "reviewer": "Reviewer_DiU7", "summary": "This paper studies vertical FL with missing features. The authors propose a new vertical FL method named **LASER-VFL** for efficient training and inference of split neural network-based models. This method relies on the strategic sharing of model parameters and on task-sampling to train a family of predictors. The authors also provide convergence rates for **LASER-VFL** under different objective functions.", "questions": "1. What do hyperparameters refer to in vertical FL? Is it the optimization parameters such as learning rate or some specific parameter in vertical FL?\n\n2. Does the proposed algorithm allow multiple local updates in each machine like traditional FL?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This paper proposes a new algorithm to address the missing features in vertical FL.\n\n2. Valid convergence analysis and experiments are provided.", "weaknesses": "1. The notations are too complex, which is not friendly for readers. The authors should try to simplify notations.\n\n2. I'm not an expert in vertical FL, so I'm confused about the **key idea** presented on page 4.  So it will be better to add workflow diagrams to explain the method.\n\n3. The experiments are conducted on the data with synthetic missing features. It would be more convincing to show results on real-world vertical FL data with missing features.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "u004YPZ8sg", "reviewer": "Reviewer_fR4j", "summary": "This paper focuses on vertical federated learning (VFL), where data is partitioned by features rather than by samples. In this setting, each client possesses only a subset of the features for a specific sample. The paper addresses the challenge of missing features in the dataset and the heterogeneous availability of clients during the federation. Standard VFL approaches cannot handle these challenges, as it assumes that all features are available during both training and inference. To tackle these issues, this work proposes LASER-VFL, which relies on strategic sharing of model parameters and task sampling to train a family of predictors, allowing training and inference using any and all available blocks of features, leading to full utilization of the data. The paper provides theoretical guarantees for non-convex objectives and demonstrates empirical results on four different datasets, showing good performance.", "questions": "How would the performance of the aggregation mechanism differ when using weighted averaging rather than simple averaging? What if some clients have more powerful feature blocks? Giving uniform weights would alter the performance compared to other concatenation methods.\n\nHow would the performance be with even larger number of clients?\n\nFor small number of clients, the combinatorial approach becomes feasible, how does the combinatorial approach (which can be considered as an upper bound) differ from the proposed method?\n\nHow does the method perform against the other discussed related works which does solve the same problem but may have different training stages/modules?\n\nHow would the results be in a convex setting? Where does it converge? Does it guarantee optimal performance of VFL with missing clients?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The authors propose an approach for VFL that enables both training and inference using all available blocks of features.\n\nThe authors provide theoretical guarantees for convergence under L-smoothness, unbiasedness, and bounded variance assumptions. \n\nThe authors conduct empirical evaluations of the proposed methods over 4 datasets, showing good performance.", "weaknesses": "For VFL models, the authors mention that split neural networks are often considered and explains that they seek to find multiple parameters for each block and for a fusion model. Then mentions that they aim to solve (1) which requires all the blocks, which contradicts the paper objective seeking VFL with missing blocks. The authors should be clear from the early stage whether they aim to solve (1), or other works aimed to solve (1). \n\nThe authors mentions that LASER-VFL enables both training and inference\nusing all available blocks of features. However, in Figure 1 (b) (our method), shows that samples 4, 6, and 7, which has missing blocks, are only used for training (and not for testing), which is not clear why.\n\nThe authors go through some previous works very briefly in the introduction without going into details of their approaches and mention that these previous works add training stages and modules, making them more complex. While related work was provided later this paragraph seemed both redundant and unclear. \n\nThe authors mentioned some works that dealt with missing features. Then surprisingly claim that to the best of their knowledge, LASER-VFL is the first method to enables both training and inference using any and all available blocks of features. (line 108: We propose LASER-VFL (Leveraging All Samples Efficiently for Real-world VFL), a novel method that enables both training and inference using any and all available blocks of features. To the best of our knowledge, this is the first method to achieve this.)\n\nCitations should be \\citep when discussing some paper (not the authors). For example in line 144 it should be in (Feng & Yu 2020; Kang et al. 2022a) not in Feng & Yu (2020); Kang et al. (2022a).\n\nIn VFL problems what cases would really requires more than a few clients to collaborate? In the empirical setting the authors considered 4 clients. If we have a few companies, why not simply do a combinatorial approach? For such few client cases, say 2 or 3, the authors could show the optimal performance of training the 2^k-1 models as an optimal target against the proposed solution. \n\nThe authors introduce a lot of notations, which should be minimized to the least possible to make it easier to follow. (While the authors clearly mentions that the notation in (3) differs slightly from (1) and (2), this still should be avoided as much as possible). \n\nWhile the authors mention several methods that dealt with missing blocks, most of these approaches were not considered in the empirical evaluation. Being more complex and requiring multiple stages or modules should not be a reason to skip the comparison. It could be that these added stages or complexities lead to better accuracy and/or better efficiency in terms of rounds to accuracy. \n\nGiving uniform weights in the aggregation of the feature representations of different blocks of features for the different clients (Eq. (4)) would alter the performance compared to other concatenation methods. Averaging and sum can cancel some information which can be a drawback compared to concatenation methods.\n\nWhile the authors claim that the method is suitable for a growing number of clients, avoiding the exponential growth in computation and memory. Empirical results should be provided to deal with more than 4 clients to see that the method indeed can achieve good performance. For example, would be nice to show for 8 and/or 16 clients. \n\nThe theory shows convergence in the non-convex setting, under L-smoothness and other assumptions, which is appreciated. However, such results do not reveal the optimality of the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DuBAhNO4jX", "reviewer": "Reviewer_wVzL", "summary": "This paper proposes to handle missing features in VFL during both training and testing by sharing model parameters and sampling different configurations to approximate the exponential number of possible predictors (with different possible non-missing features).\nThe intuition is that entirely local models are robust to missing features but poor predictors while global models are stronger predictors but not robust missing features.\nThe paper proposes to train a set of models for all possible subsets of missing features by sharing model parameters and sampling different missingness patterns, called \"tasks\" in this paper, during training time.\nThe sampled tasks are weighted to approximate the loss function of all missing value patterns using Monte Carlo sampling of the tasks.\nThe paper provides theoretic results on convergence and empirically demonstrates their method compared to several baseline methods.", "questions": "- How does this paper compare to the methods proposed in Ganguli et al., [2023]? They seem to be doing something very similar including the dropout mechanism during training. Perhaps, the authors missed this very related work.\n\n- The model architecture resembles a single-layer GNN with summation aggregation. Each client has it's own fusion predictor. Is this correct or am I missing something?\n\n- While the weighting method for learning an \"exponential number of models\" is interesting, it is perhaps a bit misleading to argue for the \"compactness\" of learning an exponential number of models. Simple dropout could also learn \"an exponential number of models\" but it is difficult to argue that these models would be anywhere near the naive solution of actually directly training an exponential number of models.\n\n- There are other types of parametric aggregation methods that can handle arbitrary sizes. For example, an attention-based aggregation or similar is a parametric aggregation method but can handle arbitrary length inputs.\n\n- The notation is a bit overwhelming and needs to be simplified and clarified. In Eq 5 alone, there are 4 different uses of the $\\mathcal{L}$ with different subscripts.\n\n- Choosing a mini-batch with the same missing features in a real-world VFL scenario would introduce bias in the training batch since it is unlikely for the missing features to be missing completely at random (MCAR). Rather, different groups of samples will likely have similar missing features. Can you explain how your method would handle this? This seems to be a severe limitation as the exact missingness pattern is unlikely to happen often unless the number of clients is very small.\n\n- Related to above, it seems that this biasedness in mini-batches would cause Assumption 2 in your theory to be violated. Thus, it is unclear if your theory is reasonable for the practical algorithm and real-world situation.\n\n- Relatedly, how do you handle the fact that there might be missingness patterns that are far more common than others? \n\n- Does this assume that all clients participate during the training process (i.e., no client leaves even temporarily)? It seems that this requires all clients to be available during training (even if they do not participate in the training round).\n\n- How are the baseline methods trained when there is missingness at test time? Are the batches with high missingness simply thrown out? It seems there are some simple baselines to adapt standard VFL by simply imputing zeros at train and test time or something like that. Stronger baselines with some naive adaptations like this for missing values would be helpful.  Currently, it seems that standard VFL is unfairly penalized.\n\n- Do your experiments assume missingness completely at random (MCAR)? I'm guessing they do but wanted to double check. This would align with your theory.\n\n- What happens if you do not weight things but just use simple dropout when sampling your missing features?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper proposes a Monte-Carlo approach with weighting to train a split-NN architecture to be robust to missing features during training and inference. The method is relatively simple and avoids complex hyperparameters or multi-stage configurations.\n\n- The paper provides theoretic convergence results on their proposed method.\n\n- The empirical results show significant improvement over baselines methods used in the paper when there is missingness at inference time.", "weaknesses": "------ After reading author response ------\nWhile I still have some concerns, the author response helped reduce them. Thus, I've increased my score by 1.\n------ Original -------\n- Either the setting is a small number of clients (say 4 as used in the experiments) and then it might be feasible to actually just train all possible models---this calls into question the motivation for learning \"an exponential number of models\" since there are only 24 models. Or, the setting is a large number of clients (say 10-100) and then finding large batches of samples such that the missing values are the same would result in either small or biased batches---which would violate assumption 2. Thus, the motivation of the paper or the requirement for large unbiased batches seem to be at odds with each other. \n\n- The assumption of unbiased batches would require the missingness to be Missing Completely at Random (MCAR) to ensure that the batches are unbiased. This seems unlikely to hold in real-world applications.\n\n- The paper seems to overclaim a bit on the novelty or significance of the model sharing aspect. The model sharing aspect is a common approach in GNNs or other modern networks such as attention-based architectures, yet these do not claim significant novelty in terms of sharing model parameters across nodes.\n\n- The paper lacks an empirical comparison to the method proposed in Ganguli et al. [2023] that can handle inference time faults. This seems to be a closely related method. They use dropout (like your task sampling), gossiping (like the ensemble-based baseline), and replication of the fusion/predictor (as you do as well).\n\n[Ganguli et al., 2023] Ganguli, S., Zhou, Z., Brinton, C. G., & Inouye, D. I. (2023). Fault Tolerant Serverless VFL Over Dynamic Device Environment. arXiv preprint arXiv:2312.16638.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YNCysYEfYd", "reviewer": "Reviewer_GeBH", "summary": "The author(s) proposed LASER-VFL to handle the missing feature challenge arising from the vertical federated learning problem. By transforming the objective into a finite sum problem, LASER-VFL can handle missing features (clients) via the task sampling technique, which is similar to client dropout. LASER-VFL is backed with theoretical analysis and some simple numerical experiments.", "questions": "By transforming the objective into a finite sum problem, is the convergence analysis almost the same as the classic convergence analysis of SGD?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The writing is clear in general;\n- The paper is well-motivated, the missing feature during training and inference is a crucial problem in vertical federated learning from a practical perspective.\n- The proposed method is simple and easy to understand.", "weaknesses": "- The experiment section is rather weak:\n    - Experiments details are missing, for example, the train/test split of dataset, the neural network architecture being used, the hyper-parameter tunning...\n    - A baseline is missing, the naïve method described in section 2 (train exponentially many predictors).\n    - The number of clients is set to 4 for all experiments, how does the result change when we increase the number of clients? If there are 4 clients, then it is not really computational prohibitive to train exponentially many predictors.\n- Notations are quite heavy in line 293-342.\n- The theoretical analysis assumes features are missed uniform randomly, which simplifies the problem significantly. By this assumption and transforming the objective into a finite sum problem, the author(s) could directly use classic convergence analysis of SGD without much technical modification. Therefore, the theoretical novelty is kind of limited.\n- Minor: missing reference at line 855.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ph8CxmpuSG", "reviewer": "Reviewer_Cmzg", "summary": "This paper focuses on vertical federated learning. More specifically, the authors present a framework called LASER-VFL, which is more robust in handling missing features and any arbitrary combination of partitions. They provide a convergence rate analysis along with  experimental results.", "questions": "- Why are there K fusion models?  Is it because each client needs a specialized fusion model for its own task?\n- Could the authors elaborate on line 2 in Algorithm 1 (Client K sample the same mini-batch .... via a shared seed)? What is the impact of this shared seed, and why is the same mini-batch important?\n- Could the authors elaborate on how domain shift is addressed? Or how would the framework generalize if there were a significant level of domain shift across clients?\n- Does the proposed framework have specific requirements for the fusion model and projectors? Also, what is the architecture of the fusion model and predictors in Section 5?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "I really enjoyed reading this paper. The presentation is good and easy to follow, and the experimental results are very convincing. More specifically:\n- The paper addresses missing features problem, which is an important area of study in VFL.\n- The proposed idea is simple yet effective. The authors create a fusion of arbitrary combinations of predictors on each client, allowing them to leverage all available features. To avoid sharing an exponential number of predictors, they propose a strategy in which only K predictors and K fusion models need to be shared. \n- They demonstrate improved performance compared to the existing works.", "weaknesses": "- While the authors present related works, it is difficult to identify which components of the proposed framework are novel when reading Sections 2 and 3. I encourage the authors to provide a more detailed comparison with existing VFL approaches that address missing features.\n- The results are convincing, especially given that they cover different levels of missing features. However, only two VFL methods are used for comparison. The authors are encouraged to include additional comparisons to better highlight the work's significance in the field of VFL.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)