Compressed-VFL: Communication-Efficient Learning with Vertically Partitioned Data

T ) when the compression error is bounded over the course of training . We provide specific requirements for convergence with common compression techniques , such as quantization and top-k sparsification . Finally , we experimentally show compression can reduce communication by over 90 % without a significant decrease in accuracy over VFL without compression . 1 INTRODUCTION . Federated Learning ( McMahan et al. , 2017 ) is a distributed machine learning approach that has become of much interest in both theory ( Li et al. , 2020 ; Wang et al. , 2019 ; Liu et al. , 2020 ) and practice ( Bonawitz et al. , 2019 ; Rieke et al. , 2020 ; Lim et al. , 2020 ) in recent years . Naive distributed learning algorithms may require frequent exchanges of large amounts of data , which can lead to slow training performance ( Lin et al. , 2020 ) . Further , participants may be globally distributed , with high latency network connections . To mitigate these factors , Federated Learning algorithms aim to be communication-efficient by design . Methods such as local updates ( Moritz et al. , 2016 ; Liu et al. , 2019 ) , where parties train local parameters for multiple iterations without communication , and message compression ( Stich et al. , 2018 ; Wen et al. , 2017 ; Karimireddy et al. , 2019 ) reduce message frequency and size , respectively , with little impact on training performance . Federated Learning methods often target the case where the data among parties is distributed horizontally : each party ’ s data shares the same features but parties hold data corresponding to different sample IDs . This is known as Horizontal Federated Learning ( HFL ) ( Yang et al. , 2019 ) . However , there are several application areas where data is partitioned in a vertical manner : the parties store data on the same sample IDs but different feature spaces . An example of a vertically partitioned setting includes a hospital , bank , and insurance company seeking to train a model to predict something of mutual interest , such as customer credit score . Each of these institutions may have data on the same individuals but store medical history , financial transactions , and vehicle accident reports , respectively . These features must remain local to the institutions due to privacy concerns , rules and regulations ( e.g. , GDPR , HIPAA ) , and/or communication network limitations . In such a scenario , Vertical Federated Learning ( VFL ) methods must be employed . Although VFL is less well-studied than HFL , there has been a growing interest in VFL algorithms recently ( Hu et al. , 2019 ; Gu et al. , 2021 ; Cha et al. , 2021 ) , and VFL algorithms have important applications including risk prediction , smart manufacturing , and discovery of pharmaceuticals ( Kairouz et al. , 2021 ) . Typically in VFL , each party trains a local embedding function that maps raw data features to a meaningful vector representation , or embedding , for prediction tasks . For example , a neural network can be an embedding function for mapping the text of an online article to a vector space for classification ( Koehrsen , 2018 ) . Referring to Figure 1a , suppose Party 1 is a hospital with medical data features x1 . The hospital computes its embedding h1 ( θ1 ; x1 ) for the features by feeding x1 through a neural network . The other parties ( the bank and insurance company ) , compute embeddings for their features , then all parties share the embeddings in a private manner ( e.g. , homomorphic encryption , secure multi-party computation , or secure aggregation ) . The embeddings are then combined in a server model θ0 to determine the final loss of the global model . A server model ( or fusion network ) captures the complicated interactions of embeddings and is often a complex , non-linear model ( Gu et al. , 2019 ; Nie et al. , 2021 ; Han et al. , 2021 ) . Embeddings can be very large , in practice , sometimes requiring terabytes of communication over the course of training . Motivated by this , we propose Compressed Vertical Federated Learning ( C-VFL ) , a general framework for communication-efficient Federated Learning over vertically partitioned data . In our algorithm , parties communicate compressed embeddings periodically , and the parties and the server each run block-coordinate descent for multiple local iterations , in parallel , using stochastic gradients to update their local parameters . C-VFL is the first theoretically verified VFL algorithm that applies embedding compression . Unlike in HFL algorithms , C-VFL compresses embeddings rather than gradients . Previous work has proven convergence for HFL algorithms with gradient compression ( Stich et al. , 2018 ; Wen et al. , 2017 ; Karimireddy et al. , 2019 ) . However , no previous work analyzes the convergence requirements for VFL algorithms that use embedding compression . Embeddings are parameters in the partial derivatives calculated at each party . The effect of compression error on the resulting partial derivatives may be complex ; therefore , the analysis in previous work on gradient compression in HFL does not apply to compression in VFL . In our work , we prove that , under a diminishing compression error , C-VFL converges at a rate of O ( 1√ T ) , which is comparable to previous VFL algorithms that do not employ compression . We also analyze common compressors , such as quantization and sparsification , in C-VFL and provide bounds on their compression parameters to ensure convergence . C-VFL also generalizes previous work by supporting an arbitrary server model . Previous work in VFL has either only analyzed an arbitrary server model without local updates ( Chen et al. , 2020 ) , or analyzed local updates with a linear server model ( Liu et al. , 2019 ; Zhang et al. , 2020 ; Das & Patterson , 2021 ) . C-VFL is designed with an arbitrary server model , allowing support for more complex prediction tasks than those supported by previous VFL algorithms . We summarize our main contributions in this work . 1 . We introduce C-VFL with an arbitrary compression scheme . Our algorithm generalizes previous work in VFL by including both an arbitrary server model and multiple local iterations . 2 . We prove convergence of C-VFL to a fixed point on non-convex objectives at a rate ofO ( 1√ T ) for a fixed step size when the compression error is bounded over the course of training . We also prove that the algorithm convergence error goes to zero for a diminishing step size if the compression error diminishes as well . Our work provides novel analysis for the effect of compressing embeddings on convergence in a VFL algorithm . Our analysis also applies to Split Learning when uploads to the server are compressed . 3 . We provide convergence bounds on parameters in common compressors that can be used in CVFL . In particular , we examine scalar quantization ( Bennett , 1948 ) , lattice vector quantization ( Zamir & Feder , 1996 ) , and top-k sparsification ( Lin et al. , 2018 ) . 4 . We evaluate our algorithm by training LSTMs on the MIMIC-III dataset and CNNs on the ModelNet10 dataset . We empirically show how C-VFL can reduce the number of bits sent by over 90 % compared to VFL with no compression without a significant loss in accuracy of the final model . Related Work . Richtárik & Takác ( 2016 ) ; Hardy et al . ( 2017 ) were the first works to propose Federated Learning algorithms for vertically partitioned data . Chen et al . ( 2020 ) ; Romanini et al . ( 2021 ) propose the inclusion of an arbitrary server model in a VFL algorithm . However , these works do not consider multiple local iterations , and thus communicate at every iteration . Liu et al . ( 2019 ) , Feng & Yu ( 2020 ) , and Das & Patterson ( 2021 ) all propose different VFL algorithms with local iterations for vertically partitioned data but do not consider an arbitrary server model . In contrast to previous works , our work addresses a vertical scenario , an arbitrary server model , local iterations , and message compression . Message compression is a common topic in HFL scenarios , where participants exchange gradients determined by their local datasets . Methods of gradient compression in HFL include scalar quantization ( Bernstein et al. , 2018 ) , vector quantization ( Shlezinger et al. , 2021 ) , and top-k sparsification ( Shi et al. , 2019 ) . In C-VFL , compressed embeddings are shared , rather than compressed gradients . Analysis in previous work on gradient compression in HFL does not apply to compression in VFL , as the effect of embedding compression error on each party ’ s partial derivatives may be complex . No prior work has analyzed the impact of compression on convergence in VFL . Outline . In Section 2 , we provide the problem formulation and our assumptions . Section 3 presents the details of C-VFL . In Section 4 , we present our main theoretical results . Our experimental results are given in Section 5 . Finally , we conclude in Section 6 . 2 PROBLEM FORMULATION . We present our problem formulation and notation to be used in the rest of the paper . We let ‖a‖ be the 2-norm of a vector a , and let ‖A‖F be the Frobenius norm of a matrix A . We consider a set of M parties { 1 , . . . , M } and a server . The dataset X ∈ RN×D is vertically partitioned a priori across the M parties , where N is the number of data samples and D is the number of features . The i-th row of X corresponds to a data sample xi . For each sample xi , a party m holds a disjoint subset of the features , denoted xim , so that x i = [ xi1 , . . . , x i M ] . For each x i , there is a corresponding label yi . Let y ∈ RN×1 be the vector of all sample labels . We let Xm ∈ RN×Dm be the local dataset of a party m , where the i-th row correspond to data features xim . We assume that the server and all parties have a copy of the labels y . For scenarios where the labels are private and only present at a single party , the label holder can provide enough information for the parties to compute gradients for some classes of model architectures ( Liu et al. , 2019 ) . Each party m holds a set of model parameters θm as well as a local embedding function hm ( · ) . The server holds a set of parameters θ0 called the server model and a loss function l ( · ) that combines the embeddings hm ( θm ; xim ) from all parties . Our objective is as follows : minimize Θ F ( Θ ; X ; y ) : = 1 N N∑ i=1 l ( θ0 , h1 ( θ1 ; x i 1 ) , . . . , hM ( θM ; x i M ) ; y i ) ( 1 ) where Θ = [ θT0 , . . . , θ T M ] T is the global model . An example of a global model Θ is in Figure 1a . For simplicity , we let m = 0 refer to the server , and define h0 ( θ0 ; xi ) : = θ0 for all xi , where h0 ( · ) is equivalent to the identity function . Let hm ( θm ; xim ) ∈ RPm for m = 0 , . . . , M , where Pm is the size of the m-th embedding . Let ∇mF ( Θ ; X ; y ) : = 1 N ∑N i=1∇θm l ( θ0 , h1 ( θ1 ; xi1 ) , . . . , hM ( θM ; xiM ) ; yi ) be the partial derivatives for parameters θm . Let XB and yB be the set of samples and labels corresponding to a randomly sampled mini-batch B of size B . We let the stochastic partial derivatives for parameters θm be ∇mFB ( Θ ; X ; y ) : = 1 B ∑ xi , yi∈XB , yB ∇θm l ( θ0 , h1 ( θ1 ; xi1 ) , . . . , hM ( θM ; xiM ) ; y ) . We may drop X and y from F ( · ) and FB ( · ) . With a minor abuse of notation , we let hm ( θm ; XBm ) : = { hm ( θm ; xB 1 m ) , . . . , hm ( θm ; x BB m ) } be the set of all party m embeddings associated with mini-batch B , where Bi is the i-th sample in the mini-batch B . We let ∇mFB ( Θ ) and ∇mFB ( θ0 , h1 ( θ1 ; XB1 ) , . . . , hM ( θM ; XBM ) ) be equivalent , and use them interchangeably . Assumption 1 . Smoothness : There exists positive constants L < ∞ and Lm < ∞ , for m = 0 , . . . , M , such that for all Θ1 , Θ2 , the objective function satisfies ‖∇F ( Θ1 ) −∇F ( Θ2 ) ‖ ≤ L ‖Θ1 −Θ2‖ and ‖∇mFB ( Θ1 ) −∇mFB ( Θ2 ) ‖ ≤ Lm ‖Θ1 −Θ2‖ . Assumption 2 . Unbiased gradients : Form = 0 , . . . , M , for a randomly selected mini-batch B , the stochastic partial derivatives are unbiased , i.e. , EB∇mFB ( Θ ) = ∇mF ( Θ ) . Assumption 3 . Bounded variance : For m = 0 , . . . , M , there exists constants σm < ∞ such that the variance of the stochastic partial derivatives are bounded as : EB‖∇mF ( Θ ) −∇mFB ( Θ ) ‖2 ≤ σ2m B for a randomly selected mini-batch B of size B . Assumption 1 bounds how fast the gradient and stochastic partial derivatives can change . Assumptions 2 and 3 require that the stochastic partial derivatives are unbiased estimators of the true partial derivatives with bounded variance . Assumptions 1–3 are common assumptions in convergence analysis of gradient-based algorithms ( Tsitsiklis et al. , 1986 ; Nguyen et al. , 2018 ; Bottou et al. , 2018 ) . We note Assumptions 2–3 are similar to the IID assumptions in HFL convergence analysis . However , in VFL settings , all parties store identical sample IDs but different subsets of features . Hence , there is no equivalent notion of a non-IID distribution in VFL . Assumption 4 . Bounded Hessian : There exists positive constants Hm for m = 0 , . . . , M such that for all Θ , the second partial derivatives of FB with respect to hm ( θm ; XBm ) satisfy : ‖∇2hm ( θm ; XBm ) FB ( Θ ) ‖F ≤ Hm for any mini-batch B . Assumption 5 . Bounded Embedding Gradients : There exists positive constants Gm for m = 0 , . . . , M such that for all θm , the stochastic embedding gradients are bounded by : ‖∇θmhm ( θm ; XBm ) ‖F ≤ Gm for any mini-batch B . Since we are assuming a Lipschitz-continuous loss function ( Assumption 1 ) , we know the Hessian of F is bounded . Assumption 4 strengthens this assumption slightly to also bound the Hessian over any mini-batch . Assumption 5 bounds the magnitude of the partial derivatives with respect to embeddings . This embedding gradient bound is necessary to ensure convergence in the presence of embedding compression error ( see appendix for details ) .

Summary: This work studies the problem of communication-efficient training on vertically partitioned data and it is relevant to the conference. The paper provides the theoretical analysis of the effect message compression has on distributed training over vertically partitioned data, and prove convergence of non-convex objectives to a fixed point at a rate of O($1/\sqrt{T}$) when the compression error is bounded over the course of training. Experimental validation is provided to show compression can reduce communication by over 90% without a significant decrease in accuracy over VFL without compression.