FEVERLESS: Fast and Secure Vertical Federated Learning based on XGBoost for Decentralized Labels

1 INTRODUCTION . Traditional centralized deep learning models , demanding to collect a considerable amount of clients ’ data to maintain high accuracy , to some degree , may increase the risk of data breaches . Data may not be easily shared among different entities due to privacy regulations and policies . To tackle this “ Data Island ” problem ( Yang et al . ( 2019a ) ) , Google proposed Federated Learning ( FL ) ( McMahan et al . ( 2017 ) ) to allow multiple clients to train a global model without sharing private data . The basic paradigm of FL is that all clients train local models with their own data , and then the information of local models , e.g. , gradients , may be exchanged to produce a global model . Based on different types of data partition ( Yang et al . ( 2019a ) ) , FL can be mainly categorized into Horizontal Federated Learning ( HFL ) and Vertical Federated Learning ( VFL ) . The former focuses on training with horizontally partitioned data where clients share the same feature space but differing in data index set . Several research works ( Shokri & Shmatikov ( 2015 ) ; Orekondy et al . ( 2019 ) ; Geiping et al . ( 2020 ) ; Li & Han ( 2019 ) ) have found that training data of HFL is still at high risk of leakage although private data is kept locally . Other studies ( Phong et al . ( 2018 ) ; Truex et al . ( 2019 ) ; Xu et al . ( 2019 ) ; Zhang et al . ( 2020 ) ; Zhu et al . ( 2020 ) ) have been dedicated to enhancing the security of HFL . On the contrary , VFL is mainly applied in the scenario of training with vertically partitioned data ( Wu et al . ( 2020 ) ; Cheng et al . ( 2021 ) ) where clients share the same data index set but differing in feature space . In this paper , our principal focus is to achieve privacy-preserving training on VFL . To best of our knowledge , many existing studies ( Hardy et al . ( 2017 ) ; Nock et al . ( 2018 ) ; Liu et al . ( 2020 ) ; Yang et al . ( 2019b ) ; Cheng et al . ( 2021 ) ; Chen & Guestrin ( 2016 ) ; Wu et al . ( 2020 ) ) have proposed innovative approaches to prevent private information breaches in the context of VFL . Specifically , ( Hardy et al . ( 2017 ) ) introduced encryption-based privacy-preserving logistic regression to safeguard the information of data indexes . ( Nock et al . ( 2018 ) ) gave a comprehensive discussion on 1Code is available at : https : //github.com/feverless111/vfl the impact of ID resolution . ( Yang et al . ( 2019b ) ) introduced a scheme without using a coordinator for a limited number of clients . Recently , ( Liu et al . ( 2020 ) ) proposed an asymmetrically VFL scheme for logistic regression tackling privacy concerns on ID alignment . Unlike the training models used in the aforementioned works , XGBoost ( Chen & Guestrin ( 2016 ) ) , which is one of the most popular models applied in VFL , can provide better interpretation , easier parameter tuning , and faster execution than deep learning in tabular data training ( Goodfellow et al . ( 2016 ) ; LeCun et al . ( 2015 ) ) . These practical features and advantages draw academia and industry ’ s attention to the research on XGBoost , especially in the privacy-preserving context . ( Wu et al . ( 2020 ) ) introduced an approach for tree-based model training through a hybrid method composing homomorphic encryption and secure Multi-Party Computation ( MPC ) ( Goldreich ( 1998 ) ; Bonawitz et al . ( 2017 ) ) . After that , ( Cheng et al . ( 2021 ) ) proposed a similar system to train XGBoost ( Chen & Guestrin ( 2016 ) ) securely over vertically partitioned data by using Additively Homomorphic Encryption ( AHE ) . By applying Differential Privacy ( DP ) ( Dwork ( 2008 ) ) , ( Tian et al . ( 2020 ) ) designed a VFL system to train GBDT without the need of encryption/decryption . However , most of the above solutions based on AHE and MPC do not scale well in terms of efficiency on training XGBoost . Beyond that , all the existing schemes basically assume that training labels are managed and processed by a sole client . In practice , a VFL scheme supporting distributed labels is necessary . For instance , multiple hospitals , clinics and health centers currently may be set to COVID-19 test spots and aim to train a model , e.g. , XGBoost , to predict with good interpretation if citizens ( living in various locations ) are infected based on their health records and symptoms . In this context , the labels ( i.e. , the test results ) are likely distributed among different health authorities - even targeting to the same group of patients , and feature space is vertically portioned . For example , a cardiac hospital only maintains heart data for the patients , while a psychiatric center holds the mental records , in which both authorities may collect and manage each of its registered patient ’ s label locally . Another common scenario could be in the financial sector where multiple bank branches and e-commerce companies prefer to build a global model to predict if their customers may pay some service ( e.g. , car loan ) on time . The banks have part of features about the customers ( e.g. , account balance , funding in-and-out records ) , while the companies may obtain other features ( e.g. , payment preference ) . Since the customers may get the same service , e.g. , loan , from different institutions , it is clear that labels must be distributed rather than centralized . In addition to efficiency and functionality aspects , one may also consider capturing stronger security for VFL . Training an XGBoost usually should involve the computation of first and second-order derivatives of the loss function ( note gradients and hessians contain labels ’ information ) , and the aggregation of them is required in each round . In the context where the labels are held by different clients , if the gradients and hessians are transmitted in the form of plaintexts and the summations of them are known to an aggregator ( whom could be one of the clients engages in training ) , inference and differential attacks ( Appendix C ) will be easily conducted by the aggregator , resulting in information leakage . To tackle these problems , we propose a fast and secure VFL protocol , FEVERLESS , to train XGBoost ( Appendix B.1 ) on distributed labels without disclosing both feature and label information . In our design , the privacy protection is guaranteed by secure aggregation ( based on a masking scheme ) and Global Differential Privacy ( GDP ) ( Appendix B.6 ) . We leverage masking instead of heavy-cost multiparty computation and we guarantee a “ perfect secrecy ” level for the masked data . In GDP , we use Verifiable Random Function ( VRF ) ( Appendix B.5 ) to select a noise leader per round ( who can not be predicted and pre-compromised in advance ) to aggregate noise from “ selected ” clients , which significantly maintains model accuracy . Our contributions can be summarized as follows . ( 1 ) We define VFL in a more practical scenario where training labels are distributed over multiple clients . Beyond that , we develop FEVERLESS to train XGBoost securely and efficiently with the elegant combination of secure aggregation technique ( based on Diffie-Hellman ( DH ) key exchange ( Appendix B.2 ) and Key Derivation Function ( KDF ) ( Appendix B.4 ) ) and GDP . ( 2 ) We give a comprehensive security analysis to demonstrate that FEVERLESS is able to safeguard labels and features privacy in the semi-honest setting , but also maintain the robustness even for the case where n− 2 out of n clients commit collusion . ( 3 ) We implement FEVERLESS and perform training time and accuracy evaluation on different realworld datasets . The empirical results show that FEVERLESS can maintain efficiency and accuracy simultaneously , and its performance is comparable to the baseline - a ” pure ” XGBoost without using any encryption and differential privacy . Specifically , training the credit card and bank marketing datasets just takes 1 % and 6.5 % more runtime than the baseline and meanwhile , the accuracy is only lower than that of the baseline by 0.9 % and 3.21 % , respectively2 . 2 PROBLEM FORMULATION . 2.1 SYSTEM MODEL . Before proceeding , we give some assumptions on our model . We suppose that a private set intersection ( Kolesnikov et al . ( 2017 ) ; Pinkas et al . ( 2014 ) ) has been used to align data IDs before the training starts , so that each client shares the same data index space I . But the names of features are not allowed to share among clients . As for the information of label distribution ( indexes indicating a label belongs to which client , e.g. , the label of i-th data instance is held by client A ) , we will consider the following conditions : ( 1 ) this information is revealed to the public in advance ; or ( 2 ) the information is not allowed to publish but the training can still be accomplished ( with extra cost ) . We also consider that the training is conducted on a dataset with m samples composing with feature space X = { x1 , · · · , xm } , each containing f features , and label set Y = { y1 , · · · , ym } . Besides , features { X ( c ) j | j ∈ { 1 , · · · , f } } and labels { y ( c ) i | i ∈ { 1 , · · · , m } } are held among n clients where each client has at least one feature and one label . X ( c ) j and y ( c ) i refer to j-th feature and i-th label owned by c-th client , respectively . Considering a practical scenario wherein training labels are distributed among clients , we propose a new variant of VFL , named VFL over Distributed Labels ( DL-VFL ) . The concrete definition is given as follows . Definition 1 ( DL-VFL ) . Given a training set with m data samples consisting of feature space X , label space Y , index space I and clients set C , we have : X c ∩ X c ′ = ∅ , Yc ∩ Yc ′ = ∅ , Ic = Ic ′ , ∀c , c ′ ∈ C , c 6= c ′ . ( 1 ) A client c participating DL-VFL shares the same sample ID space I with the corresponding labels , where a single label belongs to only one client . And different clients hold the subset of X sampled from feature space . To achieve privacy-preserving XGBoost training , we further define two roles . Definition 2 ( Source client ) . A source client with split candidates wants to compute the corresponding Lsplit based on Eq. ( 4 ) . But some labels are missing so that ∑ gi and ∑ hi are unable to derive . For the case that a source client does not hold all labels in the current split candidates , we propose a solution based on secure aggregation and global differential privacy to help the source client to compute Lsplit while safeguarding other clients ’ privacy . We consider the two conditions regarding if label distribution is publicly known . We find that if we keep label distribution hidden , we will take extra communication overhead to perform training . The detailed explanation is given in Appendix F. Note each client may have a chance to act as a source client because all the labels are distributed , where the source client leads the Lsplit computation , and clients provide missing label values to the source client . To achieve GDP , we define noise leader who is selected fairly and randomly from all clients ( except for the source client ) - preventing clients from being compromised beforehand . Definition 3 ( Noise leader ) . By using VRF , a noise leader is responsible for generating the maximum leader score , aggregating differentially private noise from a portion of clients and adding the noise to the gradients and hessians . Note we summary the main notations in Table 1 ( see Appendix A ) .

Summary: The paper introduces a novel setting for vertical federated learning in which labels are distributed among clients, and proposes a novel fast and secure protocol for this setting based on XGBoost . The protocol enables secure aggregation of gradients and hessians for XGBoost via (a) a masking scheme based on Diffie-Hellman key exchange and a key derivation function and (b) global differential privacy. Security analysis presented in the paper demonstrate that label/feature privacy is persevered even if n-2 clients collude. Finally, the efficacy of the proposed protocol is demonstrated via experimental evaluations on multiple datasets.