FEVERLESS: Fast and Secure Vertical Federated Learning based on XGBoost for Decentralized Labels

1 INTRODUCTION . Traditional centralized deep learning models , demanding to collect a considerable amount of clients ’ data to maintain high accuracy , to some degree , may increase the risk of data breaches . Data may not be easily shared among different entities due to privacy regulations and policies . To tackle this “ Data Island ” problem ( Yang et al . ( 2019a ) ) , Google proposed Federated Learning ( FL ) ( McMahan et al . ( 2017 ) ) to allow multiple clients to train a global model without sharing private data . The basic paradigm of FL is that all clients train local models with their own data , and then the information of local models , e.g. , gradients , may be exchanged to produce a global model . Based on different types of data partition ( Yang et al . ( 2019a ) ) , FL can be mainly categorized into Horizontal Federated Learning ( HFL ) and Vertical Federated Learning ( VFL ) . The former focuses on training with horizontally partitioned data where clients share the same feature space but differing in data index set . Several research works ( Shokri & Shmatikov ( 2015 ) ; Orekondy et al . ( 2019 ) ; Geiping et al . ( 2020 ) ; Li & Han ( 2019 ) ) have found that training data of HFL is still at high risk of leakage although private data is kept locally . Other studies ( Phong et al . ( 2018 ) ; Truex et al . ( 2019 ) ; Xu et al . ( 2019 ) ; Zhang et al . ( 2020 ) ; Zhu et al . ( 2020 ) ) have been dedicated to enhancing the security of HFL . On the contrary , VFL is mainly applied in the scenario of training with vertically partitioned data ( Wu et al . ( 2020 ) ; Cheng et al . ( 2021 ) ) where clients share the same data index set but differing in feature space . In this paper , our principal focus is to achieve privacy-preserving training on VFL . To best of our knowledge , many existing studies ( Hardy et al . ( 2017 ) ; Nock et al . ( 2018 ) ; Liu et al . ( 2020 ) ; Yang et al . ( 2019b ) ; Cheng et al . ( 2021 ) ; Chen & Guestrin ( 2016 ) ; Wu et al . ( 2020 ) ) have proposed innovative approaches to prevent private information breaches in the context of VFL . Specifically , ( Hardy et al . ( 2017 ) ) introduced encryption-based privacy-preserving logistic regression to safeguard the information of data indexes . ( Nock et al . ( 2018 ) ) gave a comprehensive discussion on 1Code is available at : https : //github.com/feverless111/vfl the impact of ID resolution . ( Yang et al . ( 2019b ) ) introduced a scheme without using a coordinator for a limited number of clients . Recently , ( Liu et al . ( 2020 ) ) proposed an asymmetrically VFL scheme for logistic regression tackling privacy concerns on ID alignment . Unlike the training models used in the aforementioned works , XGBoost ( Chen & Guestrin ( 2016 ) ) , which is one of the most popular models applied in VFL , can provide better interpretation , easier parameter tuning , and faster execution than deep learning in tabular data training ( Goodfellow et al . ( 2016 ) ; LeCun et al . ( 2015 ) ) . These practical features and advantages draw academia and industry ’ s attention to the research on XGBoost , especially in the privacy-preserving context . ( Wu et al . ( 2020 ) ) introduced an approach for tree-based model training through a hybrid method composing homomorphic encryption and secure Multi-Party Computation ( MPC ) ( Goldreich ( 1998 ) ; Bonawitz et al . ( 2017 ) ) . After that , ( Cheng et al . ( 2021 ) ) proposed a similar system to train XGBoost ( Chen & Guestrin ( 2016 ) ) securely over vertically partitioned data by using Additively Homomorphic Encryption ( AHE ) . By applying Differential Privacy ( DP ) ( Dwork ( 2008 ) ) , ( Tian et al . ( 2020 ) ) designed a VFL system to train GBDT without the need of encryption/decryption . However , most of the above solutions based on AHE and MPC do not scale well in terms of efficiency on training XGBoost . Beyond that , all the existing schemes basically assume that training labels are managed and processed by a sole client . In practice , a VFL scheme supporting distributed labels is necessary . For instance , multiple hospitals , clinics and health centers currently may be set to COVID-19 test spots and aim to train a model , e.g. , XGBoost , to predict with good interpretation if citizens ( living in various locations ) are infected based on their health records and symptoms . In this context , the labels ( i.e. , the test results ) are likely distributed among different health authorities - even targeting to the same group of patients , and feature space is vertically portioned . For example , a cardiac hospital only maintains heart data for the patients , while a psychiatric center holds the mental records , in which both authorities may collect and manage each of its registered patient ’ s label locally . Another common scenario could be in the financial sector where multiple bank branches and e-commerce companies prefer to build a global model to predict if their customers may pay some service ( e.g. , car loan ) on time . The banks have part of features about the customers ( e.g. , account balance , funding in-and-out records ) , while the companies may obtain other features ( e.g. , payment preference ) . Since the customers may get the same service , e.g. , loan , from different institutions , it is clear that labels must be distributed rather than centralized . In addition to efficiency and functionality aspects , one may also consider capturing stronger security for VFL . Training an XGBoost usually should involve the computation of first and second-order derivatives of the loss function ( note gradients and hessians contain labels ’ information ) , and the aggregation of them is required in each round . In the context where the labels are held by different clients , if the gradients and hessians are transmitted in the form of plaintexts and the summations of them are known to an aggregator ( whom could be one of the clients engages in training ) , inference and differential attacks ( Appendix C ) will be easily conducted by the aggregator , resulting in information leakage . To tackle these problems , we propose a fast and secure VFL protocol , FEVERLESS , to train XGBoost ( Appendix B.1 ) on distributed labels without disclosing both feature and label information . In our design , the privacy protection is guaranteed by secure aggregation ( based on a masking scheme ) and Global Differential Privacy ( GDP ) ( Appendix B.6 ) . We leverage masking instead of heavy-cost multiparty computation and we guarantee a “ perfect secrecy ” level for the masked data . In GDP , we use Verifiable Random Function ( VRF ) ( Appendix B.5 ) to select a noise leader per round ( who can not be predicted and pre-compromised in advance ) to aggregate noise from “ selected ” clients , which significantly maintains model accuracy . Our contributions can be summarized as follows . ( 1 ) We define VFL in a more practical scenario where training labels are distributed over multiple clients . Beyond that , we develop FEVERLESS to train XGBoost securely and efficiently with the elegant combination of secure aggregation technique ( based on Diffie-Hellman ( DH ) key exchange ( Appendix B.2 ) and Key Derivation Function ( KDF ) ( Appendix B.4 ) ) and GDP . ( 2 ) We give a comprehensive security analysis to demonstrate that FEVERLESS is able to safeguard labels and features privacy in the semi-honest setting , but also maintain the robustness even for the case where n− 2 out of n clients commit collusion . ( 3 ) We implement FEVERLESS and perform training time and accuracy evaluation on different realworld datasets . The empirical results show that FEVERLESS can maintain efficiency and accuracy simultaneously , and its performance is comparable to the baseline - a ” pure ” XGBoost without using any encryption and differential privacy . Specifically , training the credit card and bank marketing datasets just takes 1 % and 6.5 % more runtime than the baseline and meanwhile , the accuracy is only lower than that of the baseline by 0.9 % and 3.21 % , respectively2 . 2 PROBLEM FORMULATION . 2.1 SYSTEM MODEL . Before proceeding , we give some assumptions on our model . We suppose that a private set intersection ( Kolesnikov et al . ( 2017 ) ; Pinkas et al . ( 2014 ) ) has been used to align data IDs before the training starts , so that each client shares the same data index space I . But the names of features are not allowed to share among clients . As for the information of label distribution ( indexes indicating a label belongs to which client , e.g. , the label of i-th data instance is held by client A ) , we will consider the following conditions : ( 1 ) this information is revealed to the public in advance ; or ( 2 ) the information is not allowed to publish but the training can still be accomplished ( with extra cost ) . We also consider that the training is conducted on a dataset with m samples composing with feature space X = { x1 , · · · , xm } , each containing f features , and label set Y = { y1 , · · · , ym } . Besides , features { X ( c ) j | j ∈ { 1 , · · · , f } } and labels { y ( c ) i | i ∈ { 1 , · · · , m } } are held among n clients where each client has at least one feature and one label . X ( c ) j and y ( c ) i refer to j-th feature and i-th label owned by c-th client , respectively . Considering a practical scenario wherein training labels are distributed among clients , we propose a new variant of VFL , named VFL over Distributed Labels ( DL-VFL ) . The concrete definition is given as follows . Definition 1 ( DL-VFL ) . Given a training set with m data samples consisting of feature space X , label space Y , index space I and clients set C , we have : X c ∩ X c ′ = ∅ , Yc ∩ Yc ′ = ∅ , Ic = Ic ′ , ∀c , c ′ ∈ C , c 6= c ′ . ( 1 ) A client c participating DL-VFL shares the same sample ID space I with the corresponding labels , where a single label belongs to only one client . And different clients hold the subset of X sampled from feature space . To achieve privacy-preserving XGBoost training , we further define two roles . Definition 2 ( Source client ) . A source client with split candidates wants to compute the corresponding Lsplit based on Eq. ( 4 ) . But some labels are missing so that ∑ gi and ∑ hi are unable to derive . For the case that a source client does not hold all labels in the current split candidates , we propose a solution based on secure aggregation and global differential privacy to help the source client to compute Lsplit while safeguarding other clients ’ privacy . We consider the two conditions regarding if label distribution is publicly known . We find that if we keep label distribution hidden , we will take extra communication overhead to perform training . The detailed explanation is given in Appendix F. Note each client may have a chance to act as a source client because all the labels are distributed , where the source client leads the Lsplit computation , and clients provide missing label values to the source client . To achieve GDP , we define noise leader who is selected fairly and randomly from all clients ( except for the source client ) - preventing clients from being compromised beforehand . Definition 3 ( Noise leader ) . By using VRF , a noise leader is responsible for generating the maximum leader score , aggregating differentially private noise from a portion of clients and adding the noise to the gradients and hessians . Note we summary the main notations in Table 1 ( see Appendix A ) .

Summary: This paper studies vertical federated learning for fast and secure XGBoost training where labels are distributed among multiple parties.  Most previous works focus on labels are centralized in one party and adapt cryptography such as homomorphic encryption, multi-party computation, and differential privacy to protect the data and label privacy. This paper instead assumes decentralized labels and combines existing secure aggregation and global differential privacy to safeguard the data and label privacy.