Practical and Private Heterogeneous Federated Learning

1 INTRODUCTION . Heterogeneous federated learning ( HFL ) ( Li & Wang , 2019 ; Chang et al. , 2019 ) , as a promising variant of federated learning ( FL ) , enables clients equipped with different computation and communication capabilities to collaboratively train their own customized models that may differ in size , numerical precision or structure ( Lin et al. , 2020 ) . In particular , clients share the knowledge of models via their predictions on auxiliary datasets , such as unlabeled problem domain datasets ( ChoquetteChoo et al. , 2021 ) and public non-problem domain datasets ( Li & Wang , 2019 ; Lin et al. , 2020 ) . This flexible approach facilitates customized FL-driven services in areas like Healthcare and Finance ( Kairouz et al. , 2019 ) , while solving the intellectual property concerns of FL models ( Atli et al. , 2020 ) . However , HFL suffers from two major limitations : ( 1 ) The assumption of auxiliary datasets may be unrealistic for many data-critical scenarios ( Zhu et al. , 2021 ) . For example , in Healthcare applications , task-related auxiliary datasets that contain patients ’ sensitive information are usually difficult to obtain due to current strict regulations like General Data Protection Regulation . ( 2 ) Sharing predictions may still leak the privacy of local data ( Papernot et al. , 2017 ) . Several works have demonstrated that given the black-box access to a trained model , adversaries can infer membership ( Salem et al. , 2019 ) and attribute information ( Ganju et al. , 2018 ) of the target sample , and even can reconstruct the original training data ( Yang et al. , 2019 ) . Therefore , to promote the deployment of HFL in real-world applications , it is crucial to solve the above two problems . To the best of our knowledge , in HFL the relaxation of the auxiliary dataset assumption has not been explored before . Specifically , it is challenging to achieve collaborative training under heterogeneous models when there is no an auxiliary dataset as a medium for the model knowledge transfer ( Li & Wang , 2019 ) . On the other hand , to mitigate the above privacy risks , a natural solution is to integrate advanced secure prediction protocols , such as CrypTFlow2 ( Rathee et al. , 2020 ) , CryptGPU ( Tan et al. , 2021 ) , and HE-transformer ( Boemer et al. , 2019b ) . These schemes can protect the private information during the model knowledge transfer by utilizing homomorphic encryption ( HE ) ( Gentry , 2009 ) , garbled circuit ( GC ) ( Yao , 1986 ) or oblivious transfer ( OT ) ( Naor & Pinkas , 2001 ) techniques ( refer to Appendix D.2 for more details ) . Unfortunately , such methods add huge computation and communication overhead due to the use of heavy cryptographic primitives . For instance , ChoquetteChoo et al . ( 2021 ) recently proposed CaPC , the first private collaborative learning scheme based on HE-transformer ( Boemer et al. , 2019b ) supporting heterogeneous models , which can be directly extended to HFL . As mentioned above , this work still suffers from efficiency issues , inherited from prior secure prediction protocols . Moreover , their scheme is implemented by the interaction between clients1 , however in real-world applications , clients ( e.g. , mobile devices ) generally can not establish direct communication channels with others ( Bonawitz et al. , 2017 ) . Therefore , the challenge here is how to efficiently implement secure prediction protocols in the realistic HFL setting . In this work , to approach the above challenges , we develop PrivHFL , a general and practical framework for privacy-preserving HFL . First , PrivHFL relaxes the assumption of dependence on auxiliary datasets and designs a simple but effective dataset expansion method only with clients ’ private datasets . To this end , we instantiate it by leveraging mixup ( Zhang et al. , 2018 ) that is originally a regularization technique to improve generalization , and also present some exploration with other data augmentation methods like cutout ( DeVries & Taylor , 2017 ) and cutmix ( Yun et al. , 2019 ) . The key idea is that the expanded data could provide good coverage of natural dataset distributions and hence could be used as an effective medium for transferring model knowledge . Second , to securely and efficiently evaluate HFL , we leverage the lightweight additive secret sharing technique ( Demmler et al. , 2015 ) to construct customized secure prediction protocols from scratch in a practical setting where there is no direct communication between clients . Our gains mainly come from the improvement in communication and computation through the elimination of costly HE and GC protocols . Moreover , in contrast to prior works that evaluate cryptographic protocols in CPUs , PrivHFL converts complex cryptographic operations to simple computations on large blocks of data , which are friendly with GPUs and can be processed by highly-optimized CUDA kernels ( Tan et al. , 2021 ) . As a result , PrivHFL is suitable for the batch prediction ( i.e. , performing multiple predictions at the same time ) with lower amortized cost . We evaluate the designed protocol on GPUs and CPUs , and the results show that our GPU-based protocol is up to 10× faster than its CPU analog . Our key contributions can be summarized as follows : • We introduce a practical HFL framework , which is independent on any auxiliary datasets while provably providing comprehensive privacy protection . • We design a simple yet effective dataset expansion method to promote the sharing of model knowledge , and construct customized cryptographic protocols for secure prediction . • Extensive experiments on SVHN , CIFAR10 , Tiny ImageNet ( including IID and Non-IID settings ) and various heterogeneous models demonstrate that PrivHFL outperforms prior art up to two orders of magnitude in efficiency and realizes roughly 10 % accuracy gains . 2 BACKGROUND . Before introducing PrivHFL , we first describe the heterogeneous federated learning and the threat model , and then review the cryptographic primitives that are required to understand our work . 2.1 HETEROGENEOUS FEDERATED LEARNING . In HFL ( Li & Wang , 2019 ; Choquette-Choo et al. , 2021 ) , the clients independently design their own unique models , but due to the model heterogeneity , they can not directly share model parameters with each other . Instead , they learn the knowledge of other models via the predictions on a task-related auxiliary dataset , where a server routes messages between the clients since they generally can not establish direct communication channels with others ( Bonawitz et al. , 2017 ; Bell et al. , 2020 ) . Specifically , clients first train local models with their own private datasets . Then , each client performs prediction on the auxiliary dataset based on the local model and sends the prediction results to the server to aggregate . Later , the server broadcasts aggregated results to clients , which will retrain local models based on the auxiliary dataset and received predictions . The whole process is 1As shown in C.5 , by carefully designing protocols , CaPC can be extended to the communication-limited setting but at the cost of increased communication overhead . iterative until each local model meets the pre-defined accuracy requirement . Details can be referred to Figure 9 and Appendix A . 2.2 THREAT MODEL . We work in an honest-but-curious adversary setting ( Goldreich , 2009 ) , where each entity ( including the clients and the server ) strictly follows the specification of the designed protocol but attempts to infer more knowledge about other clients ’ private information such as model parameters and private datasets . Moreover , to maintain the reputation and provide more services , the server does not collude with any clients . Formally , an attacker either corrupts the server or a subset of clients but not both . This setting is reasonable and has been widely instantiated in previous works ( Phong et al. , 2018 ; Sun & Lyu , 2021 ; Choquette-Choo et al. , 2021 ) . 2.3 CRYPTOGRAPHIC PRIMITIVES . Additive secret sharing . We adopt lightweight 2-out-of-2 additive secret sharing over the ring ZL ( Demmler et al. , 2015 ) as the cryptographic building block . We let Share ( x ) denote the sharing algorithm that takes as input x in ZL and outputs random sampled shares [ x ] 0 , [ x ] 1 with the constraint x = [ x ] 0 + [ x ] 1 in ZL . Arithmetic operations can be implemented in the sharing form as shown in Appendix C.4.1 . The reconstruction algorithm Recon ( [ x ] 0 , [ x ] 1 ) takes as input the two shares and outputs x = [ x ] 0 + [ x ] 1 in ZL . The security of the additive secret sharing protocol guarantees that given a share [ x ] 0 or [ x ] 1 , the value x is perfectly hidden . Pseudorandom generator . A Pseudorandom Generator ( PRG ) takes as input a uniformly random seed and a security parameter κ , and outputs a long pseudorandom string . The security of PRG ensures that the output is indistinguishable from the uniform distribution . In PrivHFL , PRGs enable two parties to generate same ( pseudo- ) random numbers without communication . We instantiate PRG with the technique from ( Matyas , 1985 ) and the seed can be generated by the Diffie-Hellman Key Agreement protocol ( Diffie & Hellman , 1976 ) . Details can be referred to Appendix C.4.2 . 3 THE PRIVHFL PROTOCOL . In this section , we introduce the high-level view of PrivHFL , followed by describing in detail our dataset expansion method and secure prediction scheme . 3.1 HIGH-LEVEL VIEW OF PRIVHFL . Synthetic Pool Sampling Query Data … Data Expansion Private Data Local Model Heterogeneous Models prediction label label Local Training Query-data Generation Secure Querying Re-training Local Model prediction Figure 1 : High-level view of PrivHFL PrivHFL follows prior HFL works ( Li & Wang , 2019 ) and iteratively optimizes the clients ’ models . Each client in PrivHFL can play the role of the querying party and the answering party at the same time , and without loss of generality , we denote them as PQ and PA , respectively . As shown in Figure 1 , in each iteration , each PQ performs four-phase operations with other PA , i.e. , local training , query-data generation , secure querying , and re-training . In detail , clients first train the local model on their own private datasets , which is the baseline any future improvements will be compared with . After that , by utilizing our dataset expansion method , each PQ can generate the query data to query other PA ( C fraction of all clients ) for prediction results . To protect the privacy of query samples , predictions and model parameters , clients adopt our secure prediction protocol and conduct collaborative querying in the ciphertext form . To the end , each client can retrain the local model based on the private dataset , as well as the query samples and corresponding prediction results . Algorithm 1 in Appendix C.1 gives the detailed description of PrivHFL . 3.2 QUERY-DATA GENERATION . To relax the assumption of auxiliary datasets , we design and instantiate an effective dataset expansion method , inspired by the success of mixup ( Zhang et al. , 2018 ) in improving model generalization and the efficiency of knowledge distillation ( Wang et al. , 2020 ) . Specifically , we repurpose mixup to construct a big synthesized pool on the small private dataset , which could provide a good coverage of the manifold of natural samples . Given any two private samples xi and xj , we generate multiple synthetic query samples by a convex combination with different coefficients λ as follows : x̃i , j ( λ ) = λ · xi + ( 1− λ ) · xj . ( 1 ) Empirically , we set λ ∈ [ 0.1 , 0.9 ] with an interval of 0.1 to generate more diverse synthetic images . We also explore the influence of different λ values in Appendix B.3 and Table 5 . This simple method can exponentially expand the size of initial dataset and hence provide more candidate samples for query . Following CaPC ( Choquette-Choo et al. , 2021 ) , we use random sampling and active learning strategies ( Tong & Koller , 2001 ) in Appendix C.3 to select informative samples from the synthesized pool . Note that an alternative solution is to directly use private datasets to query like knowledge distillation ( Hinton et al. , 2015 ) . We compare against this method in Section 4 . Extensions of mixup-based method . Dataset expansion based on private samples is a universal and modular method , therefore it can be readily extended with techniques in data augmentation literature . For example , one may replace the mixup-based dataset expansion in PrivHFL with recent methods such as cutout ( DeVries & Taylor , 2017 ) and cutmix ( Yun et al. , 2019 ) . We present some exploration and experiments with these extensions in Appendix B.1 and Figure 11 .

Summary: PrivHFL is a protocol for collaborative learning that does not reveal the private data to the server or answering parties $P_A$ if they operate as honest-but-curious entities. PrivHFL tackles the bottleneck of private inference by building a new protocol from scratch that can run on GPU. It is not assumed that parties have new data or that there is a public pool of data to run inference on but the additional query data for knowledge transfer is obtained by augmenting the private data with MixUp.