Practical and Private Heterogeneous Federated Learning

1 INTRODUCTION . Heterogeneous federated learning ( HFL ) ( Li & Wang , 2019 ; Chang et al. , 2019 ) , as a promising variant of federated learning ( FL ) , enables clients equipped with different computation and communication capabilities to collaboratively train their own customized models that may differ in size , numerical precision or structure ( Lin et al. , 2020 ) . In particular , clients share the knowledge of models via their predictions on auxiliary datasets , such as unlabeled problem domain datasets ( ChoquetteChoo et al. , 2021 ) and public non-problem domain datasets ( Li & Wang , 2019 ; Lin et al. , 2020 ) . This flexible approach facilitates customized FL-driven services in areas like Healthcare and Finance ( Kairouz et al. , 2019 ) , while solving the intellectual property concerns of FL models ( Atli et al. , 2020 ) . However , HFL suffers from two major limitations : ( 1 ) The assumption of auxiliary datasets may be unrealistic for many data-critical scenarios ( Zhu et al. , 2021 ) . For example , in Healthcare applications , task-related auxiliary datasets that contain patients ’ sensitive information are usually difficult to obtain due to current strict regulations like General Data Protection Regulation . ( 2 ) Sharing predictions may still leak the privacy of local data ( Papernot et al. , 2017 ) . Several works have demonstrated that given the black-box access to a trained model , adversaries can infer membership ( Salem et al. , 2019 ) and attribute information ( Ganju et al. , 2018 ) of the target sample , and even can reconstruct the original training data ( Yang et al. , 2019 ) . Therefore , to promote the deployment of HFL in real-world applications , it is crucial to solve the above two problems . To the best of our knowledge , in HFL the relaxation of the auxiliary dataset assumption has not been explored before . Specifically , it is challenging to achieve collaborative training under heterogeneous models when there is no an auxiliary dataset as a medium for the model knowledge transfer ( Li & Wang , 2019 ) . On the other hand , to mitigate the above privacy risks , a natural solution is to integrate advanced secure prediction protocols , such as CrypTFlow2 ( Rathee et al. , 2020 ) , CryptGPU ( Tan et al. , 2021 ) , and HE-transformer ( Boemer et al. , 2019b ) . These schemes can protect the private information during the model knowledge transfer by utilizing homomorphic encryption ( HE ) ( Gentry , 2009 ) , garbled circuit ( GC ) ( Yao , 1986 ) or oblivious transfer ( OT ) ( Naor & Pinkas , 2001 ) techniques ( refer to Appendix D.2 for more details ) . Unfortunately , such methods add huge computation and communication overhead due to the use of heavy cryptographic primitives . For instance , ChoquetteChoo et al . ( 2021 ) recently proposed CaPC , the first private collaborative learning scheme based on HE-transformer ( Boemer et al. , 2019b ) supporting heterogeneous models , which can be directly extended to HFL . As mentioned above , this work still suffers from efficiency issues , inherited from prior secure prediction protocols . Moreover , their scheme is implemented by the interaction between clients1 , however in real-world applications , clients ( e.g. , mobile devices ) generally can not establish direct communication channels with others ( Bonawitz et al. , 2017 ) . Therefore , the challenge here is how to efficiently implement secure prediction protocols in the realistic HFL setting . In this work , to approach the above challenges , we develop PrivHFL , a general and practical framework for privacy-preserving HFL . First , PrivHFL relaxes the assumption of dependence on auxiliary datasets and designs a simple but effective dataset expansion method only with clients ’ private datasets . To this end , we instantiate it by leveraging mixup ( Zhang et al. , 2018 ) that is originally a regularization technique to improve generalization , and also present some exploration with other data augmentation methods like cutout ( DeVries & Taylor , 2017 ) and cutmix ( Yun et al. , 2019 ) . The key idea is that the expanded data could provide good coverage of natural dataset distributions and hence could be used as an effective medium for transferring model knowledge . Second , to securely and efficiently evaluate HFL , we leverage the lightweight additive secret sharing technique ( Demmler et al. , 2015 ) to construct customized secure prediction protocols from scratch in a practical setting where there is no direct communication between clients . Our gains mainly come from the improvement in communication and computation through the elimination of costly HE and GC protocols . Moreover , in contrast to prior works that evaluate cryptographic protocols in CPUs , PrivHFL converts complex cryptographic operations to simple computations on large blocks of data , which are friendly with GPUs and can be processed by highly-optimized CUDA kernels ( Tan et al. , 2021 ) . As a result , PrivHFL is suitable for the batch prediction ( i.e. , performing multiple predictions at the same time ) with lower amortized cost . We evaluate the designed protocol on GPUs and CPUs , and the results show that our GPU-based protocol is up to 10× faster than its CPU analog . Our key contributions can be summarized as follows : • We introduce a practical HFL framework , which is independent on any auxiliary datasets while provably providing comprehensive privacy protection . • We design a simple yet effective dataset expansion method to promote the sharing of model knowledge , and construct customized cryptographic protocols for secure prediction . • Extensive experiments on SVHN , CIFAR10 , Tiny ImageNet ( including IID and Non-IID settings ) and various heterogeneous models demonstrate that PrivHFL outperforms prior art up to two orders of magnitude in efficiency and realizes roughly 10 % accuracy gains . 2 BACKGROUND . Before introducing PrivHFL , we first describe the heterogeneous federated learning and the threat model , and then review the cryptographic primitives that are required to understand our work . 2.1 HETEROGENEOUS FEDERATED LEARNING . In HFL ( Li & Wang , 2019 ; Choquette-Choo et al. , 2021 ) , the clients independently design their own unique models , but due to the model heterogeneity , they can not directly share model parameters with each other . Instead , they learn the knowledge of other models via the predictions on a task-related auxiliary dataset , where a server routes messages between the clients since they generally can not establish direct communication channels with others ( Bonawitz et al. , 2017 ; Bell et al. , 2020 ) . Specifically , clients first train local models with their own private datasets . Then , each client performs prediction on the auxiliary dataset based on the local model and sends the prediction results to the server to aggregate . Later , the server broadcasts aggregated results to clients , which will retrain local models based on the auxiliary dataset and received predictions . The whole process is 1As shown in C.5 , by carefully designing protocols , CaPC can be extended to the communication-limited setting but at the cost of increased communication overhead . iterative until each local model meets the pre-defined accuracy requirement . Details can be referred to Figure 9 and Appendix A . 2.2 THREAT MODEL . We work in an honest-but-curious adversary setting ( Goldreich , 2009 ) , where each entity ( including the clients and the server ) strictly follows the specification of the designed protocol but attempts to infer more knowledge about other clients ’ private information such as model parameters and private datasets . Moreover , to maintain the reputation and provide more services , the server does not collude with any clients . Formally , an attacker either corrupts the server or a subset of clients but not both . This setting is reasonable and has been widely instantiated in previous works ( Phong et al. , 2018 ; Sun & Lyu , 2021 ; Choquette-Choo et al. , 2021 ) . 2.3 CRYPTOGRAPHIC PRIMITIVES . Additive secret sharing . We adopt lightweight 2-out-of-2 additive secret sharing over the ring ZL ( Demmler et al. , 2015 ) as the cryptographic building block . We let Share ( x ) denote the sharing algorithm that takes as input x in ZL and outputs random sampled shares [ x ] 0 , [ x ] 1 with the constraint x = [ x ] 0 + [ x ] 1 in ZL . Arithmetic operations can be implemented in the sharing form as shown in Appendix C.4.1 . The reconstruction algorithm Recon ( [ x ] 0 , [ x ] 1 ) takes as input the two shares and outputs x = [ x ] 0 + [ x ] 1 in ZL . The security of the additive secret sharing protocol guarantees that given a share [ x ] 0 or [ x ] 1 , the value x is perfectly hidden . Pseudorandom generator . A Pseudorandom Generator ( PRG ) takes as input a uniformly random seed and a security parameter κ , and outputs a long pseudorandom string . The security of PRG ensures that the output is indistinguishable from the uniform distribution . In PrivHFL , PRGs enable two parties to generate same ( pseudo- ) random numbers without communication . We instantiate PRG with the technique from ( Matyas , 1985 ) and the seed can be generated by the Diffie-Hellman Key Agreement protocol ( Diffie & Hellman , 1976 ) . Details can be referred to Appendix C.4.2 . 3 THE PRIVHFL PROTOCOL . In this section , we introduce the high-level view of PrivHFL , followed by describing in detail our dataset expansion method and secure prediction scheme . 3.1 HIGH-LEVEL VIEW OF PRIVHFL . Synthetic Pool Sampling Query Data … Data Expansion Private Data Local Model Heterogeneous Models prediction label label Local Training Query-data Generation Secure Querying Re-training Local Model prediction Figure 1 : High-level view of PrivHFL PrivHFL follows prior HFL works ( Li & Wang , 2019 ) and iteratively optimizes the clients ’ models . Each client in PrivHFL can play the role of the querying party and the answering party at the same time , and without loss of generality , we denote them as PQ and PA , respectively . As shown in Figure 1 , in each iteration , each PQ performs four-phase operations with other PA , i.e. , local training , query-data generation , secure querying , and re-training . In detail , clients first train the local model on their own private datasets , which is the baseline any future improvements will be compared with . After that , by utilizing our dataset expansion method , each PQ can generate the query data to query other PA ( C fraction of all clients ) for prediction results . To protect the privacy of query samples , predictions and model parameters , clients adopt our secure prediction protocol and conduct collaborative querying in the ciphertext form . To the end , each client can retrain the local model based on the private dataset , as well as the query samples and corresponding prediction results . Algorithm 1 in Appendix C.1 gives the detailed description of PrivHFL . 3.2 QUERY-DATA GENERATION . To relax the assumption of auxiliary datasets , we design and instantiate an effective dataset expansion method , inspired by the success of mixup ( Zhang et al. , 2018 ) in improving model generalization and the efficiency of knowledge distillation ( Wang et al. , 2020 ) . Specifically , we repurpose mixup to construct a big synthesized pool on the small private dataset , which could provide a good coverage of the manifold of natural samples . Given any two private samples xi and xj , we generate multiple synthetic query samples by a convex combination with different coefficients λ as follows : x̃i , j ( λ ) = λ · xi + ( 1− λ ) · xj . ( 1 ) Empirically , we set λ ∈ [ 0.1 , 0.9 ] with an interval of 0.1 to generate more diverse synthetic images . We also explore the influence of different λ values in Appendix B.3 and Table 5 . This simple method can exponentially expand the size of initial dataset and hence provide more candidate samples for query . Following CaPC ( Choquette-Choo et al. , 2021 ) , we use random sampling and active learning strategies ( Tong & Koller , 2001 ) in Appendix C.3 to select informative samples from the synthesized pool . Note that an alternative solution is to directly use private datasets to query like knowledge distillation ( Hinton et al. , 2015 ) . We compare against this method in Section 4 . Extensions of mixup-based method . Dataset expansion based on private samples is a universal and modular method , therefore it can be readily extended with techniques in data augmentation literature . For example , one may replace the mixup-based dataset expansion in PrivHFL with recent methods such as cutout ( DeVries & Taylor , 2017 ) and cutmix ( Yun et al. , 2019 ) . We present some exploration and experiments with these extensions in Appendix B.1 and Figure 11 .

Summary: This paper introduces a heterogeneous federated learning framework without requiring public datasets by designing a dataset expansion method and constructing cryptographic protocols for secure prediction. The main strength of this paper is regarding the experimental results. The authors have provided experiments on different datasets, heterogeneous local models, various degree of non-IID-ness across clients, and showed the runtime of three steps in their proposed secure querying protocol.