Practical and Private Heterogeneous Federated Learning
Authors: Hanxiao Chen
Meng Hao
Hongwei Li
Guangxiao Niu
Guowen Xu
Huawei Wang
Yuan Zhang
Tianwei Zhang
Venue: ICLR 2022 Submitted (2022)

Abstract: Heterogeneous federated learning (HFL) enables clients with different computation/communication capabilities to collaboratively train their own customized models, in which the knowledge of models is shared via clients' predictions on a public dataset. However, there are two major limitations: 1) The assumption of public datasets may be unrealistic for data-critical scenarios such as Healthcare and Finance. 2) HFL is vulnerable to various privacy violations since the samples and predictions are completely exposed to adversaries. In this work, we develop PrivHFL, a general and practical framework for privacy-preserving HFL. We bypass the limitations of public datasets by designing a simple yet effective dataset expansion method. The main insight is that expanded data could provide good coverage of natural distributions, which is conducive to the sharing of model knowledge. To further tackle the privacy issue, we exploit the lightweight additive secret sharing technique to construct a series of tailored cryptographic protocols for key building blocks such as secure prediction. Our protocols implement ciphertext operations through simple vectorized computations, which are friendly with GPUs and can be processed by highly-optimized CUDA kernels. Extensive evaluations demonstrate that PrivHFL outperforms prior art up to two orders of magnitude in efficiency and realizes significant accuracy gains on top of the stand-alone method.

Under review as a conference paper at ICLR 2022
PRACTICAL AND PRIVATE HETEROGENEOUS FEDER- ATED LEARNING
Anonymous authors Paper under double-bling review
ABSTRACT
Heterogeneous federated learning (HFL) enables clients with different compu- nation/communicatioo capabilities to collaboratively train their own customized models, which the knowledge of models is shared via clients' predictions on an auxiliary unlabeled dataset. However, there are two major limitations: I) The assumption of auxiliary datasets may be unrealistic for data-critical scenarios such as Healthcare and Finance. 2) HFL is vulnerable to various privacy viola- tions since the samples and predictions are completely exposed to adversaries. this work, we develop PrivHFL, general and practical framework for privacy- preserving HFL. We bypass the limitation of auxiliary datasets by designing a simple yet effective dataset expansion method. The main insight is that expanded data could provide good coverage of natural distributions, which is conducive to the sharing of model knowledge. To further tackle the privacy issue, we exploit the lightweight additive secret sharing technique to construct series of tailored cryp- tographic protocols for key building blocks, such secure prediction. Our pro- tocols implement ciphertext operations through simple vectorized computations, which are friendly with GPUs and can be processed by highly-optimized CUDA kernels. Extensive evaluations demonstrate that PrivHFL outperforms prior art up to two orders of magnitude efficiency and realizes significant accuracy gains on top of the stand-alone method.
INTRODUCTION
Heterogeneous federated learning (HFL) (Li & Wang 2019 Chang et al., 2019). as promising variant of federated learning (FL.), enables clients equipped with different computation and commu- nication capabilities to collaboratively train their own customized models that may differ in size, nu- merical precision or structure (Lin et al. 2220). particular, clients share the knowledge of models via their predictions on auxiliary datasets, such as unlabeled problem domain datasets (Choquette- Choo et al. 202T) and public non-problem domain datasets (Li & Wang 2019 Lin et al.) 2020). This flexible approach facilitates customized FL-driven services in areas like Healthcare and Fi- nance (Kairouz et al., 2019), while solving the intellectual property concerns of FL models (Atli al. 2020). However. HFL suffers from major limitations: (1) The assumption of auxiliary datasets may be unrealistic for many data-critical scenarios (Zhu et al. 2021). For example, in Healthcare applications, task-related auxiliary datasets that contain patients" sensitive information are usually difficult to obtain due to current strict regulations like General Data Protection Regula- tion. (2) Sharing predictions may still leak the privacy of local data (Papernol al. 2017). Several works have demonstrated that given the black-box access to a trained model, adversaries can infer membership (Salem et al. 2019) and attribute information (Ganju et al. 2018) of the target sample, and even can reconstruct the original training data YYang et al. 2019). Therefore, to promote the deployment of HFL in real-worle applications, is crucial to solve the above two problems.
To the best of our knowledge, in HFL the relaxation of the auxiliary dataset assumption has not been explored before. Specifically, it challenging to achieve collaborative training under heterogeneous auxiliary dataset Wang 2019). On the other hand, to mitigate the above privacy risks, a natural solution is to integrate advanced secure prediction protocols, such as Cryp FFoo2 (Rathee et al.) 2020), CryptGPU (Tan jet al. 202T), and HE-transformer (Boemer et al.) 2019b). These schemes can protect the private in- formation during the model knowledge transter by utilizing homomorphic encryption (HE) (Gentry)
models when there is
medium for the model knowledge transfer (Li &

---

Under review as a conference paper at ICLR 2022
2009). garbled circuit (GC) Yao 1986 or oblivious transfer (OT) (Naor & Pinkas 2001 techniques (refer to Appendix D.2 for more details). Unfortunately, such methods add huge computation and communication overhead due to the use of heavy cryptographic primitives. For instance, (Chqquetee- Choo et al. (202T) recently proposed CaPC, the first private collaborative learning scheme based on Hlsttrassormmrr (Boemer et al., 2019b) supporting heterogeneous models, which can be directly extended to HFL. As mentioned above, this work still suffers from efficiency issues, inherited from prior secure prediction protocols. Moreover, their scheme implemented by the interaction between c [ however in real-world applications, clients (e.g., mobile devices) generally cannot establish direct communication channels with others (Bonawitz et al. 2017). Therefore, the challenge here is how to efficiently implement secure prediction protocols in the realistic HFL setting.
In this work, to approach the above challenges, we develop PrivHFI., a general and practical frame- work for privacy-presevee iary datasets and designs a simple but effective dataset expansion method only with clients' private datasets. To this end, we instantiate it by leveraging mixup (Zhang et al., 2018) that is originally a regularization technique to improve generalization, and also present some exploration with other data augmentation methods like cutout VDeVries & Taylor 2017) and cutmix (Yun et al. 2019)- The key idea is that the expanded data could provide good coverage of natural dataset distributions effective medium for transferring model knowledge. Second, curely and efficiently evaluate HFI., we leverage the lightweight additive secret sharing technique (Demmler al. 2015) to construct customized secure prediction protocols from scratch in practi- cal setting where there is no direct communication between clients. Our gains mainly come from the improvement communication and computation through the elimination of costly HE and GC pro- tocols. Moreover, in contrast to prior works that evaluate cryptographic protocols in CPUs, PrivHFL converts complex cryptographic operations to simple computations on large blocks of data, which are friendly with GPUs and can be processed by highly-opiinized CUDA kernels (Tan et al. 2021]) As result, PrivHFL is suitable for the batch prediction (i.e., performing multiple predictions at the same time) with lower amortized cost. We evaluate the designed protocol on GPUs and CPUs, and the results show that our GPU-based protocol is up to 10x faster than its CPU analog. Our key contributions be summarized as follows:
HFL. First, PrivHFL. relaxes the assumption of dependence on auxil-
and hence could be used
We introduce a practical HFL framework, which is independent any auxiliary datasets while provably providing comprchensive privacy protection.
We design a simple yet effective dataset expansion method to promote the sharing of model knowledge, and construct customized cryptographic protocols for secure prediction.
Extensive experiments SVHN, CIFARIO, Tiny ImageNet (including IID and Non-III settings) and various heterogeneous models demonstrate that PrivHFL outperforms prior art up two orders of magnitude in efficiency and realizes roughly 10% accuracy gains.
BACKGROUND
Before introducing PrivHFL, we first describe the heterogeneous federated learning and the threat model, and then review the cryptographic primitives that are required to understand our work.
2.1 HETEROGENEOUS FEDERATED LEARNING
In HFL (L & Wang 2019 Choquette-Chen et al. 2021). the clients independently design their own unique models, but due to the model heterogeneity, they cannot directly share model param- eters with each other. Instead, they learn the knowledge of other models via the predictions task-related auxiliary dataset, where a server routes messages between the clients since they gener- ally cannot establish direct communication channels with others (Bonawitz et al.) 2017 Bell al.) 2020). Specifically, clients first train local models with their own private datasets. Then, each client performs prediction on the auxiliary dataset based on the local model and sends the prediction re- sults to the server to aggregate. Later, the server broadcasts aggregated results to clients, which will retrain local models based on the auxiliary dataset and received predictions. The whole process is
As shown in C.5 setting but the cost of increased communication overhead.
by carefully designing protocols, CaPC can be extended to the communication-limin
2

---

Under review as a conference paper at ICLR 2022
iterative until each local model meets the pre-defined accuracy requirement. Details can be referred to Figure 9 and Appendix A
2.2 THREAT MODEL
We work in konest-but-curion adversary setting (Goldreich 2009). where each entity (including the clients and the server) strictly follows the specification of the designed protocol but attempts to infer more knowledge about other clients' private information such as model parameters and private datasets. Moreover, to maintain the reputation and provide more services, the server does not collude with any clients. Formally, an attacker either corrupts the server or subset clients but not both. This setting is reasonable and has been widely instantiated in previous works (Phong et al. 2018 Sun Lyu 2021 Choquette-Choo al. 202I).
2.3 CRYPTOGRAPHIC PRIMITIVES
Additive secret sharing. We adopt lightweight 2-out-of-2 additive secret sharing over the ring ž [Demmler et al. 2015) as the cryptographic building block. We let Share(z) denote the sharing al- gorithm that takes as input x in Z, and outputs random sampled shares [z]o. [a]] with the constraint [xlo + [2]] 1 in Zz. Arithmetic operations can be implemented in the sharing form as shown in
Appendix C.4.1 The reconstruction algorithm Recon( [a]]1) takes as input the two shares and
outputs 0 + [xr] in Zr.. The security of the additive secret sharing protocol guarantees that given a share [ or [z]11, the value x is perfectly hidden.
Pseudorandom generator. A Pseudorandom Generator (PRG) takes as input uniformly random seed and a security parameter k, and outputs a long pseudorandom string. The security of PRG ensures that the output is indistinguishable from the uniform distribution. In PrivHFL, PRGs enable two parties to generate same (pseudo-) random numbers without communication. We instantiate PRG with the technique from (Matyas. 1985) and the seed can be generated by the Diffie-Hellman Key Agreement protocol (Diffie & Hellman 1976). Details can be referred to Appendix C.4.2
3 THE PRIVHFL PROTOCOL
In this section, we introduce the high-level view of PrivHFL, followed by describing in detail our dataset expansion method and secure prediction scheme.
3.1 HIGH-LEVEL VIEW OF PRIVHFL
Expansion
Southatic
prediction
Local Model
Sampling
label
Private Data
Local Model
Query
Hoteregemenus
Local Training
Query-data Generation
Secure Querying
Re-training
Figure High-level view PrivHFL
PrivHFL follows prior HFL. works [Li Wang 2019) and iteratively optimizes the clients' models. Each client in PrivHFL, can play the role of the querying party and the answering party at the same time, and without loss of generality, we denote them as PQ2 and PA, respectively. As shown in Figure in each iteration, each PQ performs four-phase operations with other PA. i.e., local training, query-data generation, secure querying, and re-training. In detail, clients first train the local model on their own private datasets, which is the baseline any future improvements will be compared with. After that, by utilizing our dataset expansion method, each Pg can generate the query data query other PA (C fraction of all clients) for prediction results. To protect the privacy of query
3

---

Under review as a conference paper at ICLR 2022
samples, predictions and model parameters. clients adopt our secure prediction protocol and conduct collaborative querying in the ciphertext form. To the end, each client can retrain the local model based the private dataset, as well the query samples and corresponding prediction results. Algorithm Tin in Appendix C.1 gives the detailed description of PrivHFI.
3.2 QUERY-DATA GENERATION
To relax the assumption of auxiliary datasets, we design and instantiate an effective dataset expan- sion method, inspired by the success of mixup (Zhang et al., 2018) in improving model general- ization and the efficiency of knowledge distillation (Wang et al. 2020). Specifically, we repurpose mixup to construct a big synthesized pool on the small private dataset, which could provide good coverage of the manifold of natural samples. Given any two private samples x; and Tj. we generate multiple synthetic query samples by convex combination with different coefficients À as follows:
(A) = À 24 (1 X) xy-
(1)
Empirically, we set À E [0.1, 0.9] with an interval of 0.1 to generate more diverse synthetic images. We also explore the influence of different À values in Appendix(B. Table This simple method can exponentially expand the size of initial dataset and hence provide more candidate samples for query. Following CaPC (Choquette- strategies (Tog & Koller (001) in Appendix C.3 to select informative samples from the synthesized pool. Note that alternative solution is to directly use private datasets query like knowledge distillation (Hinton et al., 2015). We compare against this method in Section
et al. (02TT)
random sampling and active learning
Extensions of mixup-based method. Dataset expansion based on private samples is a universal and modular method, therefore it can be readily extended with techniques in data augmentation literature. For example, one may replace the mixup-based dataset expansion in PrivHFL with recent methods such as cutout (De Vries & Taylor (017)) and cutmix (Yun et al. 2019). We present some exploration and experiments with these extensions Appendix B.1 and Figure
3.3 SECURE QUERYING PROTOCOL
As shown in Appendix G and Figure 19 it is entirely possible to reconstruct the original image from the mixup-synthetic samples. To further protect the privacy of query samples as well as predictions and model parameters, secret sharing technique, rather than heavy HE and GC techniques. Recall that in our querying protocol, three entities are included, i.e., the server, PQ and Pa. The challenge here is how to efficiently implement such protocol under strict communication constraints, i.e., the communication channels cannot be established among clients. To tackle this challenge, the key idea is to outsource the secure querying task to the server and Pa. and then design a Po-assisted customized protocol to accelerate the evaluation. Moreover, important design principle is to conduct a GPU-friendly protocol that mainly includes vectorized operations and hence is suitable for batch prediction with better amortized costs. In the following, we decompose our scheme into three steps: query-data sharing, secure prediction, and result aggregation.
design secure querying protocol by exploiting the lightweight additive
Query-data sharing. We first construct PRG seeds in pairs for Pq. PA and the server, denoted as SKQA. Sks and Sksq. which are used to generate same random numbers without communication (refer to Figure 18 and Appendix C.4.T). Figure 2 shows our query- sharing protocol II in which Po secret-shares the query data to the server and PA for secure prediction. In particular, Po non-interactively shares [z)o with PA using PRGs on the same seed SliqA. After that, Pq computes and sends [z]]
to the server.
Pg Query data.
Server
PA
r PRG(S
r PRG(Skea)
[x], =x-
[x]p
Figure 2: Secure query-data sharing protocol II
Secure prediction. Recently, many works achieve secure prediction protocols in client-server set- ting (i.e., 2-party setting), such the SOTA CrypTFlow2 (Rathee et al. 2020 HE-Traosformer
4

---

Under review as a conference paper at ICLR 2022
(Boemer et al. 2019a) adopted by CaPC (Choquette- Choo et al. 2021 However, such methods add a huge overhead due to the use heavy cryptographic primitives, c.g., HE and GC. The most efficient secure prediction protocol by far is CryptGPU (Tan al. 202I) under the 3-party setting, but cannot be directly applied to HFL. due to the inability to communicate between clients. To achieve the efficiency of secure prediction and adapt to the communication-limtge scenarios, design customized protocols for the linear layers and non-linear layers from scratch. Figure [7 Appendix C.2 gives graphic depiction of end-to-end secure prediction. Below, elaborate on the evaluation of the linear layers, ReLU and MaxPooling.
P
Server
Pa
[s]
Model parameter [xlo lele- PRG(S)qa)
a. [c]b- PRG(S
- (Rk(S)
+ PRG(S
a
[c]= ab [clo
[x].-b
b (w + a)b-(ch
Iylo = +[x -b) [clu
Figure 3: Secure matrix multiplication protocol Mammul
(i) Linear layers. Linear layers' evaluation follows the idea of the Beaver's multiplication in Ap- pendix [.4.I] but we improve the communication efficiency using PRGs. Specifically. PA and the server compute matrix multiplication wz, where the model parameter is held by PA and the input r secret-shared between PA and the server. Given that wa wx + w(z)] PA can compute w[rrjo locally. As shown in Figure 3 we design an efficient protocol Mamuul for evaluating w l In particular, PQ first generates three random matrices a, and [c]o using PRGs, and computes [c] b. Accordingly, the server on Sksq. and PA obtains a and [c]p using PRGs based on SkQA- (a, b, c) is the Beaver's triple with the constrain
that satisfies [c] + [clo-
generate the same b using PRGs based
ab mask the inputs of PA and the server. Later, the server and PA call the
Beaver's 's multiplication protocol in Appendix C.4.1 to jointly compute [v] and [ylo- i.e. the secret shares of wa. Note that the evaluation of colneceonn, convolutional, and AvgPooling layers can be derived directly from Matmul
(Wagh et al. 2019 Rathee et al.) 2020).
(ii) ReLU. The formula of ReLU is ReLU(x) = DReLU(2), where DReLU(z)) 1 sign(z) namely that it equals 1 if æ 2 0 and 0 otherwise. Since the multiplication of x and DReLU can be implemented by the protocol IlMarmul, we mainly focus on the evaluation of (R))U( i.e., sig. The insight is to convert the calculation of sign(r) to the calculation of (r I) where a random positive number. Based on the above observation, we design a secure DReLU protocol TRRLL shown in Figure 4 In detail, PA and the server first generate random positive value
using PRGs with Sky. and compute [=]= = r[x]] locally. Then they send the secret-shared value [a] to Po- Note that since is randomly generated, the value of r x is also random number in the domain. Similar ideas are also applied to other srivacy-reseeving machine learning works (Wagh) ctt al. 2019; Shen et al. 2020). After that, Po computes the sign of i.e., the sign of z. sign (z) shared to the server and PA based on protocol Istare in Figure
Pe
Server [x]
PA [x] r PRG(S)sa)
+- (RG())s)) [+]== r[x];
[= r(x]o
8,8' PRG(Skga)
8, PRG()))))
[2] [x]
a 5
[2]0 [8]++8
sign(z)
[y] sign(z)
Dy 8"
Figure 4: Secure DReLU protocol IDRELLU
DDepp learning models consist of sequence of linear layers (e.g., fully-connected layers, convolutional layers and AvgPooling) and non-linear layers (c.g., ReLU and MaxPooling). 'The sign(z) function is the most significant hit (MSB) of the value x.
5

---

Under review as a conference paper at ICLR 2022
(iii) Maxpooling. Maxpooling values is computed by using a tree-reduction algorithm, which recursively partitions the input into two halves and then compares the elements of each half. Specif- ically, clients arrange the m values into 2-ary tree with depth log, m, and evaluate the tree in top-down fashion. Compared with the method in Rathee et al. (2020) that requires m - 1 commu- nication rounds, our method achieves lower communication rounds with comparable computational overhead. In each comparison of two secret-shared element [xx] and [y]. we reduce it to the eval- uation of ReLU. We observe max( [v]) = [eLU([*] (y]) + [y]. and hence the computational complexity of Maxpooling evaluation mainly comes from the evaluation of m ReLU.
Result Aggregation. After the secure prediction, the server and each answering party hold the shares of predicted logits [zy].J E [n], which will be aggregated and then returned to Pq- As shown in Figure 5 PA and Po first generate a random value rj based on PRGs. Then each P computes [sy]o rj and sends it to the server. The server aggregates all received values and sends
Σj=1 (123)o ry + [mz]i) Pq. Thus, Pq can reconstruct the aggregated logit y Ej=1 2. and obtains the soft label of the query data.
Pv
P E [m]] Ityle - PRG(Skga)
byhu [M]
(SK
Ei-, (xyl))
[)-i( - -17-1) y softmax(()
Figure 5: Secure result aggregation protocol I[
GPU Acceleration. Our protocols mainly consists of GPU-friendly vectorized secret-sharing, which can be processed by highly-optmmzzed CUDA kernels. However, for multiplication opera- tions, existing CUDA kernels are designed to operate on floating- poiiingg-pint---oo inputs. In PrivHFL, we typically compute over integer values. To leverage optimized kernels for protocol acceleration, our PrivHFL. integrates the CUDALong Tensor abstract in CryptGPU (Tan et al. (021) that embeds the integer-valued cryptographic operations into floating-point arithmetic (refer to Appendix C.6 for more details). However, CrypIGPU's protocols cannot be directly extended to PrivHFL in the communication-limitc scenario, and hence redesign all cryptographic protocols from scratch.
Security analysis of PrivHFL. We give a formal security proof in Appendix F Intuitively, PrivHFL reveals zero information PA and the server, and only reveals the final aggregated prediction to Pq. since all intermediate values are secret-shared Given the above, a corrupted PA cannot learn anything about the query data of querying parties, while the confidentiality of answering parties' model parameters against corrupted Po is also protected.
3.4 DISCUSSIONS
Discussions on differential privacy extension. The differential privacy (DP) guarantee can com- plement PrivHFL. A well-designed DP mechanism can be used as plug-and-play module to pre- vent privacy leakage from the aggregated result. While several works have been proposed in the deep learning domain (Paperno et al. 2017 Sun & Lyu 2021 Proqueete-Chhoo al. (02I), this non-trivial to design a customized DP mechanism for HFL, because the privacy-utility tradeoff is difficult to resolve. Especially, the privacy guarantee will deteriorate with the increase of corrupted clients, unless is mitigated by adding more DP noises at the cost of accuracy. In our setting, we assume up to 1 (n is the number of clients) clients will be escalated to the worst case. Therefore, it is an interesting and challenging work to design a high-utility DP mechanism in the distributed scenario where multiple clients may be corrupted.
be corrupted such that the above problem
Discussions on scalability. In PrivHFL. and general heterogeneous federated learning, each client can play the role of the querying party and the answering party at the same time (Li & Wang 2019). This will incur the overhead of (n²)) secure predictions for each iteration, where n is the number of clients. As the size of the model or the number of clients increases, such overhead issue
Note that the reveal in the DReLU protocol masked intermediate values, rather than plain values.

---

Under review as a conference paper at ICLR 2022
(especially communication overhead) will become the main bottleneck for the scalability of the system. Currently, the system can only carry a small number of clients and medium-sized models. Hence, we view PrivHFL first step in constructing privacy-preservin protocols for HFL.
4 EVALUATION
4.1 EVALUATION SETUP
Datasets and models. We evaluate PrivHFL on three image datasets (SVHN, CIFARIO and Tiny ImageNet). By default, we assume independent and identically distributed (IID) training data among clients, and the Non-IlI) setting can found in Appendix [B.]] For SVHN and CIFARIO, 10, following CaPC set the number of clients
50 and use VGG-7, ResNet-8 and ResNet-10 architectures as the clients' local models. For Tiny ImageNet, we use ResNet- 14, ResNet-16, ResNet- architec- tures and set 10. Unless otherwise stated, only report the accuracy after one iteration, and each model architecture is used by n/3 clients. Refer to Appendix for more experimental setup.
Cryplographic protocol. We build PrivHFL top of CryptGPU (Tan et al. 2021). but reimple- ment the underlying cryptographic protocols proposed in Section 3
We set the security parameter 128. As recommended by CryptGPU, set secret-sharing protocols over the 64-bit ring Zza,, and encode inputs using a fixed-point representation with 20-bit precision.
90
85
Accuracy
Accuracy
80
75
70
10
Iterations
Iterations
(a) SVHN
(b) CIFARIO
Figure 6:
The performance of PrivHFL on differrnt query datasets: the private dataset and the synthetic datset based on mixup. We report the average accuracy of each model architecture.
4.2 EVALUATION ON THE HETEROGENEOUS FEDERATED LEARNING
Test accuracy of PrivHFL on different fractions of participating clients and differ- ent numbers of query data. We also report the model accuracy trained on the local dataset (i.e., baseline described in Section and the model accuracy retrained on the private Mata-based query.
Table 1:
SVHN
CIFAR1O
Tiny ImageNet
C = 0.6
C= 0.8
C=1
C 0.6
C 0.8
C=1
C= 0.6
C = 0.8
C'
Baseline
75.46
56.66
22.26
Private data
79.43
79.56
80.29
60.82
61.01
61.49
24.89
25.11
25.23
2.5K
80.09
80.32
81.69
62.87
63.05
63.23
25.82
26.03
26.23
5.0K
83.32
83.52
83.82
63.04
63.44
63.69
26.22
26.46
26.75
mixup data
7.5K
84.54
84.78
85.12
62.97
63.64
63.88
27.14
27.54
27.75
10K
84.58
84.97
85.62
63.79
63.82
64.56
27.67
28.19
28.46
Table I shows the detailed results the performance improvement (i.e.. accuracy gains) brought by Pri where we randomly select synthetic data to query instead of using active learning strate- gies. We observe that for SVHN and CIFAR10 10, the accuracy gain is about 10% when use 10K'

---

Under review as a conference paper at ICLR 2022
mixup samples. However, using the private dataset-basee query strategy only increases the accuracy by about 4%. Table I also shows different accuracy gains as the number of mixup data increases, in which compared with 2.3K mixup data, obtain 4.65%, 1.33%, 3.23% increased accuracy using 10K samples on SVHN, CIFAR10 and Tiny ImageNet, respectively. This is because more synthetic data could provide better coverage of natural dataset distributions. Besides, with the increase of participating fraction C, the accuracy improves slightly. Figure O and Figure ( show the improve- ment of different heterogeneous models under different iterations and different numbers of query data. We observe that as the number of query data increases, PrivHFL consistently outperforms the baseline for the above datasets and heterogeneous models.
100
VD0?
70
VGG?
Reshet14
90
Res
Accuracy (96)
ResNetB
ReoNet10
Accuracy (%)
30
ResNeti
RecNett B
Accuracy
25
60
70
20
55
60
50
2500
5000
7500
10000
2500
5000
75:00
10000
25:00
5000
7500
10000
Number Query Data
Number of Query Data
Number of Query Data
(a) SVHN
(b) CIFARIO
(c) Tiny ImageNet
Figure 7: Test accuracy of heterogeneous models after PrivHFL as the number of query data increases. Dashed lines represent the baseline, i.e., the test accuracy before executing PrivHFL.
Ablation study of PrivHFL. Figure II(c) in Appendix B.1 illustrates the effectiveness of different data expansion methods. Figure 12 and Figure 16 in Appendix [B.3] further illustrate the impact of active learning strategies and the number of private data samples on the test accuracy, respectively. Besides, we also show the test accuracy of PrivHFL. on CIFAR and SVHN for different degrees
of Non-IID-ness in Figure 15 in Appendix B.3
4.3 EVALUATION ON THE CRYPTOGRAPHIC PROTOCOLS
Table Runtime (sec) of the three steps in PrivHFL's secure querying protocol. The runtime of secure prediction represents the query time of each client on different models. CIFARIO and SVHN have the same time due to the same input size and model architecture.
Dataset
Queries
1. Query-data sharing
Secure prediction
3. Result aggregation
VGG7
ResNet8
ResNet10
1000
5.08
55.97
87.71
106.46
0.09
CIFARIO
2500
7.16
133.52
206.20
256.39
0.12
(SVHN)
5000
11.32
271.89
428.54
512.88
0.30
ResNet14
ResNet16
ResNer18
Tiny ImageNet
1000
9,87
737.43
962.51
1075.13
0.18
2500
18.78
1778.69
2356.33
2537.49
0.32
We mainly focus on the extra overhead caused by the secure querying phase. To clearly illustrate the efficiency of PrivHFL, unless otherwise specified, we only show the overhead of one communication round as shown in Section 3 Recall that our secure querying protocol consists of three phases, i.e., query-data sharing, secure prediction, and result aggregation. As shown Table the main cost of framework comes from the second phase, where the evaluation of batched secure prediction is required. Specifically, it takes 4.5 minutes to evaluate 5000 query samples securely on VGG7 and CIFAR10. Besides, only 11.32 seconds and 0.3 seconds are spent the query-data sharing and result aggregation phases. More runtime is required to evaluate Tiny ImageNet because of increased input size and model architecture.
To demonstrate the effectiveness of PrivHFL's secure prediction protocol, we compare with CrypT- Flow2 (Rathee al. 2020) and CryptGPU (Tan et al. 2021). In Table of Appendix B.2 also
8

---

Under review as a conference paper at ICLR 2022
Table Runtime (sec) and communication cost (MB) of secure prediction for different meth- ods on CIFAR1O. CrypTFlow2 (Rathee et al. 2020) is the SOTA 2-party protocol that contains two variants (OT-based and HE-based) and CryptGPU (Ta et al. 2021) is the SOTA 3-party GPU- friendly protocol. We report the cost of single prediction on three models.
VGG7
ResNet8
ResNet10
Method
Time
Comm
Time
Comm.
Time
Comm.
CrypTFlow2-OT
Rathee et al.
2020
39.22
15562.9
39.35
21261.6
55.76
28517.6
CrypTFlow2-HB
(Rathee et al.
2020
48.70
651.51
56.21
1110.39
97.46
1395.18 18
CryptGPU (Tan et al. 2021
1.61
144.51
2.02
131.39
2.79
221.57
This work (CPU)
1.03
66.61
1.63
73.85
2.00
105.88
This work (GPU)
0.35
66.61
0.36
73.85
0.53
105.88
compare with CaPC (Choquette-hhov al.) 2021) that uses HE-Transformer as the building block. Table summarizes this improvement for three models over CIFARIO. We observe that, PrivHFL achieves significant improvement all the models in terms of runtime and communication costs. To be specific, PrivHFL requires 105.2-112.0 x less runtime and 233.6-269.3 x less communica- show a greater
tion compared with 2-T Similarly, compared with FFypTHH advantage (i.e.. 139.1-1823 1-88..8 x) in runtime, and relatively small advantage (i.e., 9.7-15.0 x) in com- munication overhead. In addition, also test the performance under the CPU architecture. Notably, even comparing with CryptGPU, our weaker CPU setting obtains an improvement of roughly 1.5 x and in terms of computation and communication overheads, respectively.
70
VGG7-GPU
VGG7-CPU
Resnel8-GPU
Resnel 8-CPU
VGG9-GPU
VGG9-CPU
60
Resnet10-GPC
Resnet CPO
Runtime (sec)
20
VGG11-GPU
VGG11 1 1-CUU
Runtime (sec)
Kesnet12-GPP
Resnet 12 CPU
50
40
10
30
20
10
0
0
10
20
30
40
50
60
10
20
30
50
60
Batch Size
Batch Size
Figure 8: Runtime (sec) batch secure prediction on CPU and GPU settings the batch size increases. We report the result of VGG-style and ResNet-style networks on CIFAR10.
To further understand the impact of GPU acceleration PrivHFL, evaluate secure prediction protocol under both the CPU and GPU settings with different batch size in Figure g We observe that the GPU-based secure prediction is always superior the CPU analogs. As the batch size increases, the advantages of GPU-based protocols becomes more pronounced, e.g., the 9.5 x reduction on ResNet8 over batch of 64 images.
CONCLUSION
In this paper, propose practical heterogeneous federated learning framework, which is inde- pendent on auxiliary datasets while provably guaranteeing the privacy of samples, model parameters and predictions. Extensive experiments demonstrate that PrivHFL outperforms prior art two orders of magnitude in efficiency and realizes about 10% accuracy gains. the future, we will further improve the scalability of the system and integrate with advanced differential privacy mechanisms.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Michel Abdalla, Mihir Bellare, and Phillip Rogaway. The oracle diffie-hellman assumptions and an analysis of dhies. In Proceedings of CT-RSA, pp. 143-158. Springer, 2001.
Buse Gul Atli, Yuxi Xia, Samuel Marchal, and Asokan. Waffle: Watermarking in federated learning. arXiv preprint 20Xv:2008.00298,, 2020.
James Henry Bell, Kallista A Bonawitz, Adrià Gascón, Tancrède Lepoint, and Mariana Raykova. Secure single-server aggregation with (poly) logarithmic overhead. In Proceedings of ACM CCS, pp. 1253-1269, 2020.
Fabian Boemer, Anamaria Costache, Rosario Cammarota, and Casimir Wierzynski. ngraph-he2: A high-throughput framework for neural network inference on encrypted data. In Proceedings of the 7th ACM Workshop on Encrypted Computing & Applied Homomorphic Cryptography, pp- 45-56, 2019a.
Fabian Boemer, Yixing Lao, Rosario Cammarota, and Casimir Wierzynski. ngraph-he: a graph compiler for deep learning on homomorphically encrypted data. In Proceedings of the 16th ACM International Conference on Computing Frontiers, pp. 3-13, 2019b.
Keith Bonawitz, Vladimir Ivanov, Ben Kreuter, Antonio Marcedone, Brendan McMahan, Sarvar Patel, Daniel Ramage, Aaron Segal, and Karn Seth. Practical secure aggregation for privacy- preserving machine learning. In proceedings of ACM CCS, pp. 1175-1191, 2017.
Alon Brutzkus, Ran Gilad-Bachrach, and Oren Elisha. Low latency privacy preserving inference. In Proceedings of ICML, pp. 812-821, 2019.
Ran Canetti. Universally composable security: A new paradigm for cryptographic protocols. In Proceedings of FOCS, pp. 136-145. IEEE, 2001.
Hongyan Chang. Virat Shejwalkar, Reza Shokri, and Amir Houmansadr. Cronus: Robust and heterogeneous collaborative learning with black-box knowledge transfer. arXiv preprint arXiv:1912.11279 2019.
Hao Chen, Wei Dai, Miran Kim, and Yongsoo Song. Efficient multi-key homomorphic encryption with packed ciphertexts with application to oblivious neural network inference. In Proceedings of ACM CCS, pp. 395 412, 2019.
Christopher A Choquette- Choo, Natalic Dullerud, Adam Dziedzic, Yunxiang Zhang, Somesh Jha, Nicolas Papermot, and Xiao Wang. Cape learning: Confidential and private collaborative learning. In Proceedings of ICLR, 2021.
Daniel Demmler, Thomas Schneider, and Michael Zohner. Aby-a framework for efficient mixed- protocol secure two-party computation. In Proceedings of NDSS, 2015.
Terrance De and Graham W Taylor. Improved regularization convolutional neural networks with cutout. arXiv preprint 1708. 04552,, 2017.
Enmao Diao, Jie Ding, and Vahid Tarokh. Heterofl: Computation and communication efficient federated learning for heterogeneous clients. In Proceedings of ICLR, 2021.
Whitfield Diffie and Martin Hellman. New directions in cryptography. IEEE Transactions Infor- mation Theory, 2246)4444-654 1976.
Canh T Dinh, Nguyen H Tran, and Tuan Dung Nguyen. Personalized federated learning with moreau envelopes. In Proceedings of NeurIPS, 2020.
Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Personalized federated learning with theo- retical guarantees: A model-agnostic meta-learning approach. In Proceedings of NewrIPS, 2020.
Karan Ganju, Qi Wang, Wei Yang, Carl A Gunter, and Nikita Borisov. Property inference attacks on fully connected neural networks using permutation invariant representations. In Proceedings of ACM CCS, pp. 613,633 2018.
10

---

machine Under review as a conference paper at ICLR 2022
Craig Gentry. Fully homomorphic encryption using ideal lattices. In Proceedings of ACM STOC, pp. 169-178, 2009.
Ran Gilad-Bachrach, Nathan Dowlin, Kim Laine, Kristin Lauter, Michael Nachrig, and John Werns- ing. Cryptonets: Applying neural networks encrypted data with high throughput and accuracy. In Proceedings of [CML, pp. 201-210, 2016.
Oded Goldreich. Foundations of cryptography. Cambridge university press, 2009.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. Proceedings of CVPR, pp. 770-778, 2016.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint drX1v:1503.02531, 2015.
Bargav Jayaraman and David Evans. Evaluating differentially private machine learning in practice. In Proceedings of USENIX Security, pp. 1895-1912, 2019.
Bargav Jayaraman and Lingxiao Wang. Distributed learning without distress: Privacy-preserving empirical risk minimization. 2018.
Chiraag Juvekar, Vinod Vaikuntanathan, and Anantha Chandrakasan. Gazelle: A low latency frame- work for secure neural network inference. In Proceedings of USENIX Security, pp. 1651-1669, 2018.
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Ad- vances and open problems in federated learning. arXiv preprint arXiv: 2019.
Brian Knott, Shobha Venkataraman, Awni Hannun, Shubho Sengupta, Mark Ibrahim, and Laurens van der Maaten. Crypten: Secure multi-party computation meets machine learning. arXiv preprint arXiv:2/09. 009844 2021.
Daliang Li and Junpu Wang. Fedmd: Heterogenous federated learning via model distillation. In Proceedings of NeurIPS Workshop on Federated Learning for Data Privacy and Confidentiality, 2019.
Tao Lin, Lingjing Kong, Sebastian U Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning. In Proceedings of NeurlPS, 2020.
Stephen M Matyas. Generating strong one-way functions with cryptographic algorithm. IBM Tech- nical Disclosure Bulletin, 27:5658-566 5559,,555559 1985.
Pratyush Mishra, Ryan Lehmkuhl, Akshayaram Srinivasan, Wenting Zheng, and Raluca Ada Popa. Delphi: A cryptographic inference service for neural networks. In Proceedings of USENIX Secu- rity, pp. 2505 2000,2222,222 2020.
Payman Mohassel and Yupeng Zhang. Secureml: system for scalable learning. In Proceedings of IEEE S&P, pp. 19-38, 2017.
Moni Naor and Benny Pinkas. Efficient oblivious transfer protocols. In Proceedings of SODA, pp- 448-457, 2001.
Nicolas Papernot, Martín Abadi, Ulfar Erlingsson, Ian Goodfellow, and Kunal Talwar. Semi- supervised knowledge transfer for deep learning from private training data. In Proceedings of ICLR, 2017.
Le Trieu Phong. Yoshinori Aono, Takuya Hayashi. Lihua Wang, and Shiho Moriai. Privacy- preserving deep learning via additively homomorphic encryption. IEEE Transactions on Infor- mation Forensics and Security, 1333-1333-1144 2018
Deevashwer Rathee, Mayank Rathee, Nishant Kumar, Nishanth Chandran, Divya Gupta, Aseem Rastogi, and Rabul Sharma. Cryptflow2: Practical 2-party secure inference. Proceedings of ACM CCS. pp. 325-342, 2020.
11

---

Under review as a conference paper at ICLR 2022
M Sadegh Riazi, Christian Weinert, Oleksandr Tkachenko, Ebrahim M Songhori. Thomas Schnei- der, and Farinaz Koushanfar. Chameleon: A hybrid secure computation framework for machine learning applications. In Proceedings of AsiaCCS, pp. 707-721, 2018.
Ahmed Salem, Yang Zhang, Mathias Humbert, Pascal Berrang, Mario Fritz, and Michael Backes. MI-leaks: Model and data independent membership inference attacks and defenses on machine learning models. Proceedings of NDSS, 2019.
Sinem Sav, Apostolos Pyrgelis, Juan R Troncoso-Pastoriza, David Froelicher, Jean-Philippe Bossuat, Joao Sa Sousa, and Jean-Pierre Hubaux. Poseidon: Privayy-rreeervig federated neural network learning. In Proceedings of NDSS, 2021.
Tobias Scheffer, Christian Decomain, and Stefan Wrobel. Active hidden markov models for in- formation extraction. In International Symposium on Intelligent Data Analysis, pp. 309-318. Springer, 2001.
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. In Proceedings of ICLR, 2018.
Claude Elwood Shannon. A mathematical theory of communication. ACM SIGMOBILE mobile computing and communications review, 5(1):3-55, 2001.
Liyan Shen, Xiaojun Chen, Jinqiao Shi, Ye Dong, and Binxing Fang. An efficient 3-party framework for privacy-preserving neural network inference. In Proceedings of ESORICS, pp. 419-439, 2020.
Lichao Sun and Lingjuan Lyu. Federated model distillation with noise-free differential privacy. In Proceedings of IJCAI, 2021.
Sijun Tan, Brian Knott, Yuan Tian, and David Wu. Cryptgpu: Fast privacy-preserving machine learning the gpu. In Proceedings of IEEE S&P, 2021.
Simon Tong and Daphne Koller. Support vector machine active learning with applications to text classification. Journal of Machine Learning Research, 2(Nov):45 66,5,,66666666 2001.
Sameer Wagh, Divya Gupta, and Nishanth Chandran. Securenn: 3-party secure computation for neural network training. Proceedings on Privacy Enhancing Technologies, 2019(3):26 49, 2019.
Dongdong Wang. Yandong Li, Liqiang Wang, and Boqing Gong, Neural networks are more pro- ductive teachers than human raters: Active mixup for data-efficient knowledge distillation from blackbox model. In Proceedings of CVPR, pp. 1598-1507, 2020.
Ziqi Yang, Jiyi Zhang, Ee-Chien Chang, and Zhenkai Liang. Neural network inversion in adversarial setting via background knowledge alignment. In Proceedings of ACM CCS, pp. 225-240, 2019.
Andrew Chi-Chih Yao. How to generate and exchange secrets. In Proceedings of IEEE FOCS, pp. 162-167, 1986.
Tehrim Yoon, Sumin Shin, Sung Ju Hwang, and Eunho Yang. Fedmix: Approximation of mixup under mean augmented federated learning. In Proceedings of ICLR, 2021.
Sangdoo Yun, Dongyoon Han, Scong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy train strong classifiers with localizable features. In Proceed- ings of ICCV, pp. 6023-6032, 2019.
Mikhail Yurochkin, Mayank Agarwal, Soumya Ghosh, Kristjan Greenewald, Nghia Hoang, and Yasaman Khazaeni. Bayesian nonparametrie federated learning of neural networks. Proceed- ings of ICML. pp. 7252-7261, 2019.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In Proceedings of ICLR, 2018.
Zhuangdi Zhu, Junyuan Hong, and Jiayu Zhou. Data-free knowledge distillation for heterogeneous federated learning. In Proceedings of ICML, 2021.
12

---

Under review as a conference paper at ICLR 2022
A HETEROGENEOUS FEDERATED LEARNING
In federated learning (FL), multiple clients collectively learn single model with a fixed architecture, which each client maintains local model for the private dataset, while the server maintains global model via aggregating the local model gradients from clients. Although successful, FL is often restrictive in practice, since it assumes all clients sharing the same model architecture. To tackle this problem, heterogeneous federated learning (HFL) is proposed, which enables clients to collaboratively train their own customized models that may differ in size or structure. Due to the heterogeneity in local models among clients, it's possible collaborate via sharing gradients like FL. As shown in Figure [ generally, HFL uses a task-related auxiliary dataset to assist in collaborative learning. To be specific, each client first trains the local model with individual private dataset and then performs inference on the auxiliary dataset to obtain the prediction results, which are sent the server to aggregate, rather than the local model gradients. Then, the server broadcasts the aggregated results to clients, and the clients will retrain local models based on the auxiliary dataset and the received predictions.
Global
Ausiliary Dataset
Server
Server
Updlat
Local Upddate
Lucal Updaate
Lacal Prodiction
Local Predicion
Local Prediction
Local Model
Local Model
Local Model
Local Model
Local Model
Local Model
Local Data
Local Data
Local Data
Locd Data
Local Data
Broadcast global model
Model prediction
Broadcast global prediction
Figure 9: Comparison with federated learning and heterogeneous federated learning
B ADDITIONAL EXPERIMENTAL RESULTS
B. UNDERSTANDING DATASET EXPANSION
Recall that we instantiate the dataset expansion method by leveraging mixup (Zhang et al. 2018) in Section 3.2 To explore the effectiveness of the mixup-based datset expansion method, we
sualize the feature distribution of original data and synthetic samples on SVHN and IIFAR10, As shown in Figure [a) and Figure Tibb the synthetic samples cover a larger part of the feature space and hence they should be more diverse and informative compared with original data. In other words, it could provide good coverage of the manifold of natural samples. Therefore, learning the predictions from other clients on the synthetic samples can further improve the accuracy of local models.
Note that our dataset expansion method is universal and reconfigurable, which can be extended by leveraging other data augmentation strategics. Figure [I(c) gives the accuracy gains under various data augmentation strategies, including random sampling from Gaussian distribution. random flip- ping, cutmix and cutout. Cutmix (Yun et al. 2019) can be formulated as ã.,; M. +1 (1-M) where M E {0, 1| a binary mask matrix of size W x H to indicate the location of drop- ping out and filling from the two images 2; and Zy. Cutout (DeVries & Taylor 2017) augments the dataset with partially occluded versions of original samples. Figure 10 shows the overview of the results of these five strategies original image. As shown in Figure [T(c). the Gaussian
13

---

Under review as a conference paper at ICLR 2022
noise-based and random flipping- based dataset expansion strategies dramatically reduce the model accuracy. The main reason that the two strategies do not necessarily model the problem domain distributions and do not provide informative natural images. In contrast, the strong data augmen- tation strategies, cutout and cutmix, are good choices for data generation in PrivHFI. Like mixup, these two methods can construct big synthesized pool small private dataset that could provide a good coverage of the manifold of natural samples, to better distill knowledge of heterogeneous models.
Gaussian Noise
Random Flipping
Cutmix
Cutout
Mixup
Overview of the results of Gussian noise, random flipping, cutmix, cutout, and mixup data augmentation strategies.
Figure 10:
B.2 EXPERIMENTAL RESULTS FOR SECURE PREDICTION
In the model querying phase, we leverage the GPU parallelism to process private querying on batch images, which effectively amortize the cost of private inference. In Table we compare our GPU-friendly method on MNIST using the three model architectures (CryptoNets Gilad Bachrach al.) 2016) and MLP (Boemer al. (019b))
al., 2016), CryptoNets- ReLU GGilad-Bachraa with HE-Transformer, which also achieves batch-axis packing for private inference. Note that CaPC (Choquette-Choo et al. 2021) also adopt Ta-Transerrmer as the building block. We can observe that PrivHFL is up three orders of magnitude faster than HE-Tram RE-Trssfformer on the CryptoNets CtyptoNets-ReLU and MLP models. The main reason is the efficiency of the protocol and the parallelism supported by the GPU. However, when evaluating CryptoNets that replaces ReLU activations with squared approximation, the improvement from PrivHFL will gradually decrease as the batch size increases, because the HE-based method does not require communication during the entire evaluation pro- cess. Note that approximations result in significant accuracy losses and degrades user experience, especially when evaluating modern large-scale models.
B.3 EXPERIMENTAL RESULTS FOR HETEROGENEOUS FEDERATED LEARNING
Impact of different active learning strategies. Figure 12 depicts the impact of different active learning strategies, in which even random sampling can drastically improve the model accuracy by
Randocs Flipping
Accuracy
70
Mixup
60
CIFARIO
SVHN
Feature distribution SVHN. (b) Feature distribution
CI- (c) Comparison with other dataset expansion methods.
FARIO.
Understanding the importance of dataset expansion on improving model perfor- mance. In (a) and (b), the bright colored points indicate the position of the original training data in the feature space, and the light blue points indicate the distribution of the expanded points based on mixup in the feature space. In (c), the left and right dashed lines are the baseline test accuracy on CIFAR and SVHN, respectively.
Figure 1I:
14

---

Under review as a conference paper at ICLR 2022
Table
Runtime (sec) of secure prediction for HrE-Tranfformer and our PrivHFL on MNIST
as the batch size increases. He-Transformer is the 2-party secure prediction protocol used in CaPC CChoquett--hhon Choo et al. 2021).
BS=128
BS=256
BS=512
BS=1024
BS=2048
Model
Ours
HE-T
Ours
HE-T
Ours
HE-T
Ours
HE-T
Ours
HE-T
CNetl
0.07
17.75
0.14
17.56
0.25
17.62
0.49
17.77
0.99
17.67
CNer2
0.07
48.83
0.15
70.14
0.26
112.42
0.51
201.42
1.01
369.51
MLP
0.05
65.01
0.06
86.37
0.10
129.81
0.16
216.61
0.29
391.13
100
70
Random
ICD
Random
ICD
90
Entropy
K-Center
Entropy
K-Center
Accuracy (%)
Margin
Accuracy (%)
Margin
80
60
70
55
60
50
2500
5000
7500
2500
5000
7500
Number of Query Data
Number of Query Data
(a) SVHN
(b) CIFARI0
Figure 12: Using PrivHFL with active learning strategies to improve model accracy on SVHN and CIFARIO datasets. Dashed line represents the baseline accuracy models, and the histogram represents the accuracy of the model after PrivHFL based on different active learning strategies.
4.46%-9.666 and 6.57%-7.22% on SVHN and CIFAR10 datasets. Nonetheless, we can also achieve additional benefits leveraging active learning, such as 1.44% gains on SVHN with the entropy sam- pling and 0.8 % gains CIFARIO with the margin sampling.
Impact the Non-IID-ness degree. We use the Dirichlet distribution Dir(a) as in Lin et al. (020 to simulate disjoint Non-IID training data. Figure 14 visualizes the distributions of Non-IID samples
among clients with different Dir(a) on CIFAR 10 dataset, where the number of clients n = 20. The value of controls the degree of Non-IID-ness, in which smaller indicates higher degree of Non-IID-ness. When & 100, the distribution closes to uniform sampling. When = 0.5, the distribution of samples of each class among clients is extremely uneven. Figure 15 shows the test accuracy of PrivHFL on SVHN and CIFARIO for different degrees of Non- IID-ness. We can observe that the higher the degree of Non-IID-ness, the lower the accuracy of the model. However, in this case, the mixup-based data expansion method can still significantly improve the performance of models.
Impact of the private data volume. Figure 16 illustrates the impact of private data volume on the test accuracy. We can observe that as the private data volume increases, the performance of models is on the rise. The main reason is that the model can learn more knowledge from more private training data, and it can also generate more mixup synthetic samples to query, so as to promote the sharing of model knowledge.
Impact of the normalization operation. As described Figure 5 we directly aggregate the pre- dicted logits instead of the normalized values. There are two major reasons: The normalization operation the secure computation will introduce extra communication and computation overheads, since it contains costly multiplication and division operations (Knott al. 202I). ii) The diverse logits of the heterogeneous models may contain informative content and hence facilitate model ac- curacy. This phenomenon is particularly prominent complex datasets. As shown in Figure 13
15

---

Under review as a conference paper at ICLR 2022
Table 5: Impact of the coefficient À in mixup on CIEARIO.
mixup À
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Accuracy
61.93
61.82
61.42
61.37
61.38
61.35
61.80
61.53
61.73
95
70
W/o normalization With normalization
Wo normalization With normalization
90
Accuracy (%)
Accuracy (%)
60
55
60
50
VGG7
Res
ResNet10
VGG7
ResNet8
ResNet1
(a) SVHN
(b) CIFARIO
Figure 13: Impact of the normalization operation the accuracy of heterogeneous models on SVHN and CIFAR10 datasets. Dashed line represents the average accuracy of heterogeneous models when only private data used for query.
the model accuracy CIFARIO without normalization is about 5% higher than that with normal- ization. On SVHN, the influence of normalization is negligible, since this classification task is relatively simple.
Impact of the coefficient in mixup. Table shows the impact of different mixup coefficients on the model accuracy over CIFARIO. For each result, we only use one coefficient generate synthetic images. We can observe that in different coefficient values, the model accuracy remains almost unchanged, and hence the coefficient of mixup has negligible effect on the accuracy of the heterogeneous models in PrivHFL. Therefore, we set À E [0.1, 0.9] with an interval of 0.1 in our experiments to generate more diverse synthetic images.
C MISSING DETAILS ON PRIVHFL
C.1 ALGORITHMIC DESCRIPTION
Algorithm 1 gives the detailed description of the PrivHFL framework. In PrivHFL, each client first trains the local model M, to convergence on the private dataset. Later, clients improve the perfor- mance of local models based on the knowledge of others via the predictions on the synthetic dataset.
:
Class
Party 1D%
Party 10s
Party IDs
100
(b)
0.5
Figure 14: Visualization of Non-IID-ness among clients with different Dirichlet distribution values on CIFARIO dataset. The size of scattered points indicates the number of training samples for class available to that client.
16

---

Under review as a conference paper at ICLR 2022
90
70
Baseline
Baseline
80
Private data query
Private data query
Accuracy (%)
Mixup data query
Accuracy (%)
60
Mixup data query
50
50
40
40
30
30
0.5
1
0.5
Degree of Non-IID-ness
Degree of Non-IID-ness
(a) The model accuracy SVHN.
(b) The model accuracy CIFARIO.
Figure 15:
The test accuracy of PrivHFL on SVHN and CIFARIO with different degrees of -en-I.D-eess. The number clients n = 20.
100
70
Baseline
Baseline
90
Private data query
Private data query
Accuracy (%)
Mixup data query
Accuracy (%)
Mixup data query
70
50
60
40
40
30
20
200
500
600
800
1000
200
500
600
800
1000
Number of Private Data Samples
Number of Private Data Samples
(a) SVHN
(b) CIFARI0
Figure 16: Impact of different amounts of private data on the accuracy of models on SVHN and CIFAR10. The number of clients 50, and the number of private data samples for each client ranges from 200 to 1000.
17

---

Under review as a conference paper at ICLR 2022
Therefore, they need first construct big unlabeled data pool based on the small number of pri- vate data, and then actively choose a subset from the pool as the query data in each iteration. Next, they perform the secure querying protocol based on the query data, and obtains the final predictions, which will be used to retrain the local model. Query-data generation and secure querying process (refer to Section 3 for more details) will iterate iter times until the model M, achieves pre-defined performance.
Algorithm The PrivHFL. framework.
Input:
Each client Pj. J E [n2), holds a private dataset Dj and customized local model Mj. iter denotes the number of training iterations, where in HFL an iteration means that all clients have completed knowledge transfer. B denotes the number of the query dataset and denotes the set of selected answering parties in current query phase.
Output: Trained models
J. [n]]
I: for each j E [n]] do
2: Pj locally trains the local model M, on Dj using the stochastic gradient descent optimization.
3: end for
4: for each iter
5: for each querying party P E [n]] do
6:
Pa generates an unlabeled synthetic pool D
pool with its own private dataset Dj by utilizing
the dataset expansion method in Section 3.2
7: Pa applies the active learning methods in Section C,3 to select the query dataset {zs hE[S
from D Proot
8:
for each answering party PA i E C do
9:
Pg secret-shares {[z]]b<[a]] with Pj and the server, based the protocol IS
10:
PA- P3 and the server jointly perform the secure querying protocol in Section 3.3 Pa secret-shares the predictions [villeet Pq and the server. end for
11:
12:
Pg computes bet with y protocol Agg with the server.
Exec 9 via performing the secure result aggregation
14: PaJ retrains its local model based on the query dataset ( yo )E(BB and private dataset DD-
end for
16: end for
C.2 GRAPHIC DEPICTION OF END-TO-END SECURE PREDICTION
Figure 17 gives graphic depiction to illustrate the end-to-end secure prediction implemented across all the layers, where the inputs are secret-shares of image x, i.e., [x]o and [z]] (as shown in Figure
20
The inputs first pass through a convolutional layer that mainly contains matrix multiplication operations W1 x (wi is the parameter of this layer) and can be implemented by the protocol I in Figure 3 The outputs of this layer are in the secret sharing form, i.e., 1yI and [y1 11 obtained by PA and the server, respectively. For the ReLU layer, recall that ReLU(g)) = (yyLU(y)). Therefore, the protocol ITR in Figure 4 is executed first obtain DReLU(g))) )'s shares (D)eLU(y))) )Jo and [DReLU()1)]1- Then, PA and the server invoke an instance of the protocol II and obtain [22)0 and [32]1. respectively. The outputs of IIDRELU are the inputs of the subsequent MaxPooling layer. As described in Section 3.3 (iii), MaxPooling on n values can be converted into n 1 ReLU operations. Therefore, the output of this layer is also in the secret sharing form. When the inference goes to the final fully-connected layer with inputs 1yn-1lo and [Vm-1] ll owned by PA and the server, respectively, the protocol Il is executed. the end, PA and the server obtain the secret-shares of the predicted logit, i.e., [logit]o and [logit]]. respectively.
C.3 ACTIVE LEARNING STRATEGIES
In order to choose query data that most likely contribute improve the performance of local mod- els, inspired by CaPC (Choquette- et al.) 202T). active learning is adopted. To be specific, active learning allows local models actively choose the data from which they learn, which con-
18

---

Under review as a conference paper at ICLR 2022
[
DiJa
D22
D
Dn-sl1
[logit]
Server
Conv (Protoral llcarma)
ReLU (Protoco !iuareas Hisccul
MaxPoolligg
MPC
(Prooccol Baa--un
(Protoool liaarral
sky XX
(Reldigh)
Ye marpoolgg))
lnght dig Te-
Pa
[x]o. 021
b/2la
Daao
Vn-ilowwx
[logit]
Figure 17: The whole private inference implemented across all the layers. Orange boxes rep- resent linear layers (including convolutional/fullyy wowoluttion//fllllooooteee////// represent non-linear layers (including ReL.U/MaXPooling layers).
layers), and blue boxes
tains various pool sampling strategies to estimate the informativeness and diversity of unlabeled samplings. These strategies can be classified into two categories. One important class that of uncertainty- approaches, which try to find hard samples using heuristics like highest entropy, such as margin sampling and entropy sampling. There are recent optimization-based methods, such as greedy-k-center sampling to obtain diverse subset of hard samples.
Definition 1 (Margin Sampling (Scheffer et al.) (007T). Margin sampling assumes that the most informative samples are those which tall within this margin of decision boundary. Formally, given an unlabeled dataset D, it can be represented as:
& = arg min zE
/min// (x.
(2)
where f (x, w) represents the distance of the data sample x from the hyperplane for class w. Using entropy as an uncertainty measure as
Definition 2 (Entropy Sampling (Shannon follows:
2001)).
& arg max RED
Σ
Po (ys x)) log Pa (y4 | x))
(3)
where Po (z) is the conditional label distribution of the model, y ranges over all possible labels.
Definition (Greedy-k-center Sampling (Sener & Savarese 2018). Solving the k-center problem, i.e. choosing b center points such that the largest distance between a data point and nearest center is minimized. Formally, this goal is defined as:
min max min s/./=' Sb jES'UN
A (x;,
(4)
where s is new chosen center points, S is the current training set.
Besides, to better balance uncertainty and diversity, we introduce informative-and-divvrrr sampling called ICD, which chooses query data based on informative and diverse criteria using margin and cluster-based sampling methods. will return highest uncertainty lowest margin points while main- taining same distribution over clusters entire dataset.
C.4 MORE BACKGROUND ON CRYPTOGRAPHY
C.4.1 SECRET SHARING AND MULTIPLICATION TRIPLES
As shown in Section 2.3 in PrivHFI., we utilize additive secret sharing to protect the privacy of sen- sitive information. Assume there are two secret-shared values [a:] and [y] that owned by two parties, addition and subtraction operations ([2) = [x] + [y]) can be done locally without any communication, which is realized as [2]; [r] + [y]e mod 204 by each party P., á 10.
(5)
=
[[æ)a + [1v]) + (w)a) = [ o [e]][v]
[ [x]][
19

---

Under review as a conference paper at ICLR 2022
Multiplying two secrets, i.c., [2] = [x] [y], however, is evaluated by using Beaver Multiplication Triple (Demmler et al. 2015). The triple refers to (a, b, c) with the constraint
ab, which can be generated using cryptographic techniques (Demmler et al., 2013) or a trusted dealer (Riazi et al. 2018). To be specific, shown in 5 [zli[y]] can be computed by party P. locally, but [x] 1-: evaluated as follows. Taking [z]o[y] as an example, we suppose Pn and Pi already hold the triples (a, [clo) and (b, [c]1), respectively. Po first sends [r]o + a (mod 264 Pt, while Pt sends [y]
(mod 261) to Po- Then Po computes one share of [=]][(]] as [x]o[[v]] -) [[clo (mod 26t), and Po computes another as ([r]o + a)(y)1 - [cc2 (mod 26a), locally. In this way, both parties can achieve secure multiplication operation, and the outputs are still in the form of secret sharing.
C.4.2 DILLEEHELLMAN KEY AGREEMENT PROTOCOL
In order to allow two parties to agree the same secret key on an insecure channel, the Diffie-Hellman Key Agreement (DH) protocol (Difie & Hellman 1976) was proposed. In PrivHFI, the DH pro- tocol is used to generate the consistent PRG seeds between clients, which consists of the following three steps:
DH. (H..parm(()) (k) + (G, 9, H) generates a group G of prime order q, along with a generator 9: and hash function H.
DH.gen(G, g. q, H) + (#+.g")) randomly samples 2; E Zg as the secret key and let gm as the public key.
DH.agree(zz (z;, g" + Sijj outputs the seed s1 = (((g)))z:).
In the DH protocol, correctness requires that for any key pairs (e.. and (2),9*)) gen- erated by two paries P. and P, using DH.gen under the same parameters (G. q, H), DH.agree(2,, g" H) = D...gree(zy, g"h , H). Besides, in the conest-but-curion adversary set- ting. security requires that for any adversary who steals g and gdy (but neither of the corresponding s; and vt)) the agreed secret 8i,j derived from those keys is indistinguishable from a uniformly ran-
dom value (Abdalla et al. 2001).
Seed generation between clients without directly communication. The communication improve- ment of our PrivHFL is largely derived from the application of PRGs. These allow any two clients to jointly generate same (pseudo-) random values that are used in MPC protocols without commu- nication. is trivial to construct PRG seeds Sks, (between PA and the server) and Sksq (between PQ and the server). For instance, the server first generates Sks and Sksq. and then sends them to Pa and Pq. respectively. However, this is challenging for constructing SkQa, given that direct communication channels can not be constructed between clients. To tackle this issue, we generate seed SkQA using the DH protocol (Diffie & Hellman 1976). classic algorithm for exchanging secret keys securely. Specifically, we first call the above DH.gen algorithm, and communicate the resulting public keys of two clients via the server as an intermediary. After that, the same seed is locally generated by the two clients via the DH.agree algorithm. We show the protocol Isedd in Figure and its security follows from the security the DH protocol (Abdalla et al. 2001).
Po Sample kg E 2 A , modp
Server Sample Skso. Skya Znp
PA Sample kG E Z. 8 mod p
R,5ksg
A
A, S.
Sksg SkØa agreee B, H)mod
SKsa Skoa DH. DH.agree(Ka..) H. Hmod
Figure 18: Secure PRG seed generation protocol 11sest
20

---

Under review as a conference paper at ICLR 2022
C.5 EXTENDING CAPC TO COMMUNICATION-LIMITES SETTINGS
By carefully designing protocols, CaPC (Choquette-Chor et al. 2021 can also be extended to a setting where there is direct communication between clients. However, as analyzed below, such extension comes at the cost of increased communication overhead. Therefore. our PrivHFL has greater advantages compared to the following modified protocols.
Recall that HE- Tr-Transformer (Boemer et al. 2019b a) provides two secure prediction schemes, i.c., a pure HE-based scheme and a hybrid scheme combining HE and GC. Given that the HE- Transforfrr framework is employed in CaPC, we design two customized protocols for HE-Transforme to achieve such communication requirement. Although is trivial to extend CaPC equipped with the HE-based scheme to the above communication-linne limited settings, it has two key problems: 1) activa- tion functions need to be approximated as low-degree polynomials, which leads to serious accuracy loss; 2) due to the inherent high computational complexity, HE-based secure prediction is difficult to extend to large-scale models. For completeness, we briefly describe the extension procedure. Pq first encrypts the query samples and asks the server pass them to PA- After that, PA evaluates cure prediction non-interactively in the ciphertext environment. Finally, PA sends encrypted masked predictions (due to privacy-prect peeeccypeessrinng aggregation in CaPC) back Po via the server acting intermediary. As mentioned above, the HE-based method is impractical and below, we elaborate on the extension of CaPC that uses the hybrid scheme as building block. In CaPC, secure predictions are executed between Po and PA. To tackle the communication limitation, we can employ secure predictions between the server and PA- We discuss the modified algorithms of the linear layer and the non-linear layer separately. In the linear layer, 1) Po encrypts query samples with HE and sends the ciphertext to PA through the server 2) PA evaluates linear layers locally, such as convolution and full-connected layers, and returns the encrypted masked results to Pg through the server. 3) Po decrypts to obtain the masked results and sends it to the server. As result, the results of linear layers are shared between the server and PA. For the non-linear layer, given that the server and PA hold shares of the linear layer's 's results, the two parties (rather than PA and Pq in CaPC) call the GC protocol to evaluate the nonlinear layer function.
Next, we analyze the efficiency of the modified hybrid protocols. For the computational cost, the modified secure prediction protocol is exactly the same as that of CaPC, and thus we mainly focus on the communication cost. In the linear layer, the modified protocol adds the communication overhead of a ciphertext (Iwo ciphertexts for the input layer) and a plaintext of the result of the linear layer. For the non-linear layer, the communication overhead is not increased, but the overhead between PA and Po is transferred to the server and PA- In summary, although CaPC can be extended to scenarios with limited communication, they sacrifice the efficiency of secure predictions. Therefore, PrivHFL shows better adaptability and efficiency in scenarios where there is no direct communication between clients.
C.6 GPU ACCELERATION
Existing general-purpose platforms on the GPUs such as NVIDIA's CUDA are designed to operate on floating-point inputs (Tan et al. 2021). On the contrary, the shares in our protocol is embedded in the ring Zze4. Thus, we need to convert the ring operations into 64-bit floating point operations to obtain GPU support. Specifically, a 64-bit floating point number has 1 bit sign, 11 bits exponent, and 52 bits precision. which can exactly represent all integers in the interval [-252, the multiplication E F can be correctly computed and recovered over the integers if and only if E, F E 2 n 122 2226]. Based on this observation, we decompose each input E and F in Eges into blocks, and the values in each block are represented by 16-bit value. Thus, the integer multiplication is converted to 16 (exactly, 10) floating point operations. Specifically, E is rewritten P1 + 232 F2 2 Fz.. The multiplication can as Eo + 2-6 E, + 2 E2 + 248 Eg, and F Fu + 2 be represented as follows:
Therefore,
E. F = (Ea + 2 E1 + 2 Ee + 2 Ea) (Fo + n0 Fi + 22 Fiz + 2dh Fa)
= Fo + 216 F, Fo + 2 Fo 2 Fo + 216 F, Fro + 22 F, Fa + 2 F. F2
(6)
+ 2 E2 28 F2E: + 2*h F3E
'To be more precise, this step is for the input layer. In the hidden layer, one of the input shares of the linear layer should be enerypted by the server and sent to PA-
21

---

Under review as a conference paper at ICLR 2022
After that, computing E F from the pairwise products requires element wise additions and scalar multiplications, which can be executed by optimized CUDA kernels on 64-bit integer values.
D RELATED WORK
Table Comparison with prior works on properties necessary for federated learning
Privacy
Usability
Efficiency
Framework
Data Privacy
Model Privacy
Model Heterogeneity
Dataset Dependency
GPU
Protocol Efficiency
Compatibility
Bonawitz
2017
Bell et al.
2020
x
Sav et al.
2021
/
X
x
Jayaraman Wang
2018
/
X
X
/
Li & Wang
2019
V
V
X
Choquette-Chon ct al.
2021
/
X
x
X
Lin al. 2200
X
Sun & Lyu
2021
/
V
Diao ct al.
2021
X
This work
D.1 HETEROGENEOUS FEDERATED LEARNING
Federated learning achieves collaboration among clients via sharing model gradients. While suc- cessful, it still faces many challenges, among which, of particular importance is the heterogeneity that appear in all aspects of the learning process. This consists of system heterogeneity (Diao et al,
2021]. model heterogeneity (Li & Wang 2019) and statistical heterogeneity (Zhu al. 2021 Statistical heterogeneity means that clients' data from real-world comes from distinct distributions (i.e., Non-IID data), which may induce deflected local optimum. Solving the statistical heterogene- ity has been extensively studied, such as Dinh et al. (2020); Zhu et al. (2021); (Yurochkin et al. (2019); Fallah al. (2020); Yoon al. [2021 and out-of-the-scope of this work. However, our PrivHFL may help alleviate the statistical heterogeneity due to customized model design and knowledge distillation based aggregation rule.
Our work mainly focuses on model heterogeneity that has been explored in recent works Li & Wang 2019, Lin et al. 2020) Choquette- Chhoo al. (02T), while the issue of system heterogeneous alleviated through the resource-customizzr model architecture design. In particular, [i & Wang (2019) proposed the first federated learning framework FedMD supporting heterogeneous models by combining transfer learning and knowledge distillation techniques. They first used a public dataset to pre-train the model and transferred to the task of private dataset. After that, to exchange the knowledge, each client used the public data and the aggregated predictions from others carrier for knowledge distillation. To further improve test accuracy, Lin et al. (Lin et al. 2020) proposed FedDF, similar FedMID, which also used model distillation technique for knowledoe sharing. The difference that they first performed FedAvg on clients' local models and integrated knowledge distillation on the aggregated model. The dependence on model averaging leads to limited model heterogeneity. Besides, Diao et al. (Diao et al., [202]) focused on heterogeneous clients equipped with different computation and communication capabilities. In their framework, each client only updated subset of global model parameters through varying the width of hidden channels, which reduces the computation and communication complexity of local models. However, this approach only learns single global model, rather than unique models designed by clients. Moreover, the above three methods rarely consider the issue of privacy leakage from the prediction results.
The privacy protection techniques (i.e., secure aggregation) have been studied in federated learning [Bonawitz et al.) 2017 Bell al.) 2020 Sav et al. 2021 Jayaraman & Wang 2018). However, HFL. More recently, Sun & Lyu
these techniques not be directly extended to
TThis work also uses mixup federated learning, but aims address the challenge of non-iid data.
22

---

Under review as a conference paper at ICLR 2022
proposed noise-free differential privacy solution for heterogeneous federated learning to guarantee each client's 's privacy. However, as shown in Jayaraman & Evans (2019). there is huge gap between the upper bounds on privacy loss analyzed by advanced mechanisms and the effective privacy loss. Thus, differentially private mechanisms offer undesirable urility-privacy trade-offs. To further formally guarantee the privacy, Chhqueete-
2021
et al.] (202T) leveraged secure multi- party computation (MPC), homomorphic encryption (HE) and privately aggregated teacher models techniques to realize confidential and private collaborative learning. Specifically, clients learn from each other collaboratively utilizing secure inference strategy based on MPC and HE protocols and a private aggregation method. As noted in the Introduction, CaPC's use of heavy cryptography leads to significant efficiency and communication overheads. Besides, existing methods require an auxiliary dataset to implement heterogeneous federated learning. However, data collection could be unrealistic in many real-world scenarios due to various reasons, such as privacy concerns and rare classes. Therefore, our work is dedicated to solving two key challenges. The first is to relax the assumption of relying on public datasets, and the second is to design an efficient cryptographic protocol for knowledge transfer during the model aggregation.
D.2 PRIVATE NEURAL NETWORK PREDICTION
Neural networks present a challenge to cryptographic protocols due to their unique structure and ex- ploitative combination linear computations and non-linear activation functions. In real scenarios, model inference can be viewed as a two-party computation case, where one party with private data wants to obtain prediction results from the other party who owns the model. During the whole pro- cess, the cryptographic protocols, typically HE and MPC, are applied to ensure the confidentiality of the private data and model.
Existing works (Boemer al. 2019b Galha-Bachrach et al. 2016 Brutzkus et al. 2019) support pure HE protocols for secure predictions. Typically, nGiraph-Hl HE (Boemer et al. 2019b a allows linear computations using CKKS homomorphic encryption scheme. However, since a solution that builds upon HE protocols should be restricted to compute low degree polynomials, the non- polynomial activation functions, such as MaxPooling and ReLU, are forced be evaluated in the clear by the party who owns private query data. This leaks the feature maps, from which adversaries may deduce the model weights. To solve this problem, Gilad Bachrach et al. (2016) and [Cen et al.) (211)) use low-degree polynomial approximation to estimate non-linear functions. Unfortunately, it will affect the accuracy of predictions, while leading to huge computation overhead.
On the other hand, server libraries (Mohassel & Zhang 2017 Knott et al. 2021 Wagh et al. 2019
Shen et al., 2020) employ primarily MPC technology in secure predictions, which provides support for linear and non-linear activations through the use of oblivious transfer (OT), garbled circuit (GC) and secret sharing. For example, CryptTen (Knott et al., 202T) performs linear operations based on n-out-of-n additive secret sharing over the ring, Zzar. However, it uses boolean secret sharing for the non-linean operations, which will result in a higher communication round. CrpytGPU (Tan et al. 2021) builds on CrypTTen, working in a 3-party setting using replicated secret shares. Although the scalability is poor, it introduces an interface to losslessly embed cryptographic operations over secret-shared values in a discrete somain into floating-poin operations, which can implement the whole inference process on the GPU.
Many other works focus on hybrid protocols, in which they combines the advantages of HE and MPC to improve prediction efficiency (Juvekar et al.) 2018 Mishra et al., 2020) Rathee et al. 2020). CrypTFlow2 (Rathee et al., 22000) points out that currently, was not clear whether Hasbased linear operations would provide the best latency. Therefore, the authors implement two class of protocols, HE-based and OTT-based, for linear operations. For non-linear layers, they also design efficient protocols based on OT. It turns out that in WAN setting, HE-based inference is always faster and in LAN setting OT and HE are incomparable. HE-transformer employs nGraph- for evaluation of linear operations, and ABY framework (Demmler et al. 2015) for GC to evaluate non-linear functions. Since non-linear operations cannot be parallelized between query data, GC is inefficient, especially for large networks with thousands of parameters. In contrast, our PrivHFL avoids the use of heavy cryptographic tools, and only employs additive secret sharing to achieve high efficiency, confidentiality and practicability.
23

---

Under review as a conference paper at ICLR 2022
E SECURITY ANALYSIS
Our security proof follows the standard ideal-world/real-woor paradigm (Canetti, 2001 in real- world, three parties interact according to the protocol specification, and in ideal-world, they have access to ideal functionality. When protocol invokes another sub- sub-protocol, we use F-hybrid model for security proof by replacing the sub-protocol with the corresponding functionality. Note that our proof works in the FPRc-hybrid model where FrRG represents the ideal functionality cor- responding to the protocol PRG. The executions in both worlds are coordinated by the environment Env, who chooses the inputs parties and plays the role of distinguisher between the real and ideal executions. We will show that the real-world world distribution is computationally indistinguishable to the idral-world distribution.
Theorem 1. Share securely realizes the functionality Fstare in the FrRe-hybrid model.
Proof. Note that Po and PA receive no messages in the protocol, and hence the sharing protocol is trivially secure against corruption of Po and Pa. Next, the only message that the server receives the value [x]] However, [xr] I - r. where given the security of PRG, r is a random value unknown to the server. Thus, the distribution of [x)] is uniformly random from the server's view and the information learned by the server can be perfectly simulated.
Theorem 2. Matmuul securely realizes the functionality JMatmal the FFRG-hybrid model.
Proof. Note that Pq receives no messages in the protocol, and hence the sharing protocol is trivially secure against corruption of Pq. The only message that PA receives is the value [x]] - 6. However, given the security of PRG, b is random value unknown to P'. Thus, the distribution of [a]] b is uniformly random from PA 's view and the information learned by PA can be perfectly simulated. Next, during the protocol, the server learns [c]1 and + However, the distribution of [c]1 and w + a is uniformly random from the server's view, since given the security of PRG, and [c] are random values unknown to the server. Thus, the information learned by the server can be perfectly simulated.
Theorem 3. I.DELLU securely realizes the functionality FDRELU in the FPRG-hybrid model.
Proof. Note that PA receives no messages in the protocol, and hence the sharing protocol trivially secure against corruption of PA- Next, the messages that the server receives are [z]o and sign(=)- 8'. However, given the security of PRG, 8 and 8' are random values unknown to the server. Thus, the distribution of [æ]o- ö and sign(z) 8' is uniformly random from the server's view and the information learned by the server can be perfectly simulated. Then, the message that Pg learns rx, where is random number in the ring. Thus, the distribution of is uniformly random from the server's view.
Theorem 4. URRL securely realizes the functionality FRELU the (FMatmul- FDR. model.
Proof. Note that as shown in Figure 4 TRRLL consists of ITreL and Ma Therefore, the ReLU protocol is trivially secure in the (FMatmuls -oRrcu)-hybridd model.
Theorem 5. Maxaoool securely realizes the functionality FMaxPool in the FRaLU-hybrid model.
Proof. Note that as shown in Figure I consists of IRRLA. Therefore, the MaxPool protocol is trivially secure in the FReLU-hybrid model.
Theorem 6. llage securely realizes the functionality FAge in the FPRG FPRG-hybrid model.
Proof. Note that PA receives no messages in the protocol, and hence the aggregation protocol is trivially secure against corruption of PA- Next, the only message that the server receives is the value [zylo - ry- However, given the security of PRG. ry is random value unknown to the server. Thus, the distribution of [æ -Ty is uniformly random from the server's view and the information learned by the server can be perfectly simulated. After the aggregation, Pq only learns the aggregated result
24

---

Under review as a conference paper at ICLR 2022
Table 7: Experiment setting of different datasets during local model training.
Loss Function
Learning Rate
Batch Size
Epoch
Number of Private Data
SVHN CIFAR 10 Tiny ImageNet
cross-entropy cross-entropy cross-entropy
0.5
256
250
1465
0.1
64
500
1000
0.01
64
500
10000
EjE(n] Zy. but is unknown to a single zy. Therefore, the aggregation protocol is secure assuming the aggregation result will not reveal privacy.
ADDITIONAL. EXPERIMENTAL SETUP
Datasets. We evaluate PrivHFL on the following standard datasets for image classification:
SVHN. SVHN is real-world image dataset obtained from house numbers in Google Street View images. Each sample is 32x RGB image. We use 73257 digits for training and 26032 digits for testing.
CIFAR1O. CIFAR-10 consists of 60,000 x2x32 RGB images in 10 classes. There are 50,000 training images and 10,000 test images.
Tiny ImageNet. Tiny ImageNet contains 100,000 images of 200 classes (500 for each class) downsized to 64x64 colored images. Each class has 500 training images, 50 valida- tion images and 50 test images.
Model architecture. We use VGG-7, ResNet-8 and ResNet-10 for SVHN and CIFARIO datasets. VGG-7 contains 6 convolutional layers, followed by fully-connected layer, where ReLU is ap- plied as the activation function and MaxPooling is used for downsampling, giving 7 layers in total. ResNet-8 use follows the same architecture in CaPC, which consists of convolutional layer, residual blocks with 4 convolutional layers, followed by fully-connected layer. Compared with the original architecture, the last block is excluded and neurons of the last layer is increased. ResNet- 10 begins with a convolutional layer, followed by residual blocks with 2 convolutional layers each block, and one final fully-connected layer, thus there are 10 functional layers overall. Besides, we use ResNet-14, ResNet-16 16 and R8 (He et al. 2016 for Tiny ImageNet.
Training procedure. Utilizing the private data, each client first trains the local model from scratch using SGD optimizer, where the detail training setting is shown in Table Next, the clients run the PrivHFL protocol to generate query-response pairs, which will be used retrain the local model. When retraining the models, they use Adam optimizer for 50 epochs with learning rate of 2e-3 decayed by factor of 0.1 on 25 epochs, where the batch size is 256 on SVHN and 64 on CIFARIO and Tiny ImageNet.
G VISUALIZATION OF MIXUP AND SECRET-SHARING SAMPLES
The mixup-synthetic data expansion method of PrivHFL is only to expand private data locally to generate large data pool, which cannot provide any privacy guarantee. Figure 19 shows the mixup images synthesized from the same pair of real images with different mixup coefficients A. As shown in the figure, when À 1, the mixup image will be identical to the real sample. Even if 0.7, 0.5, 0.2, the original images can still be visually recognized.
To further protect the Po's privacy, the mixup images are secret-shared between the server and PA rather than being disclosed directly to them. As shown in Figure 20 the secret-shared images look like two random noises, since in secret sharing each share is randomly sampled from the ring Ezer. To be specific, for mixup image sr the server owns 11, in which each element is randomly sampled from the ring Zga4, Meanwhile, the answering party PA owns a random 2g with xp = x--1 E Zzas. Besides. note that in our secure querying protocol, the intermediate values always maintain such secret sharing invariant, so that neither party can steal any private information about real datasets, local models and prediction results from the knowledge they obtain.
25

---

Under review as a conference paper at ICLR 2022
1-2=0
1-2= 0.3
1-2= 0.5
1- 0.8
Figure 19: Different mixup-synthetic images from the same pair of the natural images by vary- ing the mixup coefficient X.
Po
Server
Pa
x
X1 E Zza4
Xg = x - X, E Zyes
Figure 20: An example of secret sharing on the mixup image. x, 31 and Do are pixel matrices, owned by the querying party PQ. the server, and the answering party PA. respectively.
26


Reviews:
[{"review_id": "zC_uACya6cW", "reviewer": "Reviewer_ZGSK", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors proposed to use synthetic dataset generated using the Mixup method. I may have the following questions regarding this approach:\n1. How can we guarantee that no privacy information will be breached using this synthetic method? \n2. Will there be possibility to reconstruct the original dataset from this simple mixture? For example, a mixup will be identical to a real sample when 𝛌 = 1. Will this still leak the privacy of local data?\n", "summary_of_the_paper": "The paper showed a new approach for heterogeneous federated learning, which uses augmented dataset instead of a public dataset for knowledge transfer between heterogeneous models. The authors also suggested a lightweight additive secret sharing technique to construct a series of tailored cryptographic protocols, which is friendly with GPU and the CUDA kernels.\nThe method was evaluated in a simulated scenario with three public imaging datasets. And shows superiority over the baseline methods, and also efficiency with regard to run time and communication cost.\n", "summary_of_the_review": "I feel not fully confident to assess the quality of the work as I am non-expert in federated learning.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "wHq45hwZk9H", "reviewer": "Reviewer_kXk2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strong points:**\n\n1. The improvement in efficiency against the previous methods, e.g., CryptGPU or CaPC.\n\n2. Good design. PrivHFL builds on top of CrytpGPU to compute over integer values (64 bit ring elements). CryptGPU embeds the integer-valued cryptographic operations into floating-point arithmetic.\n\n**Weak points:**\n\n1. **Private data revealed in plain text if $P_Q$ colludes with the server**. The revealed data is an augmentation of the private dataset using MixUp but it still leaks information about the private data. In contrast, e.g., in CaPC, once the server (the 3-rd trusted party - Privacy Guardian) and answering parties collude, only the predicted label is revealed but not the private data.\n\n2. **The logits are aggregated** (as described in Figure 5) but these values are not normalized across models so each model can have a very different impact on the final predictions. This makes the method more vulnerable to attacks.\n\n3. **MixUp was already used for Federated Learning in [1]**.  Yoon et al. propose to apply MixUp to Federated Learning for addressing the challenge of non-iid data.\n\n4. MixUp is not a state-of-the-art (SOTA) data augmentation method. Could MixMatch [2] (it also shows improvements for MixUp) or FixMatch [3] or other SOTA be used? The other data augmentation methods from Figure 10 are rather rudimentary. \n\n5. **No DP provided**. The revealed logits leak information about private data from the answering parties ($P_A$).\n\n6. **Intermediate representations are released to $P_Q$**. From Section 3.3 \"After that, $P_A$ and the server disclose the secret-shared value [z] to $P_Q$, which then computes the sign of z, i.e., the sign of x. sign(z) is shared to the server and $P_A$ by using the protocol in Figure 2.\" Figure 4 shows that scaled intermediate results (inputs to the ReLU layers): $z = r([x]_1 + [x]_2)$ are revealed to the querying party ($P_Q$). How much can $P_Q$ infer about the model from these intermediate results? It seems that $P_Q$ can infer the model architecture that is used by $P_A$. Is z some scaling of the original intermediate representation, or every value from the intermediate representation can be scaled differently to obtain the representation sent to the querying party. The latter might make the method practically robust since the querying party is rather not able to infer the parameters of the model used by the answering party. However, this should be confirmed formally. In the former case (i.e., z is only a scaled intermediate representation - input to ReLU), the parameters of the model used by $P_A$ might be extracted from z. However, it is claimed at the end of Section 3.3 that: \"We give a formal security proof in Appendix E. Intuitively, PrivHFL reveals zero information to the answering parties $P_A$ and the server, and only reveals the final aggregated prediction to the querying party $P_Q$, since all intermediate values are secret-shared. Given the above, a corrupted $P_A$ cannot learn anything about the query data of querying parties, while the confidentiality of responding parties’ model parameters against corrupted $P_Q$ is also protected.\"\n\n7. **No assumption on a public dataset**. For example, CaPC does not assume that there is a public dataset, thus the following is not correct: “Besides, to the best of our knowledge, **existing methods all require a public dataset** to implement heterogeneous federated learning.” Since the data is always encrypted in CaPC, the querying party can send the private data with a guarantee that it is not revealed to any other party. Additionally, CaPC can serve as a private consultation protocol. The natural new data to be labeled come from new patients or clients (in the case of hospitals or banks, respectively). \n\n8. **No direct communication between clients.** This is easy to fix. For instance, CaPC can be extended to a setting where there is no direct communication between clients. If we have parties P1, P2, P3. P1 and P2, as well as P1 and P3 can communicate directly. They just need to encrypt their messages and ask P1 to pass them along.\n\n9. CaPC is a modular protocol. CryptGPU, which has been published recently (after CaPC), could replace HE-transformer to accelerate the private inference while maintaining CaPC's strong privacy and confidentiality guarantees.\n\n10. Active learning was also used in CaPC.\n\n**Questions:**\n1. Why is scalability an issue (in terms of the number of clients and size of the models)? Could you elaborate on it, please? \nFigure 6: How many models have which architecture? \nTable 1: For “Private data” is the assumption for the baseline that there are not labels for the private data and they are obtained by querying the answering parties?\n2. How is the whole private inference implemented across all the layers? PrivHFL does not provide the source code. It would be informative to see how really the additive sharing is used for private inference and when the intermediate results are revealed to the querying party $P_Q$. \n3. Should not DReLU be defined as: $\\frac{1}{2}*(1 + sgn(x))$ ?\n\n**Comments:**\n1. Section D.1 is a repetition of the introduction.\n\n**Typos:**\n\ninferior than -> inferior to (Section 3.2)\n\nexperiment setup -> experimental setup (Section 4.1)\n\nthey \"are\" most uncertain (Section C.1.)\n\nSuggestion: \"number of private data\" -> \"number of private data samples\"\n\nthey using -> they use (Section F)\n\n\\citep instead of \\cite or \\citet should be used in most cases (e.g., the first paragraph in the Introduction).\n\n**References:**\n\n[1] FedMix: Approximation of MixUp under Mean Augmented Federated Learning. Tehrim Yoon, Sumin Shin, Sung Ju Hwang, Eunho Yang. (ICLR 2021).\n\n[2] MixMatch: A Holistic Approach to Semi-Supervised Learning. David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, Colin Raffel. arxiv:1905.02249 (NeurIPS 2019)\n\n[3] FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence. Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Han Zhang, Colin Raffel.  (NeurIPS 2020)\n", "summary_of_the_paper": "PrivHFL is a protocol for collaborative learning that does not reveal the private data to the server or answering parties $P_A$ if they operate as honest-but-curious entities. PrivHFL tackles the bottleneck of private inference by building a new protocol from scratch that can run on GPU. It is not assumed that parties have new data or that there is a public pool of data to run inference on but the additional query data for knowledge transfer is obtained by augmenting the private data with MixUp.", "summary_of_the_review": "The paper shows much better performance (e.g., much faster execution of the private inference) but this is at the cost of lowering the security level. Additionally, the aggregation through logits makes the method more vulnerable to attacks than other proposed protocols, such as, CaPC. MixUp and Active Learning methods were already used in federated learning (FedMix, CaPC) so this is cannot be classified as a technical novelty.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "hfj7WJwHjSl", "reviewer": "Reviewer_3JAx", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Query data generation: It is not clear how $\\lambda$ is selected?  Is there any theoretical justification for the claim that \"this methods provides a good coverage of the manifold of natural samples\"? Given two private samples, the authors generate multiple query samples. It is not clear how the private samples, $\\lambda$, and the number of synthetic samples for each given pair of private samples should be selected to achieve the optimal coverage of natural distributions. \n\nQuery-data sharing:  It is not clear how $Sk_{QA}$ can be constructed without direct communication channels between clients and considering that each client can play the role of both querying party and responding party at the same time (possibly responding to multiple queries)? \n\nResult aggregation: It is assumed that all clients participate. This is not the typical scenario in federated learning. In addition, a unique $r_j$ is generated for each client, which is not consistent with other steps of the algorithm. \n\nExperiments: How much will the accuracy degrade if you set secret-sharing protocols over other rings? \n\nAlgorithm 1: Why do you need line 5 while each client already trained local models on its private dataset? \n\nMinor comment: \n\nOn the RHS of Figure 4, $\\delta'_i$ should be $\\delta'$\n\n", "summary_of_the_paper": "This paper introduces a heterogeneous federated learning framework without requiring public datasets by designing a dataset expansion method and constructing cryptographic protocols for secure prediction. The main strength of this paper is regarding the experimental results. The authors have provided experiments on different datasets, heterogeneous local models, various degree of non-IID-ness across clients, and showed the runtime of three steps in their proposed secure querying protocol.   ", "summary_of_the_review": "This paper is well-written and provides extensive experiments which show the efficiency of PrivHFL and accuracy gains compared to a baseline, which uses models trained on the local datasets. Some aspects of the dataset expansion method and  the secure querying protocol are unclear, which require further clarifications. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: A heterogeneous federated learning framework is proposed which does
not require auiliary public data sets, and does not reveal the private
data to the server or answering parties if they operate as
honest-but-curious entities. It builds a new protocol for private
inference, which can run on GPUs, and proposes a dataset expansion
method to not need an auxiliary data set. The paper presents extensive
empirical experiments on the method.

The paper was extensively discussed with the authors. The concerns
included both technical issues and more general issues on missing DP
guarantees and realisticness of the threat model. Many of the issues
were resolved by the clarifications provided by the authors, and as a
result two reviewers increased their scores. However, all reviewers still
place the paper to the borderline.

While the paper contains solid work, and improves efficiency compared to
previous models, this is a borderline paper where the final judgement
needs to be based on importance of the presented new contributions in
advancing the field. The paper may not yet quite reach the bar, but
I believe the reviewer comments have enabled the authors to improve the
paper for further work.
Decision: Reject