GuardHFL: Privacy Guardian for Heterogeneous Federated Learning
Authors: Hanxiao Chen
Meng Hao
Hongwei Li
Guangxiao Niu
Guowen Xu
Tianwei Zhang
Xilin Zhang
Venue: Submitted to ICLR 2023 (2023)

Abstract: Heterogeneous federated learning (HFL) enables clients with different computation and communication capabilities to collaboratively train their own customized models via a query-response paradigm on auxiliary datasets. However, such paradigm raises serious privacy issues due to the leakage of highly sensitive query samples and response predictions. Although existing secure querying solutions may be extended to enhance the privacy of HFL with non-trivial adaptation, they suffer from two key limitations: (1) lacking customized protocol designs and (2) relying on heavy cryptographic primitives, which could lead to poor performance. In this work, we put forth GuardHFL, the first-of-its-kind efficient and privacy-preserving HFL framework. GuardHFL is equipped with a novel HFL-friendly secure querying scheme that is built on lightweight secret sharing and symmetric-key techniques. Its core is a set of customized multiplication and comparison protocols, which substantially boost the execution efficiency. Extensive evaluations demonstrate that GuardHFL outperforms the state-of-the-art works by up to two orders of magnitude in efficiency.

Under review as a conference paper at ICLR 2023
GUARDHFL: PRIVACY GUARDIAN FOR HETEROGE- NEOUS FEDERATED LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Heterogeneous federated learning (HFL) enables clients with different computa- tion and communication capabilities to collaboratively train their own customized models via a paradigm raises serious privacy issues due to the leakage of highly sensitive query samples and response predictions. Although existing secure querying solutions may be extended to enhance the privacy of HFL. with non-trivial adaptation, they suffer from two key limitations: (1) lacking customized protocol designs and (2) relying on heavy cryptographic primitives, which could lead to poor performance. In this work, we put forth GuarcHFL. the first-of oft----iii-- efficient and privacy- preserving HFL framework. GuardHFL, is equipped with novel HFL- secure querying scheme that is built on lightweight secret sharing and symmetrie- key techniques. Its core is a set of customized multiplication and comparison protocols, which substantially boost the execution efficiency. Extensive evalua- tions demonstrate that GuardHFL outperforms the state-of-the-arr works in both runtime and communication overhead.
paradigm auxiliary datasets. However, such
INTRODUCTION
As a promising variant of federated learning (FL.), heterogeneous federated learning (HFL.) (Li & Wang, 2019) enables clients equipped with different computation and communication capabilities to collaboratively train their own customized models that may differ in size, numerical precision or structure (Lin et al., 2020). In particular, the knowledge of models is shared via a query-response paradigm auxiliary datasets, such as unlabeled datasets from the same task domain (Choquette- Choo et al., 2021) or related datasets from different task domains (Li & Wang. 2019; Lin et al., 2020). In such paradigm, each client queries others with samples in the auxiliary querying dataset, and obtains aggregated response predictions via a centralized cloud server'. Then he retrains his local model on the query data and corresponding predictions. This flexible approach facilitates customized FL-driven services in areas like healthcare and finance (Kairouz et al., 2019), while resolving the intellectual property concerns FL models (Tekgul et al., 2021).
However, HFL suffers from several privacy issues. First, directly sharing query samples violates their privacy. For example, in healthcare applications, the auxiliary datasets may contain patients' medical conditions. Disclosure of such information is illegal under current regulations like General Data Protection Regulation. Second, sharing response predictions may still compromise the privacy of local data (Papernot et al., 2016). Several works have shown that given black-box access to model, adversaries can infer the membership (Salem et al., 2019) and attribute information (Ganju et al., 2018) of the target sample even reconstruct the original training data (Yang et al., 2019).
Although traditional FL systems, the privacy issue could be mitigated through well-studied secure gradient aggregation protocols (Bell et al., 2020), it becomes more challenging to realize this guar- antee in HFL, due to the heterogeneity of the clients' models (refer to Appendix A.2.3). To bridge this gap, possible solution is to structurally integrate into HFL existing secure querying (a.k.a. private inference) schemes (Rathee et al., 2020; Huang et al., 2022; Wagh et al., 2019; Tan et al., 2021). These schemes utilize various cryptographic primitives, including homomorphie encryption
As demonstrated in Bonawitz et al. (2017); Bell et al. (2020), clients (e.g., mobile devices) in real-world applications are generally widely distrihuted and coordinated only by the server.

---

Under review as a conference paper at ICLR 2023
(HE) (Gentry, 2009), garbled circuit (GC) (Yao, 1986) oblivious transfer (OT) (Asharov al., 2013), to provide rigorous privacy guarantees for query data and prediction results. While it is possi- ble to non-trivially extend these secure querying schemes (refer to Section 2.3), they have two major limitations: (1) the underlying protocols are not customized for HFI.; (2) they incur huge overhead due to the adoption of heavy cryptographic primitives. These bottlenecks lead to poor performance and hinder the efficient instantiation of HFL. Therefore, it is necessary but challenging to provide customized protocols and implement privacy-preserving HFL with desirable performance.
In this work, we introduce GuardHFL, L, the first efficient and address the above challenges?. GuardHFL is built upon the standard HFL training paradigm (Li & Wang. 2019). which contains three stages: local training, querying and local re-training (refer to Section 2. We formalize the workflow of HFI,, and present novel HFIL-friendly secure querying scheme as an important building block. The core of our scheme is a set of customized multiplication and comparison protocols, which substantially boost the execution efficiency compared to existing works. More precisely, we optimize the parallel prefix adder (PPA) logic (Harris, 2003) to build more advanced comparison protocol. Over an E-bit ring, these optimizations reduce the evaluation of log AND gates and also the number of the communication rounds, which are two crucial factors that dominate the performance. Moreover, our PRF-based multiplication protocol only communi- cates 3 elements in an é-bit ring, achieving significant improvement compared the widely used Beaver's multiplication solution (Beaver, 1991; Demmler et al., 2015). We provide formal privacy guarantees for the designed protocols, and evaluate Guar on different datasets (SVHN, CI- FARIO, Tiny ImageNet), system configurations (IID and Non-IID training sets) and heterogeneous models. Results show that GuardHFL outperforms the sfate-of-the-art works in efficiency while ensuring the model utility.
HFL framework to
2
BACKGROUND
2.1
HETEROGENEOLS FEDERATED LEARNING
We briefly review the workflow of HFL (Li & Wang, 2019), where clients independently design their own unique models. Due to such model heterogeneity, clients cannot directly share model parameters with each other as in the traditional FL. Instead, they learn the knowledge of other models via query-response mechanism, which is similar as the knowledge distillation technique (Hinton et al., 2015). To be more precise, each client PQ (called the querying party) performs three- phase operations collaboratively with a server: (1) Local training: PQ first trains the local model on his private dataset. (2) Querying: The server selects C fraction of clients the responding parties PA. Given the auxiliary querying dataset, the server receives the prediction results from these PA. computes the aggregated result and returns it back to Pq- (3) Local re-training: Po then retrains the local model based on the private dataset, as well as the query samples and corresponding predictions. Each client in HFL can play the roles of the querying party and the responding party at the same time, and the above process is iterated until each local model meets the pre-defined accuracy requirement. Note that as illustrated in existing works (Bonawitz et al., 2017: Bell al., 2020), the server is responsible for routing the messages between clients, since the clients (c.g., mobile devices) generally cannot establish direct communication channels with others.
GuardHFL L is in line with the above paradigm with the additional benefit of privacy protection. The only difference lies in the acquisition of auxiliary query samples in the querying stage. In general HFL (Li & Wang, 2019), there is large public auxiliary dataset (used as query samples) that every party can access. However, considering the privacy limitation, such public dataset is hard to collect in real-world scenarios such as healthcare. To tackle this problem, in Gua rdHL, each party can locally construct a synthesized querying set based on its private training samples, by utilizing existing data augmentation strategies (refer to Section 3.4).
2.2 THREAT MODEL
As described in Section 1, in the querying phase of HFL, the query samples, prediction results and model parameters may contain sensitive information that is of interest to adversaries. In line
Choquette- Chho et al. (2021) presented a general collaborative learning scheme, called CaPC, which en- ables each party to improve his local model from others' models using the secure querying scheme (Boemer al., 2019b). However, it cannot be directly applied to the HFI. scenario as it requires cross-client communi- cation. Mcanwhile, causes prohibitively high overhead (refer Section 4.1).
2

---

Under review as a conference paper at ICLR 2023
Secure querying
Local data
Query data
Server
Hederagenoous models
Train
Retrain
Querving party
Responding parties
Label
Prediction
Figure I:
High-level view of GuardHFL
with prior works (Phong et al., 2018; Sun & Lyu, 2021; Choquette-Choo et al., 2021), we consider an conest-but-curious adversary setting (Goldreich, 2009), where each entity (including the clients and the server) strictly follows the specification of the designed protocol but attempts to infer more knowledge about these private information of other clients. Moreover, to maintain the reputation and provide more services, the server does not collude with any clients, namely that attacker either corrupts the server or a subset of clients but not both.
2.3 EXTENDING EXISTING SECURE QUERYING SOLUTIONS TO HFI.
To provide privacy guarantees against adversaries in Section 2.2, the clients and the server need to privately execute the querying process. Although this process consists of three entities (i.e., Po. the server and PA), is non-trivial to directly extend existing secure 3-party computation proto- cols (3PC) (Wagh et al., 2019; 1021; Knott et al., 2021; Tan et al., 2021) to instantiate this process. The main reason is the incapability of direct communication between Po and PA in realistic HFL scenarios (Bonawitz et al., 2017: Bell et al., 2020), which hinders the usage of these 3PC solutions HFL, unless we redesign the underlying protocols and make substantial modifications to their corresponding implementations. On the other hand, we can extend state-of-the-arr 2PC solutions (Rathee et al., 2020; Huang et al., 2022) into this process via using the server as the communication medium with adaptive protocol modifications (refer to Appendix A.2.4 for more details). Unfor- tunately, as mentioned in Section 1, such extension comes at the cost of heavy computational and communication complexity. Motivated by these challenges, we design set of lightweight and cus- tomized protocols for improving the efficiency of the secure querying phase (Section 3), which show significant performance gains over extending the advanced 2PC schemes to HFL (Section 4.1).
2.4 CRYPTOGRAPHIC PRIMITIVES
Secret sharing. GuardHFL adopts the 2-out-of-2 arithmetic secret sharing scheme over the ring Zoz (Shamir, 1979; Demmler et al., 2015). Share(z) denotes the sharing algorithm that takes a as input and outputs random sampled shares [z]o.- [z]] with the constraint
[z]o [x]1 in Zze. The
reconstruction algorithm Recon ([x)). [xr).. [z]]1) takes the two shares as input and outputs 2 = [z]o + [x]1 in Zzr. Besides, our comparison protocol adopts the boolean secret sharing (Shamir, 1979; Demmler
et al., 2015), where x E Zz is shared as [x:]6] and [x:] B satisfying [x:] @ [x: B = x in Zz. The security ensures that given [z:) or [x]] (similarly, [xr]6 [z] the value of is perfectly hidden. Arithmetic operations on secret-shared values can be implemented with existing techniques (Appendix A.2.2).
Pseudo-random Function. A pseudo-random function y PRF(Sk;, 2)) is deterministic function that takes a uniformly random seed Sk and payload x as input and outputs a fixed-length pseudo- random string y. The security of PRFs ensures that the output indistinguishable from the uniform distribution. In GuardHF L, PRFs enable two parties to generate the same pseudo-random values without communication. Details can be found Appendix A.2.2.
3 GUARDHFL
GuardHFL is built upon standard HFL systems shown in Section 2.1 and enhances their privacy protections with cryptographic techniques. Figure 1 shows the overview of GuardHFL, and the detailed description given in Algorithm 1. Similar as vanilla HFL, it includes three phases: Local training, Secure querying and Local re-training. Since Local training and Local re-truining are standard HFL. training processes without privacy issues, below we focus on formalizing our core construction, i.c., Secure querying. As detailed in Section 2.3, extending existing secure querying solutions to HFL introduces expensive overhead due to the usage of heavy cryptographic primitives and the lack of customized protocols. To tackle this challenge, we propose tailored secure querying scheme utilizing lightweight secret sharing and PRF techniques, which decomposed into three steps: secure query-date data sharing, secure model prediction and secure result aggregation.
In generally, Po first constructs querying samples locally using data argumentation strategies (Sec- tion 3.4). Since querying samples imply the semantic information of private training data, they
3

---

Under review as a conference paper at ICLR 2023
Algorithm The GuardHFL framework
Input:
Each client P,. j [r), holds a private datasct D, and customized local model My. iter is the number of iterations. B is the number of query samples and C is the set of selected responding parties in the current query-response phase.
Output: Trained models M,.J E [n].
I: for each j E [ni] do
2: P; locally trains the local model Mj on Dj using the stochastic gradient descent optimization.
3: end for
for each iter do
5: for each querying party P... j E [r]] do
6:
Pg randomly samples query data {rahocte) from the auxiliary querying dataset that are generated via the data argumentation strategies described in Section 3.4.
7:
for each responding party PÁ E do
8:
Pd secret-shares [[m)]]ee (B) with Pí and the server, based on protocol Ussar in Figure 2. Pá. P2 and the server jointly perform the secure model prediction protocol in Section 3.2.
9:
10:
PA secret-shares the predictions ([vi])etjs to Po and the server.
11:
end for
12:
Pg computes (B))) where yo Eiec vy. via protocol Hage Figure with the server.
13:
Pg retrains M, based on the query dataset (as, yo (oE(R) and Dj.
14:
end for
15:
end for
cannot be directly exposed the server and PA for prediction. Therefore, GuardiFI secret-shares the query samples to the server and PA using the designed secure query-data sharing protocol (Sec- tion 3.1). Then given the secret-shared samples, PA. PQ and the server jointly execute the proposed secure model prediction protocol (Section 3.2) to obtain the secret-shared inference log- its. After that, the secure result aggregation protocol (Section 3.3) comes in handy, which takes as input the secret-shared logits and returns the aggregated results to Po. During the entire querying phase, GuardHF L maintains the following invariant (Rathee et al., 2020; Huang et al., 2022): the server and PA start each protocol with arithmetic shares of inputs, and end with arithmetic shares of outputs over the same ring. This allows us to sequentially stitch the proposed protocols to obtain fully private querying scheme. The formal security analysis is given in Appendix A.3.
3.1 SECURE QUERY-DATA SHARING
To perform secure prediction based on secret sharing techniques, Po first secret-shares the query data with the server and PA. Considering the communication constraint between PQ and PA. utilize PRFs to share x. Specifically, we first construct PRF seeds in pairs for Pq. PA and the server, denoted as Skiq A. SksA. and Sksq, which are used to generate the same random values between two parties without communication (refer to Figure 12 in Appendix 2.2.2). After that, Po can share using protocol ISbare as shown in Figure 2. In particular, PQ non-interactively shares [æ]o with PA using PRFs on the seed SKQA Then PQ computes [z]] = r and sends it to the server.
Pe Query data x +- PRE(SR
Server
P
e PRF(SkOa)
[x] =x-
[x] =r
Figure 2: Secure query-data sharing protocol II
3.2 SECURF MODEL. PREDICTION
In this step, the server and PA execute secure prediction on the secret-shared query data with the assistance of Po. Figure 11 in Appendix A.2.1 gives end-to-end graphic depiction. Below elaborate the customized protocols for three components: linear layers, ReLU and MaxPooling.
Linear layers. Linear layers consist of fully-connection, convolution, average pooling and batch normalization, and the main operation of these layers is matrix multiplication (Wagh et al., 2019; Huang et al.. 2022). We utilize PRFs and secret sharing to design customized matrix multiplication protocol I which is not only compatible with communication. limited HFL settings, but also improves communication efficiency. Specifically, shown in Figure 3, PA and the server aim to
4

---

Under review as a conference paper at ICLR 2023
compute war, where the model parameter w is held by PA and the shares [z]o. [a:]1 of a are held by PA and the server, respectively. Given that wx = w(z|u + w PA can compute w[zlo locally. To evaluate w/z]1 Po first generates three random matrices a, b and [clo using PRFs, meanwhile, computes and sends [c/ that satisfies [c] + [clo ab in Zxe² to the server. At the same time, using PRFs, the server generates the same and PA generates the same a and [clu- Then Pa and the server can learn [ylo and [y]ı (i.e., the secret shares of wr), respectively, through one round of interaction. Overall, the communication cost is 3 within I communication round.
Remark. In our fixed-pont exed-point representations, prevent values from overflowing due to the multiplica- tion operations, we use the truncation technique from Mohassel & Zhang (2017), which is consistent with existing methods (Mishra et al., 2020; Wagh et al., 2019). This technique simply truncates the extra LSBs of fixed-point values, albeit at the cost of a 1-bit error, which is negligible on the model accuracy (Mohassel & Zhang, 2017).
P9
Sever
PA Model parameter af Hrle a. a.[e]== (R)(
[s]
a.[e]=- g) 5 - PRF(SKsp)
è - PRF(S)so)
[c] ab- [cls
[x]=
[y].= (w a)b-fch
Dlee +o0))))(((++))
Figure 3: Secure matrix multiplication protocol Mamuul
ReLU. ReLU is computed as ReLU(x) = x (1 + MSB(x)), where MSB(x) equals 0 if x 2 0 and I otherwise. Thus, the evaluation of ReLU consists of MSB and multiplication operations. In the following, we propose an optimized MSB protocol building on the parallel prefix adder (PPA) logic (Harris, 2003; Mohassel & Rindal, 2018; Patra et al.. 2021). Before giving specific optimizations, we first review the PPA- based MSB method.
PPA-based MSB method. Given that the bit decomposition of [z]o and [z]] are eg...., 61 and ft....,Ji. Si. respectively, the PPA-based method evaluates MSB(x) = ee @ fe i ce, where et is the f-th earry bit. Note that C = Ct-1 A (et-1 ① fc-1) @ (et-1 A ft-1). Given the carry signal tuple (of p)) where of =, A f. and p e; + fi for i E [6]. at can be reformulated
C = g e (ph_, 2) e.... ... ① (pf A ... A p2 g9).
The PPA evaluates Eq. by constructing log l-layer tree and traversing from the leaves with inputs (gl p²) for i E [6] until reaching the root. Each node l at the n-th layer (n E [log e\) is adhered with the following operation:
gh g e (0) A and ph = PJ+1 A Pj
n-1
where it takes as inputs of two adjacent signal tuples (ofF) -fft)) and (g) n- P, ), and outputs a signal tuple (gl p²). Finally, er is obtained as g) log and MSB(x) = el + Se $ log! Overall, this method requires 38 4 AND gates within log f communication rounds, where each AND gate is evaluated by the standard Beaver triple-based multiplication protocol (Appendix A.2.2). Therefore, it totally needs 30 40 bits communication.
Optimizations. Straightforwardly adopting the above PPA-based MSB method cannot achieve the best efficiency. The performance bottleneck comes from: (1) unnecessary AND gates evaluation introduces extra communication overhead; (2) more communication rounds due to the separation of input computation and tree evaluation. To solve these challenges, firstly, we simplify the PPA circuit via removing unnecessary AND gates. In each level n E {1, -.. logg f). we eliminate the generation of "least significant" pf that consumes an evaluation of AND but it unnecessary for computing MSB. For the same reason, we also remove the evaluation of gf. Secondly, we can further reduce the communication complexity by utilizing the above multiplication protocol Ma to evaluate remaining AND gates, instead of the Beaver triple-hased method. Thirdly, we integrate the round of communication used to compute inputs of for i E {1,2. ... .L- 1} into the evaluation of the PPA circuit via non-trivial modification of the evaluation logic. Overall, our MSB protocol totally communicates 98- 3 log - 12 bits within log l rounds, a 3.44 communication improvement over the above based MSB method. Algorithm 2 gives the detailed construction of Hmmb
laa b, [c]o, [e]1) with the constrain c ab in Zyzt can be seen a variant of the Beaver's multiplication triple. Details refer Appendix A.2.2.
5

---

Under review as a conference paper at ICLR 2023
Algorithm 2 Secure MSB Protocol mmsb
Input: The arithmetic shares [xr]
Output:
The boolean shares [msb(=:)] I: Initiate g" and p" with size Let oi and pl are the i-th positions of g" andd p" respectively.
2: Po and Pi set the bit decomposition of [z)o and [z]] to ég, ei and fe. fi respectively.
Po Pi invoke IaNND with inpuls e; and fi to obtain [g.] for E [e 1].
4: Po sets [pt]0 e; and P. sets [pī][ ft.
5: Po and Pi invoke HaND with inputs [pS_1] and [ to obtain [oū]* for i E (2, (2, (
6: for E {2, ... log E do
7:
E (2. ...... . do
= 2, Po
Pi invoke HAND with inputs [9ā=2]] and [pis1] 1 to obtain [t.] and sets
10 102_11 10 [t.] R. Else Po and P. invoke HAND with inputs [0ū.-1] and [p2,] to obtain [t]]
and sets 0:1 = lo @ lt1"..
9:
end for
10:
Po and P1 invoke IAND with inputs [9i] and [po] 8 to obtain [4]] a and sets [gi]* [95] 1 [4]]
11:
for i E (2. .... 2 do
12:
r 2, Po and P1 invoke AAND with inputs [p) and 1p5 20 to obtain [p;] Else Po and Pi invoke IAND with inputs jp,] and 1p to obtain [p]]
13:
end for
14: end for
15: Po sets [msb(x)]8 @ [gil and Pi sets [msb(x)]f fe [oi]f
After obtaining [MSB(x)) B need compute [x]] (1 + [MSB(=))*), i.e., the secret shares of (z. Given 20 = (x)) and 21 = @ [MS((r)) 3 for simplicity, we have ReLU(x) = ([=]o- + (2) 21 22021) zo|2]0 + 21 [2]1 + 21 220) [z]o + z0 22 [z]1 The first two terms can be evaluated locally by PA and the server respectively. while the other two terms are evaluated using protocol Maaamll For example, to compute 21 220)[=r] the protocol Matmal inputs tu- = (1 2zp) [z]o and 21 provided by PA and the server, and outputs [ta=1]o and [f0=1] to PA and the server, respectively. Finally, PA and the server learn [y]o and [y] respectively, such that y ReLU(x)) The complete secure ReLU protocol IRRL is shown in Figure 4.
Po
Server [
PA
[x] [msb(:)]² (0)((=))-++
Mmmsh
[msb(>)]F
2, = [fs 4= (1 22,1th
(clo, (c))))))))) (c)))...
Maamal
[604s 14%
Po [tszalo
[y], [8], Bazdettt
Iylo Rolala Roy
Figure 4: Secure ReLU protocol TRRLU
Maxpooling, The evaluation of Maxpooling can be performed with the protocol URRLU as well as a tree-based round optimization that recursively partitions the values into two halves and then compares the elements of each half. Precisely, the entities arrange the input of m elements into a 2-ary tree with the depth log m, and evaluate the tree in a top-down fashion. each comparison of two secret-shared elements [ar] and [v], we utilize the observation of max(2r], [v]) ReLU((ar]
(v]) + [y]. Hence the protocol complexity of Maxpooling mainly comes from the evaluation of - 1 ReLU. Besides, as illustrated in Wagh et al. (2019); Mishra et al. (2020), AvgPooling can be evaluated locally without communication.
3.3 SECURE RESULT AGGREGATION
After the secure prediction, the predicted logit [ı:a] is secret-shared between the server and each responding party Pi, where E [C]] and C' is the set responding parties. To prevent privacy leakage from single prediction (Salem et al., 2019; Ganju et al., 2018; Yang et al., 2019), we return the aggregated logit to Po via the secure aggregation protocol A in Figure 5. Specifically, PA and Po first generate a random value r; based on PRFs. Then each PA computes and sends [æı]o-r; to the server. The server sums all received values and sends the masked aggregation to Po. which will reconstruct the aggregated logits of the query data. Notice that our secure aggregation protocol can be extended to output the aggregated label rather than the logit, using the above TRRLU protocol.

---

Under review as a conference paper at ICLR 2023
Pe
Server
P [ Tt PRF(SK
[xh, [c]
ri PRE(SK
Bolo-n
k
y softmax(k (m
Figure 5: Secure result aggregation protocol IAg
3.4 DISCUSSION
Query data construction. Unlike existing HFL works relying on auxiliary datasets the query data (Choquette-Chiee Choee-Chhhohh et al., 2021; Lin et al., 2020), we demonstrate the feasibility of model knowledge transfer in GuarcHFL L by constructing synthesized query set based on private training data, to alleviate potential limitations (e.g,, privacy, acquisition and storage) of auxiliary datasets. A simple solution is to directly use the private training data to query, like well-studied knowledge distillation (Hinton et al., 2015). Moreover, we also construct synthesized dataset via the mixup method (Zhang et al., 2018) (Appendix A.1.2). The synthesized dataset construction is a universal and modular method, and it can be readily extended with advanced data augmentation strategies, such as cutout VDeVs & Taylor, 2017) and cutmix (Yun et al., 2019). Note that this process does not reveal any private information, since the samples are constructed locally by the querying party, without involving any other parties and their private datasets. We present some exploration and experiments in Appendix 1.1.2 and Figure 10(c).
GPU-friendly evaluation. Our scheme is friendly with GPUs and can be processed by highly- optimized CUDA kernels (Tan et al., 2021). As discussed above, the cryptographic protocols of GuardHFL only involve simple vectorized arithmetic operations, rather than homomorphic encryp- tion and garbled circuits in prior works (Rathee et al., 2020; Huang al., 2022: Choquette-Chor et al., 2021). As a result, GuardHFL L is suitable for batch querying (i.e., executing multiple query- ing at the same time) with a lower amortized cost. We evaluate the designed protocols on GPUs in Section 4.1 and show the advantage of GPU acceleration over CPUs in Appendix A.1.2.
EVALUATION
Datasets and models. We evaluate GuardHFL on three image datasets (SVHN, CIFARIO and Tiny ImageNet). By default, we assume independent and identically distributed (IID) training data among clients. We also simulate disjoint Non-IID training data via the Dirichlet distribution Dir(a)) in Lin et al. (2020). The value of a controls the degree of Non-IID-ness, where smaller a indicates higher degree of Non-IID-ness. Moreover, we simulate the heterogeneity property in HFL. In particular, for SVHN and CIFARIO, we set the number of clients n 50 and use VGG-7, ResNet-8 and 10 architectures as the clients' local models. For Tiny ImageNet, we set use ResNet-14, ResNet-16, and ResNet-18 architectures. Each model architecture is used by n/3 clients. Besides, the query data constructed via two methods shown in Section 3.4: using the private training data (Q-priv) or synthesizing samples (Q-syn) via mixup (Zhang et al., 2018).
10 and
Cryptographic protocol. Following existing works (Rathee al., 2020; Tan et al., 2021), we set secret-sharing protocols over a 64-bit ring Zzes, and encode inputs using a fixed-point representation with 20-bit precision. The security parameter is 128 in the instantiation of PRFs. Unless otherwise stated, we only report the performance on the GPU accelerator. More experimental setup is given in Appendix A.I.I.
4.1 EFFICIENCY
We report the efficiency of GuardHFL, and compare it with CaPC (Choquette- et al., 2021) and instantiations of HFL based on state-of-the-ar secure querying protocols (Rathee et al., 2020; Huang et al., 2022; Tan et al., 2021).
End-to-end performance. We show the extra overhead introduced by GuardHFL compared with the vanilla HFL system in the plaintext environment. This is caused by the secure querying phase, which consists of three steps described in Section 3. Table 1 reports the runtime of each step for different models and d We observe that the cost is dominated by the secure model prediction step. Specifically, takes 16.9 minutes to evaluate 5000 query samples securely on VGG-7 and CIFAR1O, and only 11.32 second and 0.3 second are spent on the secure query-data sharing and secure result aggregation steps. More time required to evaluate Tiny ImageNet because of larger input sizes and model architectures.
'To clearly illustrate the efficiency of Gua rdHFI, unless otherwise specified, we only show the overhead of one user in each iteration descrihed in Section 3.
7

---

Under review as a conference paper at ICLR 2023
Table 1: Extra runtime (sec) of CuardHFLL over vanilla HFL systems the plaintext environment. CIFARIO and SVHN have the same runtime due to the same input size and model architecture.
of
Query
Result
Dataset
Secure prediction
Queries
data sharing
aggreg
VGG-7
RESNET-8
RESNET-10
1000
5.08
205.46
270.78
305.46
0.09
CIFARIO (SVHN)
2500
7.16
511.63
657.83
758. 16
0.12
5000
11.32
1019.12
1346.79
1521.23
0.30
RESNET-14
RESNET-16
RESNET-18 18
TINY
1000
9.87
2700.96
2971.47
3084.81
0.18
IMAGENET
2500
18.78
6815.69
7217.28
7503.5
0.32
Table 2: Comparison with CaPC on runtime (sec) over MNIST and three heterogeneous models as the batch size (BS) of query data increases.
CryptoNets
EryptoNets-Rel.. ReI.U
MLP
Model
GUARDHELL
CAPC
GUARDHFL
CAPC
GUARDHFL
CAPC
BS=128
0.03
17.75
0.24
48.83
0.75
65.01
BS=256
0.05
17.56
0.31
70.14
0.83
86.37
BS=
0.07
17.62
0.50
112.42
1.05
129.81
BS=1024
0.13
17.77
0.89
201.42
1.58
216.61
Comparison with CaPC. As described in Section I, similar to GuardHFL, CaPC (Choquette- al., 2021) was proposed to support private collaborative learning utilizing the secure querying scheme (Boemer et al., 2019b), but with the unrealistic cross-client communication. In Table 2, we compare the secure querying process of GuardHFL L with CaPC. Following CaPC's setup, we eval- et al., 2016), CryptoNets-Rel.U (Gilad-
uate three small-scale models (CryptoNets (Gilad-Bachrand Bachrach et al., 2016) and MI.P (Boemer et al., 2019b)) on MNIST. We observe that Gua rd is two orders of magnitude faster than CaPC on the three models. In terms of communication over- head, we provide a theoretical comparison. (1) For linear layers, CaPC requires to communicate homomorphic ciphertexts within 2 rounds. GuardHFL needs communicating 3 ring elements (each with 64-bit). Note that the size of ciphertexts is much larger than the size of the ring elements. (2) For non-linear layers, e.g., ReLU, CaPC adopts the garbled circuit technique that requires 2 rounds with 8CA- 4A communication bits (A 128 and & = 64 in our setting) (Rathee et al., 2020). Gua rdHFL only requires communicating 15e- log 12 bits, 70x improvement over CaPC.
Comparison with SOTA works To further demonstrate the efficiency of GuardHFI, we instan- tiate HFL based on SOTA 2PC schemes, including Cheetah (Huang et al., 2022) and CrypTFlow2 (Rathee et al., 2020), using the methods described in Appendix A.2.4. Table 3 reports the run- time and communication comparison of the secure prediction phase over CIFARIO. We observe that GuardHFL achieves a significant efficiency improvement on three heterogeneous models. For example, GuardHFL requires 57.4~75.6x less runtime and 8.6 8.6~12.77 less communication com- pared to CrypTFlow2. This is because the latter needs heavy HE-based multiplication and OT-based comparison within multi-communication rounds. Moreover, as shown in Section 2.3, extending 3PC protocols such CryptGPU (Tan et al., 2021) to HFL is non-trive trivial. However, since GryptGPU is one of the most advanced protocols under GPU analogs built on CrypTen (Knott et al., 2021), we also compare with it assuming no communication limitation. We would like to mention that despite such an unfair comparison, Gua rdHFL still has performance advantages, i.e., roughly 2.1 x and 2.0x in computation and communication overheads, respectively.
4.2 ACCURACY
We report the accuracy of each heterogeneous model in GuardHFL, and explore the impact of various factors on the model accuracy such as the Non-IID setting, and the number of query data.
End-to-end model accuracy. Table 4 reports the model accuracy on three datasets in GuardHFL. We observe that for SVHN and CIFAR10, using Q-priv to query can increase the accuracy by about 4%, while the accuracy gain is about 10% when using 10K query samples with Q-syn. The main reason is that synthetic samples could provide good coverage of the manifold of natural data. We also observe that more synthetic query data can achieve better performance from Table 4. Further- more, with an increased number of participating clients, the accuracy improves slightly. Figure shows the accuracy curves versus the number of iterations. We use SVHN and CIFAR10 as exam- ples, as they converge much faster with better readable curves than Tiny ImageNet. We can observe that each heterogeneous model on both datasets can converge well based on two types of query data, and Q-syn shows better performance.
8

---

Under review as a conference paper at ICLR 2023
Table 3: Comparison with advanced secure prediction protocols on runtime (sec) and communica-
tion (MB) cost over three heterogeneous models.
VGG-7
ResNett-8
ResNet-10
Method
TIME
CоMM.
TIME
COMM.
TIME
COMM.
CRYPIFLOW2
48.70
651.51
56.21
1110.39
97.46
1395.18
CHEETAR
3.95
116.14
4.29
94.51
6.79
169.35
CRYPEGPU
1.61
144.51
2.02
131.39
2.79
221.57
GUARDEFL
0.73
75.52
0.98
87.60
1.29
120.26
Table 4: The model accuracy of three datasets in Gua rdHFL L. on different ratios of participating clients (0.6, 0.8 and 1), and querying strategies (Q-priv and Q-syn).
Dataset
SVHN
CIFARIO
Thy ImageNet
Ratio of clients
0.6
0.8
0.6
0.8
0.6
0.8
Before GuardEFL,
75.46
56.66
22.26
Q-priv
79.43
79.56
80.29
60.82
61.01
61.49
24.89
25.11
25.23
2.5k
80.09
80.32
81.69
62.87
63.05
63.23
25.82
26.03
26.23
5.0K
83.32
83.52
83.82
63.04
63.44
63.69
26.22
26.46
26.75
Q-syn
7.5K
84.54
84.78
85.12
62.97
63.64
63.88
27.14
27.54
27.75
10K
84.58
84.97
85.62
63.79
63.82
64.56
27.67
28.19
28.46
90
70
Impact of Non-IID datasets. We il-
lustrate the impact of Non-IID data on model accuracy in Figure 7, us- ing CIFARIO as an example. Figures 7(a). 7(b) and 7(c) visualize the dis- tributions of Non-IID samples among clients with different Dir(a.) When
Accuracy
Accurary
80
8
75
88
6
10
B 10
= 100, the distribution is close to uniform sampling. When a 0.5, the sample distribution of cach class among clients is extremely uneven. From Figure 7(d) we observe that the higher the degree of Non- IID-ness, the lower the accuracy of models. Notably, GuardHFL can still significantly improve the performance of models under the Non-IID environment.
Iterations
Iterations
Figure 6: Accuracy curves of each heterogeneous model in GuardHFL. Left (SVHN); Right (CIFAR10)
Defore Guadito
200
...
3.5 Degree Non aasss
100
Party
Pairy
Party I0c
(a) a = 100
(b) a=1
(c) a 0.5
(d) Accuracy
Figure 7: Visualization of Non-IID-ness, among clients with different Dirichlet distribution a on CIFARIO. The size of scattered points indicates the number of training samples of that class. Impact of other factors. Due to space constraints, report other experimental results in Ap- pendix A.1.2. In particular, Figure 9 shows the accuracy of each heterogeneous model with different numbers of query data. Figures 10(a) and 10(b) illustrate the impact of different numbers of pri- vate training data on SVHN and CIFAR1O. Figure 10(c) details the impact of different query data construction methods.
5 CONCLUSION
We propose GuardHFL, an efficient and private HFL framework to formally provide the privacy guarantees of query samples, model parameters and prediction results. The core constructions of GuardHFL are customized multiplication and comparison protocols based on lightweight secret sharing and PRFs techniques. Extensive experiments demonstrate that GuardHFL outperforms prior art in both communication and runtime performance. We consider the following future direc- tions. (1) The communication cost of Gua rdFFL, which is also the limitation of the standard HFL. paradigm, will be further improved. One possible mitigation is to extend the insight of the k-regular graph in FL (Bell et al., 2020) to HFL, and carefully design protocols from scratch. The main idea is that in FL it is enough for each party to speak to k < n 1 other parties via the server, where is the number of parties. (2) The security of GuardHFL will be improved to defeat more powerful malicious adversaries. Unfortunately, even using the best-known technique (Koti et al., 2021), the overhead will be increased by several orders of magnitude. We leave these improvements as future work.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Michel Abdalla, Mihir Bellare, and Phillip Rogaway. The oracle diffie-hellman assumptions and an analysis of dhies. In Proceedings of CT-RSA, pp. 143-158. Springer, 2001.
Gilad Asharov, Yehuda Lindell, Thomas Schneider, and Michael Zohner. More efficient oblivious transfer and extensions for faster secure computation. In Proceedings of ACM CCS, 2013.
Donald Beaver. Efficient multiparty protocols using circuit randomization. In Proceedings of CRYPTO, 1991.
James Henry Bell, Kallista A Bonawitz, Adrià Gascón, Tancrède Lepoint, and Mariana Raykova. Secure single-server aggregation with (poly) logarithmic overhead. In Proceedings of ACM CCS, 2020.
Fabian Boemer, Anamaria Costache, Rosario Cammarota, and Casimir Wierzynski. ngraph-he2 A high-throughput framework for neural network inference encrypted data. In Proceedings of the ACM Workshop on Encrypted Computing & Applied Homomorphic Cryptography, 2019a.
Fabian Boemer, Yixing Lao, Rosario Cammarota, and Casimir Wierzynski. ngraph-he: a graph compiler for deep learning on homomorphically encrypted data. In Proceedings of the ACM International Conference on Computing Frontiers, 2019b.
Keith Bonawitz, Vladimir Ivanov, Ben Kreuter, Antonio Marcedone, H Brendan McMahan, Sarvar Patel, Daniel Ramage, Aaron Segal, and Karn Seth. Practical secure aggregation for privacy- preserving machine learning. In proceedings of ACM CCS, 2017.
Alon Brutzkus, Ran In Proceedings of ICML, 2019.
and Oren Elisha. Low latency privacy preserving inference.
Ran Canetti. Universally composable security: A new paradigm for cryptographic protocols. In Proceedings of FOCS, 2001.
Hao Chen, Wei Dai, Miran Kim, and Yongsoo Song. Efficient multi-key homomorphic encryption with packed ciphertexts with application to oblivious neural network inference. In Proceedings of ACM CCS, 2019.
Christopher A Choquette-Chor Natalie Dullerud, Adam Dziedzic, Yunxiang Zhang, Somesh Jha, Nicolas Papernot, and Xiao Wang. Cape learning: Confidential and private collaborative learning. In Proceedings of ICLR, 2021.
Daniel Demmler, Thomas Schneider, and Michael Zohner. Aby-a framework for efficient mixed- protocol secure two-party computation. In Proceedings of NDSS, 2015.
Terrance DeVries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint ww..iv:770...o 045522 2017.
Enmao Diao, Jie Ding, and Vahid Tarokh. Heterofl: Computation and communication efficient
federated learning for heterogeneous clients. In Proceedings of ICLR, 2021.
Whitfield Diffie and Martin Hellman. New directions in cryptography. IEEE Transactions on Infor- mation Theory, 22(6):644-633 1976.
Canh T Dinh, Nguyen H Tran, and Tuan Dung Nguyen. Personalized federated learning with moreau envelopes. In Proceedings of NeurIPS, 2020.
Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Personalized federated learning with theo- retical guarantees: A model-agnostic meta-learning approach. In Proceedings of NeurIPS, 2020.
Karan Ganju, Qi Wang, Wei Yang, Carl A Gunter, and Nikita Borisov. Property inference attacks fully connected neural networks using permutation invariant representations. In Proceedings of ACM CCS, 2018
Craig Gentry. Fully homomorphic encryption using ideal lattices. Proceedings of ACM STOC, 2009.
10

---

Under review as a conference paper at ICLR 2023
Ran BaaaddBhcch Nathan Dowlin, Kim Laine, Kristin Lauter, Michael Nachrig, and John Wern- ing. Cryptonets: Applying neural networks to encrypted data with high throughput and accuracy. In Proceedings of ICML, 2016.
Oded Goldreich. Foundations of cryptography. Cambridge university press, 2009.
David Harris. A taxonomy of parallel prefix networks. In Proceedings of Asilomar Conference on Signals, Systems & Computers, pp. 2213-2217, 2003.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint 10Xvv:1503.02531, 2015.
Zhicong Huang, Wen-jie Lu, Cheng Hong, and Jiansheng Ding. Cheetah: Lean and fast secure two-party deep neural network inference. In Proceedings of USENIX Security, 2022.
Bargav Jayaraman and David Evans. Evaluating differentially private machine learning in practice. In Proceedings of USENIX Security, 2019.
Bargav Jayaraman and Lingxiao Wang. Distributed learning without distress: Privacy-preserving empirical risk minimization. 2018.
Chiraag Juvekar, Vinod Vaikuntanathan, and Anantha Chandrakasan. Gazelle: A low latency frame- work for secure neural network inference. In Proceedings of USENIX Security, 2018.
Peter Kairouz, Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Ad- vances and open problems federated learning. arXiv preprint arXiv:1912.04999 2019.
Marcel Keller and Ke Sun. Secure quantized training for deep learning. In Proceedings of ICML, 2022.
Brian Knott, Shobha Venkataraman, Awni Hannun, Shubho Sengupta, Mark Ibrahim, and Laurens van der Maaten, Crypten: Secure multi-party computation meets machine learning. Proceed- ings of NIPS, 2021.
Nishat Koti, Mahak Pancholi, Arpita Patra, and Ajith Suresh. Swift: Super-fast and robust privacy- preserving machine learning. In Proceedings of USENIX Security, 2021.
Junghyun Lee, Eunsang Lee, Joon-Won Woo Lee, Yongjune Kim, Young-Sit Kim, and Jong-Seon No. Precise approximation of convolutional neuralnetworks for homomorphically encrypted data.
MXXiv:2005.10873 2021.
Daliang Li and Junpu Wang. Fedmd: Heterogenous federated learning via model distillation. In Proceedings of NeurIPS Workshop on Federated Learning for Data Privacy and Confidentiality, 2019.
Tao Lin, Lingjing Kong, Sebastian U Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning. Proceedings of NeurlPS, 2020.
Pratyush Mishra, Ryan Lehmkuhl, Akshayaram Srinivasan, Wenting Zheng, and Raluca Ada Popa. Delphi: A cryptographic inference service for neural networks. In Proceedings of USENIX Secu- rity, 2020.
Payman Mohassel and Peter Rindal. Aby3: A mixed protocol framework for machine learning. In Proceedings of ACM CCS, 2018.
Payman Mohassel and Yupeng Zhang, Secureml: A system for scalable privacy-preserving machine learning. Proceedings of IEEE S&P, 2017.
Nicolas Papernot, Martín Abadi, Ulfar Erlingsson, Ian Goodfellow, and Kunal Talwar. Semi- supervised knowledge transfer for deep learning from private training data. In Proceedings of ICLR, 2016.
Arpita Patra, Thomas Schneider, Ajith Suresh, and Hossein Yalame. Aby2.0: Improved mixed- protocol secure two-party computation. In Proceedings of USENIX Security, 2021.
11

---

Under review as a conference paper at ICLR 2023
Le Trieu Phong, Yoshinori Aono, Takuya Hayashi, Lihua Wang, and Shiho Moriai. Privacy- preserving deep learning via additively homomorphic encryption. JEEE Transactions on Infor- mation Forensics and Security, 1365)1133- 2018.
Deevashwer Rathee, Mayank Rathee, Nishant Kumar, Nishanth Chandran, Divya Gupta, Aseem Rastogi, and Rahul Sharma. Cryptflow2: Practical 2-party secure inference. In Proceedings of ACM CCS, 2020.
M Sadegh Riazi, Christian Weinert, Oleksandr Tkachenko, Ebrahim M Songhori. Thomas Schnei- der, and Farinaz Koushanfar. Chameleon: A hybrid secure computation framework for machine learning applications. In Proceedings of AsiaCCS, 2018.
Ahmed Salem, Yang Zhang, Mathias Humbert, Pascal Berrang, Mario Fritz, and Michael Backes. MI-leaks: Model and data independent membership inference attacks and defenses machine learning models. In Proceedings of NDSS, 2019.
Sinem Sav, Apostolos Pyrgelis, Juan R Troncoso-Pastoriza. David Froelicher, Jean-Philippe Bossuat, Joao Sa Sousa, and Jean-Pierre Hubaux. Poseidon: Privacy-preserving federated neural network learning. In Proceedings of NDSS, 2021.
Adi Shamir. How to share secret. Communications of the ACM, 22(11):612-613 1979.
Lichao Sun and Lingjuan Lyu. Federated model distillation with noise-free differential privacy. In Proceedings of ICCAI, 2021.
Sijun Tan, Brian Knott, Yuan Tian, and David J Wu. Cryptgpu: Fast privacy-preserving machine learning on the gpu. In Proceedings of IEEE S&P, 2021.
Buse GA Tekgul, Yuxi Xia, Samuel Marchal, and N Asokan. Waffle: Watermarking in federated learning. In Proceedings on IEEE SRDS, 2021.
Sameer Wagh, Divya Gupta, and Nishanth Chandran. Securenn: 3-party secure computation for neural network training. Proceedings on Privacy Enhancing Technologies, 2019(3):26 2019.
Sameer Wagh, Shruti Tople, Fabrice Benhamouda, Eyal Kushilevitz, Prateek Mittal, and Tal Rabin. Falcon: Honest-majority maliciously secure framework for private deep learning. Proceedings on Privacy Enhancing Technologies, 1:188-205 2021.
Ziqi Yang, Jiyi Zhang, Ee-Chien Chang, and Zhenkai Liang. Neural network inversion in adversarial setting via background knowledge alignment. In Pruceedings of ACM CCS, 2019.
Andrew Chi-Chih Yao. How to generate and exchange secrets. In Proceedings of IEEE FOCS, 1986.
Tehrim Yoon, Sumin Shin, Sung Ju Hwang, and Eunho Yang. Fedmix: Approximation of mixup under mean augmented federated learning. In Proceedings of ICLR, 2021.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. Proceed- ings of ICCV, 2019.
Mikhail Yurochkin, Mayank Agarwal, Soumya Ghosh, Kristjan Greenewald, Nghia Hoang, and Yasaman Khazaeni. Bayesian nonparametric federated learning of neural networks. In Proceed- ings of ICML, 2019.
Hongyi Zhang. Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In Proceedings of ICLR, 2018.
Zhuangdi Zhu, Junyuan Hong, and Jiayu Zhou. Data-free knowledge distillation for heterogeneous federated learning. In Proceedings of ICML, 2021.
12

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 MORE DETAILS ON EXPERIMENT EVALUATION
A.1.1 EXPERIMENTAL SETUP
Datasets. We evaluate GuardHFL on the following standard datasets for image classification: (1) SVHN is a real-world image dataset obtained from house numbers in Google Street View images, which contains 600,000 32x32 RGB images of printed digits from 0 to 9. (2) CIFARIO consists of 60,000 x2 32 RGB images in 10 classes. There are 50,000 training images and 10,000 test images. (3) Tiny ImageNet contains 100,000 images of 200 classes downsized to 64 x64 colored images. Each class has 500 training images, 50 validation images and 50 test images.
Experimental configuration. Each entities, i.e., Pg. PA and the server, is run on the Ubuntu 18.4 system with Intel(R) 562 Xeon(R) CPU E5-2620 v4(2. 10 GHz) and 16 GB of RAM and NVIDIA 1080Ti GPU. We leverage Py PyTorch's ' distibbit package for communication similar as Knott et al. (2021); Tan al. (2021). We ran our benchmarks in the LAN setting, where following Huang et al. (2022) the bandwidth is about 384 MBps and the latency is 0.3ms.
Training procedure. At the local training phase, each client trains the local model from scratch using stochastic gradient descent optimization. For SVHN, CIFAR10, and Tiny ImageNet, the loss function is cross-entropy with the learning rate of 0.5, 0.1, 0.01, respectively. Besides, the batch size is 256, 64 and 64, respectively. When the clients retrain the local model at the local retraining step, they use Adam optimizer for 50 epochs with learning rate of 2c-3 decayed by factor of 0.1 on 25 epochs, where the batch size is 256 on SVHN, and 64 on both CIFARIO and Tiny ImageNet.
A.1.2 EXPERIMENTAL RESULTS
Impact of GPU acceleration. To further explore the impact of GPU acceleration, we evaluate GuardHFL on both CPU and GPU with different batch sizes of query data. Figure 8 reports the sults of VGG-style and ResNet-style networks on CIFARIO, where the GPU-based setting is always superior the CPU analogs. As the batch size increases, the advantage of GPU-based protocols becomes more pronounced, e.g., 14x runtime reduction on ResNet-8 over the batch size 64.
250
VO
VODT CPU VodD CPU MESTT-Cr
Resost&
Searett CPU
Runtime (sec)
200
CPU
Runtime (sec)
300
Resocttb-C2 CPU Beenst12-GUU
Seaset10-CPP d-cgu-
150
200
100
100
50
10
20
30
60
0
10
20
30
50
60
Batch Size
Batch Size
Figure 8: The runtime of GuardHFLL under CPU/GPU with varied batch sizes of query data CIFARIO.
Impact of the number of query data. Figure 9 shows the accuracy of each heterogeneous model with different numbers of query data. We observe that GuardiFL consistently improves the model accuracy above datasets and heterogeneous models. Specifically, the number of query data increases (from 2.5K to 10K), the accuracy of all three models increases by about 5%.
Impact of the number of private training data. Figures 10(a) and 10(b) illustrate the model accuracy of GuardHFL under different number of private training data on SVHN and CIFARIO 10. We can observe that as the number of training data increases, the model performance is the rise. The main reason is that models can learn better on more training data and can construct more synthetic samples to query, as promote the transfer of model knowledge.
Impact of query data construction strategies. Figure 10(c) gives the model accuracy under three advanced data augmentation strategies, including cutmix (Yun et al., 2019), cutout (DeVries &
13

---

combination two images 2; and z; Under review as a conference paper at ICLR 2023
VG87
RecNet34 ReoNetl 5
Accuracy (%6
90
Ros
65
ResNet1o
ReaNet8
Acmuracy (%)
ResNett
80
Acouracy
60
70
20
60
5.
15
50
50
10
7500
5000
7500
10000
5000
7500
10000
Number Query Data
Number of Query Data
Number of Query Data
(a) SVHN
(b) CIFARIO
(c) Tiny ImageNet
Figure 9: The accuracy of each heterogeneous model as the number of query data increases. Dashed lines represent the model accuracy before GuardHFL.
100
70
Betore GuarditFL
Before GuarditLL
GuardittL (0 (o-pavi
60
Guardi1PL
BeGore GuardHFL.
Ancuracy
70
GuardEFFL (Qnsvm)
Accuracy
50
Guard PL (CHmm))
Accuracy (%6
80
70
Cutnut
Cutmax
Mixup
40
.0
30
30
20
Number of Private Data Sample
Number Private Data Sample
CIFARIO
SVHN
(a) SVHN
(b) CIFARIO
(c) Various query data construction
methods
Figure 10: The model accuracy under different number of training data and query data construction methods on SVHN and CIFAR10.
Taylor, 2017), and mixup (Zhang et al., 2018). Cutmix (Yun al., 2019) can be formulated as テi, = x; -1-30 (1-M)- -2,, where M E {0, 1]
1 x is binary mask matrix size W x to indicate the location of dropping out and filling from the two images x, and zy. Cutout (Devries & Taylor, 2017) augments the dataset with partially occluded versions of original samples. Mixup (Zhang al., 2018) constructs synthetic samples by with different coefficients x, in which ž. = X I2 + (1 X) Tz- We observe that those strategies good choices for query data construction in GuardHFL.
A.2 MORE DETAILS ON THE DESIGNED SCHEME
A.2.1 GRAPHIC DEPICTION OF END-TO-END SECURE PREDICTION SCHEME
Figure 11 gives a graphic depiction to illustrate the end-to-end secure prediction scheme, where the input is secret-shared sample [x). The whole process maintains the following invariant: the server and PA begin with secret shares of the input and after each layer, end with secret shares (over the same ring) of the output. Honest- fuutrso-bu-currous security of GuardHF) will follow trivially from sequential composibility of individual layers. To be specific, [a]] first passes through convolutional layer that can be formalized as the secure matrix multiplication operation w1 [x]] (wi is the parameter) using protocol Ma Figure The secret-shared outputs this layer, i.e., [y1]o la and [yı]ı. are obtained by PA and the server, respectively. For the subsequent ReLU layer, protocol TREI in Figure exccuted to return [yzlo and [y2]1 to PA and the server respectively. Then Maxpooling on [y2] can be evaluated via protocol IRRALU described in Section 3.2. outputs the secret-shared values [ysla and [ys]1. When the secure prediction reaches the final fully-connected layer with inputs [Vn-1]o and [y,-1]1, protocol Ma is executed. In the end, Pa and the server obtain the secret-shared predicted logit, i.e., [logit]o and [logit] 1, respectively.
A.2.2 MORE DETAILS CRYPTOGRAPHIC PROTOCOLS
Secret sharing and Beaver's multiplication protocol. As shown in Section 2.4, GuardHFL uti- lizes the additive secret sharing primitive protect the privacy of sensitive information. Assuming two secret-shared values are [æ] and [y] owned by two parties P,, E (0, 1}, addition and subtraction operations ([=) [xc] + [y] in Zezr) can be realized locally without any communication, i.e., each P;
14

---

Under review as a conference paper at ICLR 2023
[x]
Di
bsl
[ogh
Server
ReLU ineteot Hos
MaxPooling iPrstac Reas!
Secure Prediction
Procedt Torawa
Proteodd Reaass
Retli(psl)
respask)sst
[x]e,
Ds.le
Lxala
Da-iJoo
[[ogieJo
Figure 11: End-to-end secure prediction protocol. Green boxes represent linear layers (including convolutional/fully commected/Avgpooling layers), and blue boxes represent non-linear layers (in- cluding ReLU//axpooling layers).
computes [z]. [a]] [y] mod 2 Besides, multiplication operation, i.e., [z] [x][[]] evalu- ated using Beaver's multiplication triples (Demmler et al., 2015), where each triple refers to (a, 6, c) with the constraint e ab that is generated by cryptographic techniques (Demmler al., 2015) or trusted dealer (Riazi et al., 2018). Specifically, the multiplication over becretshhrrng values can be evaluated in the following:
(3)
== xy = ([z)o + [b]o + Mi) [olo[vlo
[x]] [=]ol + [x]][v]o
where for E {0, 1}. [x]+ can be computed locally, and [x]+ will be evaluated as follows. Taking [rlo[y]]
example, assuming Po and P1 already hold (a, [clo) and (b, [c]1). respectively, Po first sends [z]g + a to Pt. while P1 sends [y] to Po- Then Po computes one share as [>]o[[v]o - b) - [clo, and Pi computes another as ([=]1 + a)ly [c]1, locally. In this way, the outputs are still in the form of secret sharing.
Diffie-Hellmar key agreement protocol. In GuardHFL, utilize PRFs to overcome the cross- client communication limitation, where the consistent PRF seed between clients are generated using the Diffie-Hellman Key Agreement (DH) protocol (Diffie & Hellman, 1976). Note that the consis- tent seed between the server and the client can be directly sampled by the server and then sent to the client without the DH protocol. Figure 12 gives the secure seed generation protocol Hssed Formally, the DH protocol consists of the following three steps:
DH. DH.param (k)) + (G. g.q, q. H) generates a group G of prime order q, along with a generator g. and hash function H.
DH.gen(G, g. q, H) + (1), randomly samples x; E Z, as the secret key and let gt as the public key,
(HH.agree(z,,g"" g", H) - st,i outputs the seed 8t.1 = (zg)))))).
Correctness requires that for any key pairs (xi,g*') and (*j,g*s) generated by two paries P. and P, using DH.gen under the same parameters (G. g, q, H), DH.agree(z; g² H) DH.agree(2,, g": H)) Besides, in GuardHFL security requires that for any adversary who steals gei and n (but neither of the corresponding x; and ι.)), the agreed secret 8t.j derived from those keys is indistinguishable from uniformly random value (Abdalla al., 2001).
Po Sample kg E TTp A gd2 mod
Sample Sk Slea E
Sample Rt e Ep 5A mad
B,Skso
R
A.Sk
Sksg Skoa = DW. agree(ky. (kg B, Hymdd
Skya Skoa DH. ag..gree(KK, A. Hmmod
Figure 12: Secure PRF seed generation protocol IT
15

---

Under review as a conference paper at ICLR 2023
A.2.3 DISTINGUISH GUARDHEL FROM OTHER PRIVATE SETTINGS.
Gua GuardHFL is in line with the standard HFL paradigm (Li & Wang. 2019) with the additional ben- efit of privacy protection. As declared in Introduction, GuardHFL, is the first-of-its-kind privacy- training efforts. The preserving HFL framework, which is different from existing federated learning (Bonawitz al., latter can be divided into two categories: (1) ppreeeeiigg 2017; Bell et al., 2020), and (2) secure multi-party training (Tan et al., 2021; Keller & Sun, 2022). In the following, give detailed analysis.
Comparison to privacy-preservve federated learning. In the ing (FL), each user locally computes the gradient with his private database, and then secure ag- gregation protocol is executed at the server side for aggregating the local gradients and updating the global model. However, as shown in Introduction, secure gradient aggregation cannot be realized in heterogencous FL (HFL), due to the heterogeneity of the clients' models. Instead, rHHL follow general HFL training paradigm (Li & Wang. 2019), which contains three steps: local train- ing, querying, and local re- re-training. GuardHFL focuses on solving the privacy issue caused by the querying stage, and mainly proposes query datasets generation (refer to Section 3.4) and secure querying protocol (refer to Section 3.1 Section 3.3).
federated learn-
Comparison to secure multi-party training. Secure multi-party training typically an out- sourced training setting, where resource-constrained clients outsource the entire training task to non-colluding multiple servers in to finally yield well-trained model. Different from secure multi-party training, GuardHFL enables clients to collaboratively and securely train their own customized models that may differ in size and structure. Moreover, as discussed above, the general HFL paradigm contains three steps: local training, querying and local re-training, where the local training and re-training stages are evaluated locally without revealing privacy. Therefore, the secure querying protocol (i.e., a customized inference protocol).
manner. It requires a secure training protocol
HFL requires HFL-friendly
A.2.4 EXTEND EXISTING 2PC PROTOCOLS TO HFL
With non- non-trivial adaptation, existing secure 2-party querying schemes (Mishra et al., 2020; Rathee et al., 2020; Huang et al., 2022) can be extended to the communication-liitte HFL setting. How- ever, as shown in Section 4. 1, such extension introduces expensive communication and computation overheads compared with our GuardHFL. In the following we divide these schemes into three cat- egories, i.c., pure OT-based protocols, pure HE-based protocols, and hybrid protocols, and give the corresponding extension designs.
To extend the pure OT-based secure querying protocols such as CrypTFlow2 (Rathee et al., 2020) into HFL, Po first secret-shares query samples to the server and PA using our protocol Issaaee in Section 3. Then the server and Pa execute secure prediction based on the method proposed in Rathee al. (2020). After that, adopting our secure aggregation protocol Tagg in Section 3.3, the aggregated predictions will return to Po. Although the OT-based schemes can be extended to HFL by combining the designed protocols in GuardHFL, it requires too many communication rounds due to the usage of OT primitives.
To extend the pure Hb-ea secure querying protocols 2021) to HFL, Po first encrypts the query samples and asks the server to pass them to PA. After that, PA evaluates secure prediction non-interactively in the ciphertext environment. Then PA sends encrypted predictions to the server. The server aggregates these encrypted predictions utilizing the additive homomorphism of HE and sends the aggregated results to Pq- Although it is trivial to extend the schemes equipped with the HE-based scheme to the communication- limited setting, they have two key problems: 1) activation functions need to be approximated as low-degree polynomials, which leads to serious accuracy loss; 2) due the inherent high computation complexity, HE-based secure prediction is difficult to extend to large-scale models.
et al., 2016; Lee et al.,
For hybrid secure querying protocols that evaluates linear layers using HE and non-linear layers using OT or GC, such as Cheetah (Huang et al., 2022). we discuss the extended algorithms for each layer separately. For the linear layer, 1) Po encrypts query samples with HE and sends the
16

---

Under review as a conference paper at ICLR 2023
ciphertext PA through the server'. 2) PA evaluates linear layers locally, and returns the encrypted masked outputs to Pq through the server. 3) Po decrypts to obtain the masked outputs, which are then sent to the server. As a result, the masked outputs of linear layers server and PA- For the non-linear layer, given that the server and Pa hold shares of the linear layer's outputs, the parties invoke the OT protocols to evaluate the non-linear functions. Therefore, such extension comes at the cost of heavy computational and communication complexity.
secret-shared between the
In summary, although existing 2PC protocols can be extended to the HFL setting with the cross- communication restriction, they sacrifice the efficiency due to the lack of customized protocols and the adoption of heavy cryptographic primitives. Therefore, GuardHFL shows better adaptability and efficiency in the natural HFL scenarios.
A.3 SECURITY ANALYSIS
Table 5: The ideal functionality
Input sharing functionality F
Input: Po: query data
Output: PA: [z]o Lzz. Server: [z]] = x - [rroo mod 2
Matrix multiplication functionality FMatmul:
Input: Server: [xr] E Zzz. PA: [xoo E Zoze, model parameter w.
Output: Server: [w]1 Ezz.. PAi (ylo = wa [y] mod 2
MSB functionality Fmsb:
Input: Server: [z]] E Zzt. PA: 1x00 E Zze.
Output: Server: [msb(x)]f Ez. PAi [msb(x)]{ = msb(æ) @ [msb(x)]] mod!
ReLU functionalityy FRELU:
Input: Server: [x]] E EZz. PAi [x]o E Zog.
Output: Server: [ı]1 Eze. PAi [y]o = ReLU(x) - [v] mod 24.
Maxpooling functionality Faaxpool:
Input: Server: [x]] E Zozr. PAi [xro E Zzr.
Output: Server: [8] E Ezze PAi [y]o = Maxpool(x) - [v] mod 2
Result aggregation functionality FAag:
Input: Server: [z:)] E igg [C]. PA: [xsJo E Zz.
Output: Pq: softmax 24).
Intuitively, GuardHFL only reveals the aggregated prediction to PQ without the responding parties' model parameters, and the server and P^ learn zero information about the querying parties' data. This is because all intermediate sensitive values are secret-shared. Next, we give formal analysis.
Our security proof follows the standard deell-world/rellwwoor paradigm (Canetri, 2001): in the real world, three parties (i.e., the server, Pq. and PA) interact according to the protocol specification, and in the ideal world, they have access to an ideal functionality shown Table 5. When a protocol invokes another sub-protocol we the F-hybrid model for the security proof by replacing the sub- protocol with the corresponding functionality. Note that our proof works in the FPRE-hybrr model where FPRF represents the ideal functionality corresponding to the PRF protocol. The executions in both worlds are coordinated by the environment Env, who chooses the inputs to parties and plays
'To be more precise, this step is for the input layer. In the hidden layer, one of the input shares of the linear layer should be encrypted by the server and sent to PA-
17

---

Under review as a conference paper at ICLR 2023
the role of distinguisher between the real and ideal executions. We will show that the worldworld distribution is computationally indistinguishable to the world distribution.
Theorem A.1. Share securely realizes the functionality F in the FeRE-hybrid model.
Proof. Note that Po and PA receive no messages IIarre and hence the protocol trivially secure against corruption of Po and PA. Next, the only message that the server receives is the value [x] 1- However, [z|] = X r, where given the security of PRF. r is a random value unknown to the server. Thus, the distribution of [x]] is uniformly random from the server's view and the information learned by the server can be simulated.
Theorem A.2. Matmul securely realizes the functionality FMttmul in the FPRE-hybrid model.
Proof. Note that PQ receives no messages in Il and hence the protocol is trivially secure against corruption of Pq. The only message that PA receives is the value [x]] b. However, given the security of PRF, b is a random value unknown to PA. Thus, the distribution of [x]1 - b computationally indistinguishable from a uniformly random distribution in PA 's view, and the information learned by PA can be simulated. Next, during the protocol, the server learns [c] and w-+ a. However, the distribution of [c] and w + is computationally indistinguishable from a uniformly random distribution in the server's ' view, since given the security of PRF, and [c]; random values unknown to the server. Thus, the information learned by the server can be simulated.
Theorem A.3, IRRLLU securely realizes the functionality FReLU in the MFMatmul- Fmsb)-hybrid model.
Note that as shown in Section 3.2, IRRL consists of msb and Ma Therefore, the ReLU protocol is trivially secure in the (FMatmuls F msb) -hybrid model.
Proof.
Theorem A.4. Maxpool securely realizes the functionality FMaxpool in the FRoLU-hybrid model.
Proof. As shown in Section 3.2, Masppoll consists of several invocations of l.ReLU. Therefore, the protocol Maxpool is trivially secure in the FReLu-hybrid model.
Theorem A.S. AAgg securely realizes the functionality Frgg in the FeRE-hybrid model.
Proof. Note that PA receives no messages in aAgg- and hence the aggregation protocol is trivially secure against the corruption of PA- Next, the only message that the server receives the value [æ:]o r.. However, given the security of PRF, T3 random value unknown to the server. Thus, the distribution [2.]0 Ta is computationally indistinguishable from a uniformly random distri- bution in the server's view and the information Iearned by the server can be simulated. After the aggregation, Po only learns the aggregated result Σisicy Vi, but unknown to each x;. Therefore, the aggregation protocol is secure assuming the aggregation result will not reveal privacy.
A.4 RELATED WORK
A.4.1 HETEROGENEOUS FEDERATED LEARNING
Federated learning (FL) achieves collaboration among clients via sharing model gradients. While successful, it still faces many challenges, among which, of particular importance the heterogeneity that appear in all aspects of the learning process. This consists of model heterogeneity (Li & Wang, 2019) and statistical heterogeneity (Zhu al.. 2021). Statistical heterogeneity means that parties' data comes from distinct distributions (i.e., Non-IID data), which may induce deflected local opti- mum. Solving the statistical heterogeneity has been extensively studied, such as Dinh et al. (2020); Zhu et al. (2021); Yurochkin et al. (2019); Fallah et al. (2020); Yoon et al. (2021), and is out of the scope of this work. However, GuardHFL may help alleviate the statistical heterogeneity due to the customized model design and the knowledge distillation-bast aggregation rule.
Our work mainly focuses on the model heterogeneity that has been explored in recent works (Li & Wang, 2019; Lin et al., 2020; Choquette-Choo et al., 2021). In particular, Li & Wang (2019) proposed the first FL framework FedMD supporting heterogeneous models by combining transfer learning and knowledge distillation techniques. They first used public dataset to pre-train the model
18

---

Under review as a conference paper at ICLR 2023
Table 6: Comparison with prior works on properties necessary for federated learning
Privacy
Usability
Efficiency
Framework
Data Privacy
Model Privacy
Model Heterogeneity
w/o Dataset Dependency
GPU
Protocol Efficiency
Compatibility
Bonawitz et al. (2017)
x
Bell al. (2020)
x
Sav et al. (2021)
/
/
x
Jayaraman Wang (2018)
X
X
Li & Wang (2019)
X
Choquetre- Meauuetee-Chonee al. (2021)
V
/
XX
X
X
x
Lin et al. (2020)
X
X
Sun & Lyu (2021)
Diao et al. (2021)
X
X
/
GuardHFL
and transferred to the task of private datasets. After that, to exchange the knowledge, each party used the public data and the aggregated predictions from others as carrier for knowledge distillation. To further improve model accuracy, Lin al. (2020) proposed FedDF, similar FedMD, which also used the model distillation technique for knowledge sharing. The difference is that they first per- formed FedAvg on parties' local models and integrated knowledge distillation on the aggregated model. The dependence on model averaging leads to limited model heterogeneity. Besides, Diao al. (2021) focused on heterogeneous parties equipped with different computation and communi- cation capabilities. In their framework, each party only updated a subset of global model parameters through varying the width of hidden channels, which reduces the computation and communication complexity of local models. However, this approach only learns single global model, rather than unique models designed by parties. Moreover, as described in Introduction, HFL suffers from sev- eral privacy issues, which not considered in the above works. The first is the direct leakage of querying samples. The task-related querying samples may contain private information such as disease diagnosis in healthcare. If the querying party directly delivers query samples to responding parties for prediction via the server, such sensitive information will be leaked. The second is the implicit disclosure based on inference attacks. Given black-box access infer the membership (Salem et al., 2019) and attribute information (Ganju et al., 2018) of the target sample or even reconstruct the original training data (Yang et al., 2019). For example, based on the prediction y of the querying sample x from the responding party PA. the querying party can launch the membership inference attack (Salem al., 2019) to infer whether x is training sample of PA- To address the privacy concern, GuardHFL provide end-to-end privacy-preservic HFL services.
model, adversaries can
The privacy-preserving techniques (i.e., secure aggregation) have been studied in federated learning (Bonawitz. et al., 2017; Bell et al., 2020; Sav et al., 2021; Jayaraman & Wang. 2018). However, these techniques can not be directly extended to srivacy-reseeving HFL. More recently, Sun & Lyu (2021) proposed a noise-free differential privacy solution for HFL guarantee each party's 's privacy. However, as shown in Jayaraman Evans (2019), there is a huge gap between the upper bounds privacy loss analyzed by advanced mechanisms and the effective privacy loss. Thus, differentially private mechanisms offer undesirable utility-privacy trade-offs. To further formally guarantee the privacy, Choquette-Choo et al. (2021) leveraged hybrid cryptographic primitives to realize confidential and private collaborative learning. Specifically, parties learn from each other collaboratively utilizing a secure inference strategy based on 2PC and HE protocols and a private aggregation method. As noted in the Introduction, CaPC's use of heavy cryptography leads to significant efficiency and communication overheads.
In summary, we give a comparison between prior FL works and GuardHFL in Table 6.
A.4.2 SECURE NEURAL NETWORK PREDICTION
Since secure prediction is critical component of GuardHFL, we briefly introduce its recent progress. Neural networks present challenge cryptographic protocols due their unique struc-
19

---

Under review as a conference paper at ICLR 2023
ture and exploitative combination of linear computations and non-linear activation functions. In real scenarios, model prediction can be viewed as a two-party computation case, where one party with private query data wants to obtain prediction results from the other party who owns the model. Dur- ing the whole process, the cryptographic protocols, typically HE and secure multi-party computation (MPC), are applied ensure the confidentiality of the private data and model parameters.
Many existing works (Boemer et al., 2019b; support pure HE protocols for secure predictions. Typically, nGraph-HE (Boemer et al., 2019b:a) allows linear computations using the CKKS HE scheme. However, since a solution that builds upon HE protocols should be restricted to compute low degree polynomials, the non-polynomial activation functions, such as Maxpooling and ReLU, are forced to be evaluated in the clear by the party who owns private query data. This leaks the feature maps, from which adversaries may deduce the model weights. To solve this problem, low-degree polynomial approximation to estimate non-lineat functions. Unfortunately, it will affect the inference accuracy, while leading to huge computation overhead.
et al., 2016; Brutzkus et al., 2019)
et al. (2016) and Chen et al. (2019) use
On the other hand, several libraries (Mohassel & Zhang, 2017: Knott et al., 2021; Wagh et al., 2019) employ primarily MPC techniques in secure predictions, which provide linear and non-linear protocols through the usage of oblivious transfer (OT), garbled circuit (GC) and secret sharing. In particular, CryptTen (Knott et al., 2021) performs linear operations based n-out-of-n additive secret sharing over the ring Zzz.. However, it requires a trusted third party to assist the secure pre- diction process, which is unrealistic in the real-world setting. CrpytGPU (Tan et al., 2021) builds on CrypTen, working in the 3-party setting using replicated secret shares. Although the scalability poor, it introduces an interface to losslessly embed cryptographic operations over secret-shared values in a discrete somain into floating-point operations, which can implement the whole inference process on the GPU. Recently, Keller & Sun (2022) proposed secure quantized training protocol that outperforms CryptGPU in the cryptographic performance. Unfortunately. this work cannot be applied in HFL and is not comparable to Gua IF... The main reasons are: (1) Gua rdHFI. and Keller & Sun (2022) are concerned with completely different tasks. GuardHFF builds on the stan- dard HFL setting, where multiple parties collaboratively train individual models with the assistance of server. Keller & Sun (2022) focuses on the outsourced training scenario, i.c., multiple servers jointly execute standard model training algorithm to obtain a well-trained model. (2) Moreover, the protocols in Keller & Sun (2022)
designed for specific network architecture, i.e., quantized neural networks, which cannot be applied to the general models in CuardHFL. Therefore, Keller & Sun (2022) and GuardHFT are two fully orthogonal works.
Many other works focus on hybrid protocols, in which they combines the advantages of HE and MPC to improve prediction efficiency (Juvekar et al., 2018; Mishra et al., 2020; Rathee al., 2020; Huang et al.. 2022). CrypTFlow2 (Rathee et al., 2020) implements two class of protocols, HE- based and OT-based, for linear operations. For non-linear layers, they also design efficient protocols based on OT. Cheetah (Huang et al., 2022) improves CrypTFlow2 with customized HE- linear protocols and improved OT-based non-linear protocols. HE- te-trasfforrmr (Boemer et al., 2019a) employs nGraph-HE for evaluation of linear operations, and ABY framework for GC to evaluate non-linear functions (Demmler et al., 2015). Since non-linear operations cannot be parallelized between query data, GC is inefficient, especially for large networks with thousands of parameters. In contrast, our GuardHFL avoids the use of heavy cryptographic tools like HE and OT, and only employs secret sharing and PRFs to achieve high efficiency, confidentiality and practicability.
20


Reviews:
[{"review_id": "nzFX_TB6g8u", "reviewer": "Reviewer_q3RN", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a privacy-preserving federated learning protocol. Unlike prior works, it avoids running the training in a privacy-preserving manner but instead uses local training on sample-answer pairs that have been obtained using private inference.\n", "summary_of_the_review": "Interesting idea with too many open questions in the execution\n", "strength_and_weaknesses": "This is an interesting idea, but not enough attention is given to privacy issues regarding the dataset used for local training (line 13 in Algorithm 1). If this dataset is useful to the model it intuitively must leak something about the other parties' datasets, and if it doesn't, how can it improve the model? In light of that, I don't think it's entirely fair to compare to prior work where the training happens completely in private.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "mTMUKwnm7YF", "reviewer": "Reviewer_VwDD", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a secure querying scheme for heterogeneous federated learning (HFL). HFL is a setting where clients in collaborative learning will train a model with different model architectures. Hence the global model cannot be directly aggregated/averaged from local client models. The proposed GuardHFL can query models on other clients to get predictions of their private data without publicly sharing the query data by multi-party encryption between Query Client, Answer Client and the Server. Experiments on r SVHN, CIFAR10, and Tiny ImageNet show the efficiency of GuardHFL. \n", "summary_of_the_review": "I unfortunately lack the expertise to evaluate the encryption method itself. The HFL motivation is interesting, but can benefit from more justification. ", "strength_and_weaknesses": "\nImproving the privacy protection techniques in FL when clients have different model architectures is an important and timely topic. \n\nThat being said, the motivation of the proposed method for federated learning does not seem to be super strong. The local training, querying and retraining approach is taken as granted. However, it is not clear to me that this should be the default, or even a popular algorithm for so-called heterogeneous FL. The authors cited (Li & Wang, 2019; Zhu et al., 2021), but the approaches in the two papers seem to be different from Figure 1 (without considering the encryption). The experiments compared with other encryption methods in collaborative learning. I would encourage the authors to think about ways to justify the training paradigm, and make stronger connections between the training paradigm and the proposed method. \n \nI would also encourage the authors to discuss more about the application setting, and limitation of the proposed method.GuardHFL assumes the communication between clients and server are stable with possibly multiple communication targeted specific clients (query and answer), which seems to be only applicable to the  cross-silo FL, not cross-device FL. Algorithm 1 line 4-14 also seems to suggest a large communication cost. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Vm3ZrB2-fYD", "reviewer": "Reviewer_Gysq", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper provides a practical framework for heterogeneous federated learning (HFL). Existing HFL frameworks suffer from privacy leakage through query data communication. Compared to prior work in HFL or adaptation with privacy-preserving feature, the proposed framework 1) is lightweight and efficient, 2) provides formal privacy guarantees. The key idea is to use several cryptographic primitives to perform secure query-data sharing, secure model prediction and secure result aggregation. This paper demonstrates the efficiency through extensive experiments. ", "summary_of_the_review": "I think this paper provides an efficient and privacy-preserving framework. The effectiveness/ efficiency is demonstrated through extensive experiments", "strength_and_weaknesses": "Strengths: 1. Compared to prior works with no privacy protection or adaptation of existing frameworks, this proposed framework is efficient and customized. \nWeaknesses: 1. Secure model predictions requires customized design for particular model architectures. I am wondering how to deal with some privacy-sensitive layers, like for example, batch normalization.\n2. It would be better if this paper could provide some attacks from an adversary perspective to demonstrate privacy.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject