DReS-FL: Dropout-Resilient Secure Federated Learning for Non-IID Clients via Secret Data Sharing
Authors: Jiawei Shao
Yuchang Sun
Songze Li
Jun Zhang
Venue: NeurIPS 2022 Accept (2022)
Keywords: Federated Learning
non-IID problem
client dropouts
privacy-preserving learning
Lagrange coded computing

Abstract: Federated learning (FL) strives to enable collaborative training of machine learning models without centrally collecting clients' private data. Different from centralized training, the local datasets across clients in FL are non-independent and identically distributed (non-IID). In addition, the data-owning clients may drop out of the training process arbitrarily. These characteristics will significantly degrade the training performance. This paper proposes a Dropout-Resilient Secure Federated Learning (DReS-FL) framework based on Lagrange coded computing (LCC) to tackle both the non-IID and dropout problems. The key idea is to utilize Lagrange coding to secretly share the private datasets among clients so that each client receives an encoded version of the global dataset, and the local gradient computation over this dataset is unbiased. To correctly decode the gradient at the server, the gradient function has to be a polynomial in a finite field, and thus we construct polynomial integer neural networks (PINNs) to enable our framework. Theoretical analysis shows that DReS-FL is resilient to client dropouts and provides privacy protection for the local datasets. Furthermore, we experimentally demonstrate that DReS-FL consistently leads to significant performance gains over baseline methods.

DReS-FL: Dropout-Resilient Secure Federated Learning for Non-IID Clients via Secret Data Sharing
Jiawei Shao', Yuchang Sun', Songze Li Jun Zhang
'Hong Kong University of Science and Technology $ Hong Kong University of Science and Technology (Guangzhou)
(jiawei. shao, yuchang. sunnsssssnnseee cecconnect.ust.hi. ust.hk, {songzeli, (eijaan)) hk
Abstract
Federated learning (FL) strives to enable privacy-preserving training machine learning models without centrally collecting clients' private data. Despite its advan- tages, the local datasets across clients in FL are -on-independent and identically distributed (non-IID), and the data-owning clients may drop out of the training process arbitrarily. These characteristics will significantly degrade the training per- formance. Therefore, we propose Dropou--Restee RRssiliet Secure Federated Learning (DReS-FL) framework based on Lagrange coded computing (LCC) to tackle both the non-lID and dropout problems. The key idea to utilize Lagrange coding to secretly share the private datasets among clients so that each client receives encoded version of the global datase[" and the local gradient computation over this dataset is unbiased. To correctly decode the gradient at the server, the gradient function has to be a polynomial in a finite field, and thus we construct polynomial integer neural networks (PINNs) to enable our framework. Theoretical analysis shows that DReS-FL is resilient to client dropouts and provides strong privacy guarantees. Furthermore, we experimentally demonstrate that DReS-FL. consistently leads to significant performance gains over baseline methods.
Introduction
Federated learning (FL) ①1 is machine learning framework in which central server coordinates a large number of clients to collaboratively train a shared model. The key idea of FL is to train the model locally by individual clients and aggregate updates globally by the server. The main target to provide privacy protection for clients' local samples and solve the "data islands" problem. However, local data are typically non-independent and identically distributed (non-IID), the model divergence during the local update may lead to unstable and slow convergence (2] 3 4. With many clients involved in the training, some of the clients could drop out the training process unexpectedly (due to poor connectivity, battery level, etc), and it will cause detrimental model performance (5). Thus, effective mechanisms are needed to tackle the non-III data distribution and client dropouts, while preserving the privacy of local datasets, which motivates this work.
To alleviate the non-IID problem, existing methods typically follow algorithm-bused approaches 2 7 8 [ and add regularization terms to mitigate the model divergence. However, these methods dropout-resilient evidenced by the empirical results in [10]. This be explained by the greatly varying data distributions among different rounds. Another fold of strategy for dealing with
the non-IID problem is data-centric approach [I]] [2] 13. 14, 15, 16 I7
[8]
19 20 21). which
generates extra training samples to construct a more balanced data distribution for each client. The common practices are share the synthesized samples
[5 [6]
GAN-based augmented
II the context of this paper, we use the global dataset to denote the concatenation of the clients' datasets.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

data [17 19 20, 21]. However, these methods may leak private information about local datasets and violate the privacy criterion in FL.
In this work, we develop a Propout-Resilien Secure Federated Learning (DReS-FL) framework to address the above problems via Lagrange coded computing (I.CC) 22 The key idea of LCC is to encode the datasets using Lagrange polynomials that create computational redundancy across the sriaacy-rreservng way to tolerate client dropouts. Before the training starts, the clients secretly share their encoded datasets with each other. This allows clients to access an encoded version of the global dataset that solves the data heterogeneous problem. In each communication round of federated training, the clients perform local gradient computations on the mini sii--btthes sampled from the encoded datasets. After collecting the uploaded computation results from surviving clients, the server performs polynomial interpolation to decode the global gradien for model training. Therefore, the training process DReS-FI. is made equivalent to centralized training and eliminates the non-lll) and dropout problems. With respect to privacy protection, the proposed framework has two salient features:
workers
It guarantees the privacy of local datasets during data sharing, i.e., no private information can be inferred from the encoded data even a certain number of clients collude.
It achieves the same privacy guarantee as the secure aggregation protocols, i.e., the server learns no information about the private dataset from single client's 's computation result.
Note that to correctly decode the gradient at the server, the gradient has to be a polynomial function in finite field, which is main design challenge of DreS-FL. To sum up, our main contributions are summarized as follows:
The proposed DReS-FL framework provides unified approach to tackle two critical problems of FL, namely, non-lID data distribution and client dropouts. Meanwhile, it maintains privacy and security guarantees such that no information about local datasets can be leaked beyond the global model parameters.
We construct polynomial integer neural nelworks (PINNs) to ensure that the gradient is a polynomial, so that cryptographic primitives can be applied for secure computation. PINN consists of affine transformation layers with parameters constrained in an integer set, and it adopts the quadratic function as the activation function. The convergence analysis of DReS-FL with PINNs is also provided.
We conduct extensive experiments on FL benchmark datasets to demonstrate the effective- ness of DReS-FL. is shown that DReS-FL outperforms baseline methods under the setting where local datasets are heterogeneous and clients may drop out of the training process arbitrarily.
Related Works
Non-IID data and client dropouts. Training with heterogeneous data is a unique challenge for FL [. which significantly affects the convergence performance [5]. The client dropouts exacerbate the non-IID problem the data distributions among different rounds could vary greatly. Many algorithm-based methods 2 6
7 g 9 attempt to mitigate the clients' model divergence, but these methods cannot solve the essence of the non-IlI problem due the intrinsic difference between minimizing the local empirical loss and minimizing the global empirical loss. Another line of work adopts data-centric methods [17 [8] 19 20 (1 to modify the local distributions. Ideally, perfect data sharing mechanism should achieve that the local datasets have the same distribution as the global dataset while maintaining the privacy guarantee. Common practices include sharing
raw datasets [1], 12), synthesized samples, [13] 14
15
16] or augmented data [17
18 19 20 21].
However, these works cannot fully preserve local data privacy in an information-theonm
sense A special data-centric method is the secret coding scheme, which has been widely utilized
2
"The global gradient corresponds to the stochastic gradient computed from mini-batches that uniformly sampled from the global dataset. For simplicity, we consider that clients only perform one local stochastic gradient descent (SGD) step in each communication round. Note that the proposed DReS-F FL framework, as discussed in Appendix D. can be extended to more general cases in which clients can run multiple local SGD steps.

---

Gradient decoding model updating
Model Sharing
Server
Clients
Local gradient computations
Dropout
Preparation: Secret data sharing
Figure 1: The DReS-FL system model. At the beginning of training, the clients secretly share the local datasets with each other. Then, the model parameters are iteratively trained by (1) local gradient computations and (2) gradient decoding and model updating until convergence.
in homomorphic encryption (HE)
30 and multiparty computation (MPC)
techniques 31 32 B3]. This coding scheme allows computations to be performed on encrypted
data and has been used for Sravacy-preservinn machine learning [26] 30 33]. However. the HE
methods often suffer from time-consuming cryptographic tools, and MPC techniques are difficult to
large number of clients. Recently, distributed secure machine learning frameworks [3 35 have been proposed for logistic regression problems. They apply Lagrange coding for secret data sharing and approximate the Sigmoid function by polynomial function. This paper proposes DReS-Fi FL to further extend these works to train deep neural networks in the FL setting.
generalize such primitives
Secure aggregation. It has been shown recently that the clients' updates in FL may reveal substantial information about the local datasets, and the private training data can be reconstructed through model inversion attacks [36 37
38) To prevent information leakage from the local models, secure 42 have been developed to allow for global aggregation without revealing the parameters of clients' models. Even if some clients may drop out, these protocols can still recover the aggregated results of the surviving clients. Existing protocols essentially rely on two main principles, including pairwise random- seedd agreement for mask cancellation and secret sharing of the random seeds construct the dropped masks [9 43
aggregation protocols 39 40
42 However, these approaches may suffer from severe performance degradation in non-IID settings, since the surviving clients in each round vary greatly, and thus the aggregate gradient biased towards different data distributions. Different from previous works, our proposed DReS-FL framework achieves the same privacy guarantee while solving the data heterogeneity problem.
3 System Model
We consider federated learning framework as shown in Fig. T that consists of one central server and data-owning clients. Each client E [N] holds local dataset (X,, Y,) of size mis where X, E xdm represents the set input features of dimension d, and Y; E Rm, xdy corresponds to the output vector of dimension dy- Accordingly, the size of the global dataset (X, Y) which
concatenates all local datasets (X,, Y.). Vi [N] denoted as m - = m,. The clients aim to jointly train neural network based on their local datasets without sharing private data samples. Particularly, gradients are computed locally and aggregated globally. However, the local data may be highly heterogeneous, and the clients may drop out at any time unexpectedly, which makes the training process unstable. Our goal to improve the convergence performance by secret data sharing while preserving the privacy of local datasets.

---

3.1 Lagrange Coded Computing for Federated Learning
The Lagrange coded computing (LCC) framework enables private computing in distributed settings to provide resiliency and efficiency [22] The key idea is using Lagrange coding to encode the data for redundant distributed computing, which fits nicely with federated learning due to its dropout- resiliency and privacy requirement. Specifically, the clients share their encoded datasets with each other and perform gradient computation over the encoded samples. The server decodes the aggregate gradient after receiving the uploaded computation results from clients. To provide a strong privacy guarantee for the datasets and correctly decode the gradient at the server, the gradient function should be a polynomial function in a finite field. However, existing neural networks cannot satisfy this requirement, since the datasets are in the real field and the gradients non-polynomial.
Polynomial integer neural networks. We define class of polynomial integer neural networks (PINNs) to ensure that the gradient is polynomial function in a finite field F, with a prime number p. First, we transform the dataset (X, Y) from the real domain to the finite domain (X, Y). Besides, a PINN consists of affine transformation layers (e.g., fully connected layers and convolutional layers) and utilizes the quadratic function the activation function. The model parameters of PINNS defined in the integer set Zp _ {-1
- Given selecting the mean squared error (MSE) as the loss function, the gradient of the input samples is a multivariate polynomial with integer coefficients, i.e. g w)) & V |Y - f(X: w) E Z"", where d represents the number of model parameters. Denoting the number of quadratic activation layers L, the degree of the gradient function g [X, Y;) is deg(g) 2
function f(X; w) and
In particular, avoid wrap-around when computing gradient in the finite field Fp, we assume the prime number p is sufficiently large without leading to overflow errors in the integer set Zp.
Lagrange coding. The proposed DReS-FL framework uses Lagrange polynomials to achieve D-resilient, -prrivae, and K-efficient coding scheme. D-Resiliency means that the global gradient can be decoded by the server in the presence of up to D client dropouts. T-privacy denotes that no information about local datasets can be inferred from the encoded data even if up to T clients collude. K K-efficiency corresponds the complexity of the coding scheme. Specifically, each private dataset is split into K shards in Lagrange coding, and the size of the encoded dataset is proportional to 1/K. Therefore, increasing the value of K reduces the communication overhead of data sharing. The following theorem characterizes the (D,T, K) -achhievve coding scheme, and its proof is available in Section IV of (22].
Theorem 1. Given the client number N and the degree of the grudient function deg(g), D-resilient, T-private, and K -efficientt Lagrange coding scheme is achievable, as long as
D+ deg(g))
Remark 1. As shown in Theorem [ there is a tradeoff among resiliency (D), privacy (T), and efficiency (K). As the sum of T and increases, the proposed framework tolerates fewer client dropouts. Specifically, the maximum value of D is N 1 deg(g) by setting T = K 1.
Remark 2. Setting the privacy parameter T 2 1, the gradient computation over the encoded samples leaks no private information according to the data process inequality. This implies that the proposed DReS-FL. framework achieves the same privacy guarantee as the secure aggregation protocols. Specifically, the server learns no information about the private dataset from single client's computation result.
DReS-FL Framework
DReS-FL consists of two main phases, shown in Fig. 1 In the first phase, the private datasets transformed from the real domain to the finite field, and data-owning clients secretly share datasets by Lagrange coding. Then, the server and the clients train PINN iteratively via (1) local gradient computations and (2) gradient decoding and model updating.
MMor details about how to calculate the degree of the gradient are deferred to Appendix E.

---

X, = 1 18 ...... & xz-
Client
Client
X
X 1
XN++
X
Client N
Client
X
& Xesa
Lagrange
X;
Coding
The secret data sharing scheme in the DReS-FL framework. Every client E [A]] secretly shares the local dataset X, to other clients j E [N]) by sending the encoded samples X; The client j will receive X, X i-ji X
Figure 2:
as encoded version of the global dataset.
4.1 Data Transformation and Secret Sharing
To guarantee information-theorelt privacy, each client has mask the datasets in finite field F, using uniformly random matrices. Firstly, the local datasets (X;, Y;) are converted from the real domain to the finite field (X,, Y,). Considering an element-wise function ( = + that transforms real value to non-negative number by adding a proper scalar we define X & Round(2' (XX), where the rounding operation is element-wiss wise that quantizes each entry to its closest integer. and I controls the quantization loss, We adopt the notation D to represent the global dataset, which is the concatenation of all the local datasets (X,, Y,) for E [N].
After converting the private datasets to the finite field, the clients adopt (D, K -)achieeabblle Lagrange coding to encode local data for secret sharing, First, each client E [N] partitions its local
dataset to K shards as X (())
1T and Y, Y
Y(KY? Assuming that
is divisible by K, we have X F and {(A)) F, F. xd, for & E [K]. A large value of K helps to reduce the complexity secret data sharing. Then, the clients add padding from T uniform random masks to the data samples for privacy protection. Each client E [N] forms the following
polynomials u; Fp + F dataset:
my xd.
and Fp + FT
xd, of degree K + T - 1 to encode the local
2 2 B
K+T + Σ
2 B3
4
II
U
II jE[K+T]VE +T]((R)
(2)
34 Bj
By. 3j
kE[K]
* (T)
k K+1
vz(z)
4
(K)
II
2 2 Bj
K+T
V(
II
- B
(3)
33
Be Bj
K
JT(K+TIM()
k=K+1
JE
where (U!')'s and (V(')ss are random noise matrices uniformly sampled from F3 xd
x and
F xd, respectively. These matrices mask the local datasets and provide privacy guarantee against up to colluding workers. The clients and the server agree on K + T distinct elements { (8,,..,8347)) from the finite field F, in advance. Particularly, setting = 3r. for k [K].
reconstruct the data shard (Bx), v. (3x)) distinct elements {ai,..., aN selected from Fp to encode the private datasets, where {az]ze[M] n
( Y(()), All the clients use the same N
{.3x 1BxhNelke [E[[K+T] Ø. Each client i obtains N encoded datasets (Xizjs Yz-z)) (u.(a;), va; for j E [N], where each (X.z,, Y..,) is sent to client j from client i. All the received encoded
'The scalar could be the absolute value of the minimum entry in dataset, which is set to in the experiments.

---

Table 1: Primary notations and descriptions
Notation
Description
Notation
Description
The global datasct that is a concate-
Transformed dataset at client finite filed Fy
(X,, Y.)
D
nation of clients' datasets (X,, Y,) [[V]
(X] y!())
k-th data shard at client
(Xi-), Y (
Encoded dataset sent from client to client j
Concatenation of received encoded
Row selection matrix C for data sampling in round and the corre- sponding index set Z
(X,, Y,)
datasets (Xi- x)+))) for
C I
[N] at client
Local mini-batch sampled from (X,, Y,) at client j in round t
(X )
k-th global mini batch sampled from DD in round t. hased on Iz
based on I+
Algorithm 1 DReS-FL
Input: Local datasets (X,, Y,) for E [M], batch size 6, initialized parameters w ( Zg" distinct elements {eg)) and (K++7)) prime number p, training round T, learning 77-
Output: Model parameter w(r).
1: Clients encode the local datasets according to and B and deliver them to other clients.
2: for / ., do
Server sends the model parameters w(+) to the clients.
4: for j 1,...,N do
5: Client j performs gradient computation on mini-batches (X) (2+,, (T.)).
6: Upload local computation results g X (Tr) Yx(Tr), (+)) to the server.
7:
end for
Server receives at least (K
1) uploads then
(1()) Decode K global gradients g( z, 2, ( for & E [A] by polynomial interpolation.
9:
Convert gradients from the finite field the integral domain g (X (()) (!)) by 6
10:
11:
Update the model by based on the aggregate gradient Σt g
(k) Y (k)
(()).
12:
end if
13:
end for
N E Ipmxd and YT I E Imnxdy for E [N]. Accordingly, the number of samples in the Y, -....... N N-jll
datasets at client j are represented as (X,. Yj), where X, Xf- X
encoded dataset is m E my Fig. demonstrates the secret data sharing scheme.
4.2 Federated Training
Local Gradient Computation. The server randomly initializes PINN at the beginning of the training process, and the model parameters are constrained to an integer set Z, during the training process. In each communication round, the server sends the model parameters to the clients, and they compute the stochastic gradient over the mini-batches with size b. Particularly, we assume that all the clients use the same row selection matrix C(+) {0, 1] bb for data sampling in each round { and the mini-batch at each client j E [N] is determined by (x(T.) Y c(9) [X,, Y,].
(I.)
Here, Ie = (1(() ....., ....,,()) ((") C [im] is a randomly selected index set in the 4-th round with 4 E [M] for E bb) The entries ( satisfy (( 1,4 (I,) Y (Tr) (()) (X
for E [b), and other entries are set to zero. Each
client j computes the stochastic gradient x(X)
(T)) (()) mod p
in the finite field, and uploads the result to the server. Particularly, each ( 3 (()):
aj, where two (K + de-ree polynomial functions uy, : IF, + Fixd, and VI, E, + Fb pxxd, are defined as
amounts to an evaluation of the polynomial ğ(uz, (z), (z, (2); (+)) at the point
*This can be achieved by setting the same random seed across all the clients. The weighted sampling method has been adopted in this work, where the number of sampled data from Xi is proportional to for i E [N]. Note that other sampling schemes also be applied in DReS-FL.

---

follows:
K+T
Uz, (2) A
())
II
Bj
U
2- Bj
(4)
PR 9
I
BE - 33
ke[k]
JE[K+TJMA
k=
jE[K+T|MR)
VI, (2)
Y
I
II JE(K+T]M(A)
3g - 83
2 2-
+
K4T
II
PA - 33
2 3/
(5)
kE[K]
k=
+1
JE[K+TJM()
with
X 1
-C X
MIT
X IT E p
U c( / f
U E e
-C ( (K)T
T E Bxdd,
v Ia [V V V E Fxdd,
I
(k) Y Every (X
1k) (uz, (3x), Vz, (3x)) for le [K] is a global mini-batch selected from the
I
global dataset D.
Gradient Decoding and Model Updating. According to and (5), the server can obtain the global (k)
gradients g(X²) L YI, w(()) for & E [K] by evaluating the polynomial (uuz, (z), Vz, (z); w()) at the point 3k. But, the server needs first recover the coefficients of this polynomial, which is
composition of the encoding polynomials ( (uz, (z), Vz, (z)) and the gradient function g. As the degree of the composite polynomial is deg (K + 1), the server requires at least ( (K+T- 1)+11
local computation results (i.e., evaluation points) to interpolate i This implies that the proposed DReS-F FFL framework can tolerate most D N deg(g)( +T -T-1) 1) client dropouts.
After decoding the global gradients, the server converts them from the finite field to the integer set Zp (k) Y/lk)
by g (XI,
(()) = (
X
I, I w(())) where (>) is an element-wise function defined
(1:
(k)
follows:
(+)) =
2
2 p-p if
if 0 <<>< 2
p-1 p-1 <><3pp
(6)
As assume that the prime number p is sufficiently large, the converted gradients do have overflow errors. Thus, the central sever updates the global model by w((-1) ((
:()), where denotes the learning rate and bK represents the global batch size Q(2) is stochastic quantization function to ensure the model parameters are in the integer set Zp after updating, which is defined as follows:
Q Σ 1 g(X
(k) y (*)),
[2] 2 +1
with probability - (2- [2])
Q(=)
with probability - [=].
Besides, the probability of rounding to [=] is proportional to the proximity of = to stochastic rounding unbiased. The overall procedure is summarized Algorithm
|=] so that the
5 Convergence Analysis
In this section we characterize the convergence performance of PINNs, which relies on the fact that the global gradients in the training process are unbiased. Define the empirical risk as f(w) & ExX,Y)= Y f(X; w)ll and the corresponding gradient as 9e (w) - TE ExXXY)~ The variables (X, Y) are drawn from the distribution of the global dataset D. To prove that DReS-FL guarantees convergence to the optimal model parameters, we first present the following assumptions to facilitate the analysis.
Y; w)]
Assumption 1. (L-smoothness) There exists a constant L > 0 such that for all W1, W2 E Zdn, we have [|ge (w1) ge (w22ll I L|[w] wzll2-
'Note that if server cannot receive enough results due to the client dropouts, training protocol continues to the next epoch without gradient decoding and model updating.
The global batch size corresponds the number of samples used for gradient computations in each round.
7

---

Table 2: Test accuracy (%) of different methods. Each experiment is repeated five times. Best results are shown in italic and second best results are in bold.
Dataset
MNIST
Cashion-MNin
EMNIST
CIEAR-10
CIFAR-IO
SVHN
FedAvg
96.17 + 0.05
81.20 + 0.07
71.50 0.28
89.54
67.71
83.82 0.20
FedAvg-IS
97.06 + 0.10
85.94 = 0.16
77.09 0.33
0483=0.00
68.92
85.27 0.09
SCAFFOLD
71.89 1 3.92
55.22 1 1.83
55.15 5.99
54.17 1 9.13
29.97
51.27 1 3.43
DReS-FI. (Ours)
97.38 + 0.08
86.60 + 0.32
78.04 + 0.29
90.31 + 0.19
69.15 + 0.27
86.04 + 0.15
Centralized
97.99 + 0.04
89.02 = 0.77
82.45 = 0.23
90.37=0.. 0.12
71.12 0.09
86.18 = 0.03
Assumption 2. (Unbiased and variance-bounded stochastic gradient) There exists constant o > 0
such that any stochastic gradient g (XXI, X w w()) satisfies (X : = ge (w(())
(k)
(k).
(1) (k)
[1 (k) : - ge (w)) s²²
and E Assumption (Unbiased and variance-bounded rounding operation) There exists a constant 7 > such that for any R, the stochastic quantization operation Q(-) satisfies EE [Q(z)] and E [[[Q(=) = I 2222
With the above preparations, we have the following theorem which ensures the convergence. The proof is deferred to Appendix A.
Theorem 2. (Convergence) Denote w as the first-order optimal solution. With Assumption 3
selecting the learning rate as 1 - O (1/v7) such that V - - - L/2 2 > 0, after T times of model updates, have:
((w(0)) ((ww)
Lo²
E
[l (w(())))))]]]] S
2bK T (7² + 1).
(8)
T/
6 Experiments
6.1 Experimental Setup
Dataset. We evaluate our proposed algorithm on several benchmark datasets: MNIST
45
Fashion-
MNIST [46,, EMNIST (Balanced) [47). CIFAR-10 10 48 CIFAR-100 [48]. and SVHN
49
Specifi-
cally, the extra training samples in the SVHN dataset are not utilized. To simulate the non-IID data distribution, we assume there are N = 20 clients the learning system and adopt the skewed label sort dataset by the labels, divide it into N shards, and assign one shard to each client. To simulate the client dropouts in the training process, we consider extreme scenario, where the dropout rate of each client is set to 0.99 with probability of 0.5 or is uniformly sampled from [0, 0.1] otherwise, More details of the datasets are deferred to Appendix B.
partition (50) to shuffle the datasets. Specifically,
Model structures. We adopt a multi-layer perception (MLP) with two hidden layers for the image classification tasks on MNIST, MNNIS,, and EMNIST datasets. Each hidden layer contains 64 neurons. For CIFAR-10, 100, and SVHN datasets, we resize the input images from 32 x 32 224 x 224 and adopt the convolutional layers of pretrained VGG model to extract 25088-dimensiona. features. To classify the extracted features, we select a two-layer MLP model with 4096 hidden units each. The baseline methods train the neural networks the real field and select the rectified linear unit (ReI.U) function as the activation function. In each communication round, clients perform one SGD step for the local model update.
DReS-FL. Our method adopts the same size PINNS to replace MLPs in the federated training, and the degree of gradient is deg(g) 8. Particularly, the extracted features from the last convolutional layer of VGG19 are secretly shared with other clients. We set the parameters K = and T = in the Lagrange coding, and the minimum number of clients needed decode the global gradient is 9.
Baselines. In the experiments, data-centric approaches
35 are not compared lack strong privacy guarantees while others 33
15
since some of them 11
15 17
34
35 cannot
support federated neural network training with multiple clients. We select algorithm based methods
as basclines, including FedAvg (1, FedAvg with importance sampling (FedAvg-IS) [5] 52)
and
8

---

98
88
78
75
Accuracy (96)
96
62
77
Arcurty
80
Accurccy
70
94-
92
75
FedAvg
Fedlug
Fedhug-1S
RedAugg-15
Fedhwg-15
Dites-Fi (Duns]
Dite-FFi (Surs)
(Gurs)
90
70
60
Round
Round
Round
(a) MNISTT
(b) Fashion-MNIST
(c) EMNIST
88
185
Acorrsy (N)
80
Acourecy
MN
Accuracy
84
86
82
80
FedAug-1S
FedAug
Red
FvsAva
78
Fedlig
FedAig
DR [Ours]
(Ours)
76
(Ours)
86
60
CIFAR-10
(e) CIFAR-100
(1) SVHIN
Figure 3: Test accuracies on different datasets.
SCAFFOLD (2]. since these methods can be easily combined with secure aggregation methods Particularly, we assume that the FedAvg-IS method knows the dropout distribution, and the local computation results weighted by the participation probability (i.e., 1 dropout probability) to mitigate bias aggregation. Besides, we also select the centralized training scheme as a performance upper bound, where the server access all the clients' datasets for model training.
6.2 Performance Evaluation
The experimental results are shown in Table and Fig. 3 The FedAvg method achieves worse performance than the centralized training scheme. This is attributed to the non-III data and client dropouts. The FedAvg-IS method improves the test accuracy com- pared with FedAvg, but there is still noticeable performance gap with the centralized training scheme. It shows that using the knowl- edge dropout distribution can partially compensate for the biases in the aggregated models, but the local data distributions are still heterogeneous and degrade the performance. Besides, SCAFFOLD has a low accuracy all the settings. As the frequency of updating local control variates is low, the estimation of the update direction highly inaccurate such that the model does not converge shown in Fig. These results are consistent with the findings [0] Our OReS-FL. method is superior all the baseline methods as the server obtain global gradients after polynomial interpolation. In addition, DReS-FL. achieves comparable performance to the centralized training scheme on some datasets, which demonstrates the effectiveness of our proposed framework in solving the non-IID and dropout problems.
60
Accuray (M
50
20
20
0.5
1.0
1.5
Round
1e3
Figure 4: Test accuracy of SCAFFOLD MNIST.
Conclusions
This paper proposed Propout-Resilien Secure Federated Learning (DReS-FL) framework via Lagrange coded computing (LCC) to simultaneously solve the data heterogeneity and dropout problems of FL, while providing privacy guarantees for the local datasets. The polynomial integer neural networks (PINNs) have been constructed to ensure that the server can correctly decode the
'Note that the secure aggregation mechanism has not been applied in the experiments, since the quantization step in secure aggregation may degrade the performance of the baselines.

---

global gradient without privacy leakage. Extensive experimental results validated the effectiveness of the proposed method. Potential limitations our method include that the degree of the gradient in PINN increases exponentially with the number of layers, which hinders training deep model for complex tasks. Besides, performing multiple local SGD steps largely increases the finite field size as the range of results grows exponentially with the number of multiplications, and thus it will lead to substantial communication overhead in model transmission. Despite some limitations, we believe DReS-FL is promising framework for many practical FL application scenarios given its effectiveness in resolving both the non-IID and client dropout problems, while with strong privacy guarantees.
Acknowledgement
The work of Songze Li is part supported by the National Nature Science Foundation of China (NSFC) Grant 62 1060057, and Foshan HKUST Projects FSUST20-FYTRI04B.
References
[1] McMahan, B., E. Moore, D. Ramage, et al. Communication-effcienn learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273-1282. PMLR, 2017.
[2] Karimireddy, S. P., S. Kale, M. Mohri, et al. Scaffold: Stochastic controlled averaging for federated learning. In International Conference on Machine Learning, pages 5132-5143. PMIR,
2020.
[3] Kairouz, P., H. B. McMahan, B. Avent, et al. Advances and open problems in federated learning, Foundations and Trends@ in Machine Learning, (M1-2))1-2101 2021.
[4] Hsieh, K., A. Phanishayec, O. Mutlu, et al. The non-iid data quagmire of decentralized machine learning. In International Conference on Machine Learning, pages 4387-4398. PMLR, 2020.
[5] Luo, M., F. Chen, D. Hu, al. No fear of heterogeneity: Classifier calibration for federated learning with non-iid data. Advances in Neural Information Processing Systems, 34, 2021.
[6] Sahu, A. K., T. Li, M. Sanjabi, et al. On the convergence of federated optimization in heteroge- neous networks. arXiv preprint 18X. 2.1277, 3:3, 2018.
[7] Li. Q., B. He, D. Song. Model-contrastive federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10713- 2021.
[8] Acar, D. A. E., Y. Zhao, R. M. Navarro, et al. Federated learning based on dynamic regularization. arXiv preprini arXiv:2 1.04663, 2021.
[9] Hsu, T.-M H., H. Qi, M. Brown. Federated visual classification with real-world data distribution. In European Conference on Computer Vision, pages 76-92. Springer, 2020.
[10] Li, Q., Y. Diao, Q Chen, et al. Federated learning non-iid data silos: An experimental study, arXiv preprint trXiv:2102.02079 2021.
[11] Zhao, Y., M. Li, L. Lai, et al. Federated learning with non-iid data. arXiv preprint 2018.
arXiv:1806.005::
[12] Yoshida, N., T. Nishio, M. Morikura, et al. Hybrid-ft for wireless networks: Cooperative
learning mechanism using non-iid data. In ICC 2020-2020 IEEE International Conference on Communications (ICC), pages 1-7. IEEF, 2020.
[13] Yoon, T., S. Shin, S. J. Hwang, et al. Fedmix: Approximation of mixup under mean augmented federated learning. In International Conference on Learning Representations. 2020.
[14] Sun, Y., J. Shao, S. Li, et al. Stochastic coded federated learning with convergence and privacy guarantees. arXiv preprint arXiv:2201.10092, 2022.
[15] Jeong, E., S. Oh, H. Kim, et al. distillation and augmentation under non-iid private data. arXiv preprint arXiv: 117, 2018.
wwwtmmiicattonfffffiieeee on-device machine learning: Federated
[16] Hao, W., M. EI-Khamy, J. Lee, et al. Towards fair federated learning with zero-shot data augmentation. Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pages 3310-3319. 2021.
10

---

[17] Zhang, L., B. Shen, A. Barnawi, et al. Feddpgan: federated differentially private generative adversarial networks framework for the detection of covid- pneumonia. Information Systems Frontiers, 23(6))1403-1415 2021.
[18) Nguyen, D. C., M. Ding, P. N. Pathirana, et al. Federated learning for covid-19 detection with generative adversarial networks in edge cloud computing. IEEE Internet of Things Journal, 2021.
[19) Jeong, E., S. Oh, H. Kim, et al. Communication-efficin on-device machine learning: Federated distillation and augmentation under non-iid private data. arXiv preprint arXiv: 1811.1149/. 2018.
[20] Zhu, Z., J. Hong, Zhou. Data-free knowledge distillation for heterogencous federated learning. In International Conference on Machine Learning, pages 12878-12889, PMLR, 2021.
[21] Li, Z., J. Shao, Y. Mao, et al. Federated learning with GAN-based data synthesis for non-IlI) clients, 2022.
[22] Yu. Q.. S. Li, N. Raviv, et al. Lagrange coded computing: Optimal design for resiliency, security, and privacy. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 1215-1225. PMLR, 2019.
[23] Shamir, A. How to share secret. Communications of the ACM, 22(1 (1:6111):2261333 19799
[24] Gentry, C. Fully homomorphic encryption using ideal lattices. In Proceedings of the forty-first
annual ACM symposium on Theory of computing, pages 169-178. 2009.
[25] Bachrcch, R., Dowlin, Laine, et Cryptonets: Applying neural networks to encrypted data with high throughput and accuracy. In International conference on machine learning, pages 201-210. PMLR, 2016.
[26] Hesamifard, E., H. Takabi, M. Ghasemi. Cryptodl:
towards deep learning over encrypted data. In Annual Computer Security Applications Conference (ACSAC 2016), Los Angeles, California, USA, vol. 11. 2016.
[27] Graepel, T., K. Lauter, M. Naehrig. MI confidential: Machine learning on encrypted data. In International Conference on Information Security and Cryptology, pages 1-21. Springer, 2012.
[28] Yuan, J. S. Yu. Privacy preserving proppppppantioon neural network learning made practical with cloud computing. IEEE Transactions Parallel and Distributed Systems, 25(1):212-2211 2013.
[29] Han, K., S. Hong, Cheon, et Logistic regression on homomorphic encrypted data at scale. In Proceedings of the AAAI Conference Artificial Intelligence, vol. 33, pages 9466-9471. 2019.
[30] Wang, Q. M. Du, X. Chen, et al. word vector training. IEEE Transactions on Knowledge and Data Engineering, 30(12):2381
collaborative model learning: The case of
2393, 2018
[31] Nikolaenko, V., U. Weinsberg, S. Joannidis, et al. presecypressrving ridge regression on hundreds of millions of records. In 2013 IEEE symposium on security and privacy, pages 334-348, IEEE, 2013.
[32] Gascón, A., P. Schoppmann, B. Balle, et al. Privacy-preserving distributed linear regression high-dimensional data. Proc. Priv. Enhancing Technol., 2017(4):345 -644, 2017.
[33] Mohassel, P., Zhang. Secureml: A system for scalable privacy-preserving machine learning. In 2017 IEEE symposium security and privacy (SP), pages 19-38. IEEE, 2017.
[34] So, J., B. Güler, A. S. Avestimehr. Codedprivatenl: A fast and for distributed machine learning. IEEE Journal on Selected Areas Information Theory, (11):441451, 2021.
framework
[35] So, J., B. Guler, S. Avestimehr. A scalable approach for privacy-preserving collaborative machine learning. Advances in Neural Information Processing Systems, 33:8054-8066, 2020.
[36] Shokri, R.. M. Stronati, C. Song, et al. Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP), pages 3-18. IEEF, 2017.
[37] Nasr, M., R. Shokri, A. Houmansadr. Comprehensive privacy analysis of deep learning. In Proceedings of the 2019 IEEE Symposium on Security and Privacy (SP), pages 1-15. 2018.
11

---

[38] Geiping, J., H. Bauermeister, H. Dröge, et al. Inverting gradients-how easy is it to break privacy in federated learning? Advances in Neural Information Processing Systems, 33:16937-16947, 2020.
[39] Bonawitz, K. and Ivanov, Vladimir and Kreuter, Ben and Marcedone, Antonio and McMahan, Brendan and Patel, Sarvar and Ramage, Daniel and Segal, Aaron and Seth, Karn. Practical secure aggregation for privacy- privacy-preserving machine learning. In proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, pages 1175-1991. 2017.
[40] Kadhe, S., N. Rajaraman, O. O. Kayluoglu, al. Fastsecagg: Scalable secure aggregation for preeevyp federated learning. arXiv preprint arXiv:2009. 11248, 2020.
[41] Yang, C.-S., J. So, C. He, et al. Lightsecagg: Rethinking secure aggregation in federated learning. arXiv preprint arXiv:2109. 14236, 2021.
[42] Jahani-Nezhad, T., M. A. Maddah-Ali, S. Li, al. Swiftagg+: Achieving asymptotically
optimal communication load in secure aggregation for federated learning. arXiv preprint arXivv:203.. 13660, 2022.
[43] So, J., B. Güler,, A. Avestimehr. Turbo-aggregate: Breaking the quadratic aggregation barrier in secure federated learning. IEEE Journal on Selected Areas in Information Theory, 2(1):479 489, 2021.
[44] Bell, J. H., K. A. Bonawitz, A. Gascón, al. Secure single-server aggregation with (poly) logarithmic overhead. In Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communications Security. pages 1253-1269. 2020.
[45] LeCun, Y., L. Bottou, Y. Bengio, al. Gradient-based learning applied document recognition. Proceedings of the IEEE, 86(1 1):2728-2324, 1998.
[46] Xiao, H., K. Rasul, R. Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv: 108.077708.07777, 2017.
[47] Cohen, G. S. Afshar, J. Tapson, al. Emnist: Extending mnist handwritten letters. In 2017 international joint conference on neural networks (IJCNN) pages 2921-2926. IEEE, 2017.
[48] Krizhevsky, A., G. Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[49] Netzer, Y., T. Wang, A. Coates, et al. Reading digits in natural images with unsupervised feature learning. 2011.
[50] Hsich, K., A. Phanishayec, O. Mutlu, et al. The non-iid data quagmire of decentralized machine learning. In International Conference on Machine Learning, pages 4387-4398. PMLR, 2020.
[51] Ren, J., Y. He, D. Wen, al. Scheduling for cellular federated edge learning with importance and channel awareness. IEEE Transactions on Wireless Communications, 19(11):7690-7703 2020.
[52] Kairouz, P., B. McMahan, B. Avent, al. Advances and open problems federated learning. Foundations and Trends@ in Machine Learning, 14(1-2):1-210, 2021.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 7
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all thcoretical results? [Yes] The proofs are provided in Appendix A.
3. If you experiments...
12

---

(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or URL)? [No] The code is
proprietary, but the public datasets used in the experiments and the instructions needed for reproduction are presented Section
Appendix B.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Please refer to Section Caetionnaand Appendix B.
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [Yes] See Section 6
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix B.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] See Appendix B.
(c) Did you include any new assets either in the supplemental material or as a URL? [No] The code is proprietary and the datasets used in our work are public datasets.
(d) Did you discuss whether and how consent
obtained from people whose data you're using/curating? [N/A] We choose the commonly-use datasets in the experiments.
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "c9N7JlvXjfp", "reviewer": "Reviewer_ULSx", "summary": "The paper considers secure federated learning (FL) and aims to tackle the challenges of non-IID data distribution across clients and dropouts. The paper proposes that each client first secret shares their data to all other clients using Lagrange Coded Computing. This involves quantizing the data and appropriately mapping it to a finite field. Then, gradients are computed over the secret shared data. Since secret sharing operates over a finite field (as opposed to real numbers), the paper proposes to use *polynomial integer neural networks*, where quadratic function is used as a non-linearity (instead of ReLU or other typical functions) and a mean squared loss is used (instead of cross-entropy loss). The server first ‘decodes’ the gradients computed on secret shared data to obtain the gradient on the quantized data. Then, it sums the gradients and takes a step to update the global model. ", "questions": "1. The key technical idea in the paper involves computing gradients on secret shared data. However, the paper does not give details on the gradient computation on secret shared data. It simply mentions that the gradient (when loss is MSE and non-linearity is square function) would be a multivariate polynomial in $(\\bar{X}, \\bar{Y})$ and parameters $\\mathbf{w}$. Without any details, it is difficult to understand how the server decodes $\\tilde{g}(\\tilde{X}_j^{(I_t)}, \\tilde{Y}_j^{(I_t)})$ in line 9 of Algorithm 1. During secret sharing, all the $K$ shards of each client are encoded together, while the server decodes gradients for each shard separately. It is important to add details and provide a proof of how the decoding happens. \nCurrently, it is not even clear how the gradient $\\tilde{g}(\\tilde{X}_j^{(I_t)}, \\tilde{Y}_j^{(I_t)})$ is defined. Is this an average of the gradients in  batch $I_t$? \n\n2. In Theorem 2, the security is mentioned in terms of  $\\tilde{g}(\\tilde{X}_j^{I_t}, \\tilde{Y}_j^{I_t})$. However, the server decodes the gradients to obtain the gradient on uncoded data $\\tilde{g}(\\bar{X}_j^{(I_t)}, \\bar{Y}_j^{(I_t)})$ for $j\\in[K]$. Is it not that the privacy for server should be proved in terms of the decoded gradient? In other words, Theorem 2 should be stated in terms of $\\tilde{g}(\\bar{X}_j^{(I_t)}, \\bar{Y}_j^{(I_t)})$? Overall, it is not clear how the server only obtains an *aggregation* of gradients. It is important to expand on this. \n\n3.  In FL all communication typically happens through the server. It is important to mention how a client will securely communicate the secret shared data to all other clients.\n\n4. The proposed scheme involves several approximations as stated below:\n* Quantization to convert real-field data into integers \n* Square function as a non-linearity instead of ReLU\n* Mean squared loss instead of cross-entropy loss\n* Stochastic quantization during update (eq. (6))\n* Neural network weights are restricted to be integers\n\nIt seems like these approximations will negatively impact the accuracy. However, performance evaluation in Fig. 2 depicts that the proposed method consistently outperforms FedAvg for all datasets. It would be important to discuss why these factors mentioned above do not impact accuracy, and if they do, then are any optimizations performed to overcome them. Otherwise, the results appear to be a bit 'magical'.\n\nIn particular, it is known that the cross-entropy loss typically performs better than MSE. Furthermore, quadratic approximation for ReLU is known to severely degrade accuracy in some settings, see e.g., Mishra et al., “DELPHI: A Cryptographic Inference Service for Neural Networks,” Usenix 2020. Why these issues are not observed in the proposed method, and if they are observed how they are overcome?\n\n5. Convergence analysis considers population loss instead of empirical loss. Convergence analysis often considers empirical risk. It would be helpful to comment on this.\n\n6. To tackle non-iid nature, there are solutions such as FedProx, which change the local optimization problem at the client side, but still perform averaging at the server side. It is straightforward to use secure aggregation with FedProx as one solution of privacy + data heterogeneity. How would the proposed method compare against FedProx + secure aggregation (or similar solutions)?\n\n7. It would be helpful to specify if the paper focuses on cross-silo setup or cross-device setup. Since cross-device setup consists of a large number of clients, performing secret sharing across all clients may be costly. On the other hand, the problem of dropouts is less severe in the cross-silo setup. It would be good to add a discussion on this.\n\n8. Footnote on page 4 says that “The constant scalar c could be the absolute value of the minimum entry in dataset (X, Y).“ Since the dataset $(X,Y)$ is distributed across clients, how is it possible to find such a scalar $c$? Also, if the value of $c$ is data dependent it may not be appropriate to call it a ‘constant’.\n\n9. What field size $p$ is used in the experiments?\n\n10. Experiments use MLP with two hidden layers with $K=1$ and $T=1$. It is not clear why the degree of gradient as a polynomial would be 8. Can the authors give more details?\n", "limitations": "As mentioned in the Weaknesses, the proposed solution would incur heavy costs during the secret sharing phase. Also, dropouts cannot be tolerated during secret sharing. In addition, as mentioned in Questions, the proposed scheme relies on several ‘approximations’, and it would be important to discuss their impact on the accuracy and/or speed of convergence.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "*Strengths:* The paper considers an important problem of tackling data heterogeneity and privacy in FL. These issues are intellectually challenging and practically important.\n\n*Weaknesses:* \n\n1. The proposed solution requires each client to secret share their data to all other clients. Clearly, this is going to incur substantial burden in terms of computation and communication costs. The paper does not explicitly characterize the computation and communication costs for secret sharing client data. (It just mentions that the ‘sharing’ parameter $K$ impacts the costs, but explicit analysis seems to be missing.)\n\n2. One of the main motivations of the paper is to tackle client dropouts. However, it looks like all clients need to be present during the phase of secret sharing data. This assumption needs to be clearly stated and it would be good to add a discussion around this. In particular, if the communication cost of transferring secret shared data to other clients is large, then this phase will take considerable time. In this case, not allowing any dropouts during secret sharing and requiring all clients to be present can be a major limitation. \n\n3. The paper does not give sufficient details in terms of gradient computation and securely transferring secret shared data to clients. (Specific comments in the Questions section.)\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "_uSod9T8pxw", "reviewer": "Reviewer_kTxQ", "summary": "This paper proposes a new framework named Dropout-Resilient Secure Federated Learning (DReS-FL) to address the non-iid and client dropout issues in federated learning. By leveraging Lagrange coded computing (LCC) and secret sharing, DReS-FL enables clients to securely share their data to build a ‘public data set’ and participate in the learning task. To further guarantee the gradients are polynomial functions in a finite field, this paper introduces the polynomial integer neural networks (PINNs) to enable the framework. The experiment results show the DReS-FL proposed can achieve higher accuracy when facing clients dropping out.\n", "questions": "1.\tTo perform Secret Sharing, the parameter K should be set before the learning task start, but whether K is constant?\n2.\tLine 156 said a large value of K can reduce the communication cost, can you explain the reason? (as I think more shards may bring more communication times between different clients)\n3.\tHow can the clients establish communication?\n4.\tAre there any rules for sharing the data? How can a client determine whom he should share the shards with?\n", "limitations": "See the questions.\nAs the communication cost is often a bottleneck for FL, involving massive communication may further worsen this issue; please consider providing the discussion.  \n", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1.\tAddressing the FL dropout issue is a hot topic, and the proposed framework is very interesting. \n2.\tThis paper proposes the polynomial integer neural networks (PINNs) and proves PINNs can ensure the correct decode of the global gradient without privacy leakage.\n3.\tThe DReS-FL has been formally analyzed to be resilient to a certain number N of client dropouts.\n\nWeakness:\nThe secret sharing mechanism is not well presented.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Uxo6XOuG1w_", "reviewer": "Reviewer_FXUt", "summary": "This paper proposes a Dropout-Resilient Secure Federated Learning framework based on Lagrange coded computing to\ntackle two problems in FL: the non-IID data distribution and client dropout.\n\nThe clients share their datasets secretly using LCC to handle the non-IID problem and client dropouts. At the beginning of the training, the clients secretly share their private datasets with each other via Lagrange coding. This allows clients to access an encoded version of the global dataset. After collecting the computation results from a certain number of surviving clients, the server performs polynomial interpolation to decode the gradient.\n\nThe authors introduce the idea of polynomial integer neural networks (PINNs) for their proposed framework to work, since they need to correctly decode the gradient at the server, and that the gradient has to be a polynomial function in a finite field. \n\nThe authors provide some theoretical analyses and numerical experiments to back their claims. ", "questions": "Have the authors also looked the LEAF datasets for the experiments? LEAF seems to be the standard dataset for FL papers, with natural client partitions and other FL constraints. ", "limitations": "As pointed out by the authors, \n\n- the generality of the idea is questionable. Requiring neural networks to be polynomial integer neural networks (PINNs) is an assumption that probably cannot be hold in practice. \n- Similarly, the degree of the gradientin a PINN increases exponentially with the number of layers of the neural network, limiting training larger models\n- multiple local SGD steps is yet not supported, which is quite common in practical FL settings", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper is well written and it is very easy to read\n- The paper is back by theoretical analyses and also backed by numerical experiments\n- The idea of the paper is quite novel and interesting, even though it may not be extensible to big and practical neural networks and practical FL settings\n\n\nWeaknesses:\n- As correctly pointed out by the authors, the generality of the idea is questionable. Requiring neural networks to be polynomial integer neural networks (PINNs) is an assumption that probably cannot be hold in practice. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "EQKiT7Q41r6", "reviewer": "Reviewer_gqjh", "summary": "This paper aims to address the non-iid and dropout problems in (privacy-preserving) FL using the Lagrange coded computing framework. Apply the LCC framework, it proposes a class of network called Polynomial Integer Neural Networks.\n", "questions": "(1) How are the baseline setups different than the main method (i.e., how do they tackle non-iid, dropout problems)?   \n(2) The author claims \"We observe that FedAvg is influenced by the dropout problem due to the biased gradients\" (in lines 295-296). How is this inferred from Table 1 and figure 2?   \n\nOverall, the paper is well written and shows enough technical strengths. Please address the questions in experiment setups and analysis. I am willing to raise my score if the experiments are good as well.\n", "limitations": "The author has stated the limitation in the main paper.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n    (1) The paper is clearly written - problems, motivations, and intuition of the main methods are easy to understand.  \n    (2) The method is novel and original (the paper shows how to use PINN to adopt LCC in the setting).  \n    (3) Theoretical guarantees are given.  \n    (4) The limitation is stated - the degree of the gradient grows fast so that the network can not be deep, which is reasonable.  \n\nWeakness:\n    (1) The experiment setup for baselines is not clearly stated - How is the FedAvg setup? Is it combined with secure aggregation methods?   \n    (2) The experiment results are not clearly explained. Please see below for concrete questions.  \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept