Picking Daisies in Private: Federated Learning from Small Datasets
Authors: Michael Kamp
Jonas Fischer
Jilles Vreeken
Venue: ICLR 2022 Submitted (2022)
Keywords: federated learning
distributed
sparse data
daisy chain
small datasets

Abstract: Federated learning allows multiple parties to collaboratively train a joint model without sharing local data. This enables applications of machine learning in settings of inherently distributed, undisclosable data such as in the medical domain. In practice, joint training is usually achieved by aggregating local models, for which local training objectives have to be in expectation similar to the joint (global) objective. Often, however, local datasets are so small that local objectives differ greatly from the global objective, resulting in federated learning to fail. We propose a novel approach that intertwines model aggregations with permutations of local models. The permutations expose each local model to a daisy chain of local datasets resulting in more efficient training in data-sparse domains. This enables training on extremely small local datasets, such as patient data across hospitals, while retaining the training efficiency and privacy benefits of federated learning.

Under review as a conference paper at ICLR 2022
PICKING DAISIES IN PRIVATE: FEDERATED LEARNING FROM SMALL DATASETS
Anonymous authors Paper under double-blind review
ABSTRACT
Federated learning allows multiple parties collaboratively train a joint model without sharing local data. This enables applications of machine learning in settings of inherently distributed, undisclosable data such as in the medical do- main. In practice, joint training is usually achieved by aggregating local models, for which local training objectives have be in expectation similar the joint (global) objective. Often, however, local datasets are so small that local objec- tives differ greatly from the global objective, resulting in federated learning fail. We propose novel approach that intertwines model aggregations with per- mutations of local models. The permutations expose each local model daisy chain of local datasets resulting in more efficient training in data-sparse domains. This enables training on extremely small local datasets, such as patient data across hospitals, while retaining the training efficiency and privacy benefits of federated learning.
INTRODUCTION
How can learn high quality models when data is inherently distributed into small parts that cannot be shared or pooled, as we for example often encounter in the medical domain (Rieke et al., 2020)? Federated learning solves many but not all of these problems. While it can achieve good global models without disclosing any of the local data, it does require sufficient data to be available each site in order for the locally trained models to achieve a minimum quality. In many relevant applications, this is not the case: in healthcare settings we often have as little as a few dozens of samples (Granlund et al., 2020; Su al., 2021; Painter et al., 2020), but also domains where DL. generally regarded as highly successful, such as natural language processing and object detection often suffer from a lack of data (Liu et al., 2020; Kang et al., 2019).
In this paper, we present elegant idea in which models are moved around iteratively and passed from client to client, thus forming a dhisy-chain that the model traverses. This daisy- chisinch allows us to learn from such small, distributed datasets simply by consecutively training the model with the data availalbe at each site. We should not do this naively, however, since it would not only lead to overfitting common problem in federated learning which cause learning to diverge (Haddad- pour and Mahdavi, 2019) - but also violate privacy, since a client can infer from model upon the data of the client received it from (Shokri al., 2017). To alleviate these issues, we propose an approach to combine daisy-chaining of local datasets with aggregation of models orchestrated by coordinator, which we term federated daisy-chaining (FEDDC).
In a nutshell, in a daisy-chain round, local models are send to a coordinator and randomly redis- tributed to clients, without aggregation. Thereby, each individual model follows its own random daisy-chain of clients. In an aggregation round, models are aggregated and redistributed, as in stan- dard federated learning- Our approach maintains privacy of local datasets, while provably guar- antees improvement of model quality of convex models with suitable aggregation method which standard federated learning cannot. For non-convex models such as convolutional neural networks, it improves the performance upon the otate-of-the-art on standard benchmark and medical datasets.
Formally, we show that FEDDC allows convergences on datasets so small that standard federated learning diverges by analyzing aggregation via the Radon point from PAC-learning perspective. We substantiate this theoretical analysis by showing that FEDDC in practice matches the accuracy of model trained the full data of the SUSY binary classification dataset, beating standard federated

---

Under review as a conference paper at ICLR 2022
learning by wide margin. In fact, FEDDC allows achieve optimal model quality with only 2 samples per client. In an extensive empirical evaluation, we then show that FEDDC outperforms vanilla federated learning (McMahan et al., 2017), naive daisy-chaining, and FedProx (Li et al., 2020a) on the benchmark dataset CIFARIO (Krizhevsky, 2009), and more importantly on two real- world medical datasets.
In summary, our contributions are as follows.
FEDDC, an elegant novel approach to federated learning from small datasets via a combi- nation of daisy-chaining and aggregation,
theoretical guarantee that FEDDC improves models in terms of c, 6--guaranttes, which standard federated averaging can not,
a thorough discussion of the privacy aspects and mitigations suitable for FEDDC, including an empirical evaluation of differentially private FEDDC, and
an extensive set of experiments showing that FEDDC substantially improves model quality for small datasets, being able to train Re on a pneumonia dataset on as little as 8 samples per client.
2 RELATED WORK
Learning from small datasets is a well studied problem in machine learning. In the literature, we find among others general solutions, such as using simpler models, and transfer learning (Torrey and Shavlik. 2010), to more specialized ones, such as data augmentation (Ibrahim et al., 2021) and few- shot learning (Vinyals et al., 2016; Prabhu et al., 2019). our scenario, however, data is abundant, but the problem is that the local datasets at each site are small and cannot be pooled.
Federated learning and its variants have been shown to learn from incomplete local data sources, e.g., non-iid label distributions (Li et al., 2020a; Wang et al., 2019) and differing feature distribu- tions (Li et al., 2020b; Reisizadeh et al., 2020a), but were proven to fail in case of large gradi- ent diversity (Haddadpour and Mahdavi, 2019) and too dissimilar label distribution (Marfoq al., 2021). For very small datasets, local empirical distributions may vary greatly from the global data distribution-
the difference of empirical to true distribution decreases exponentially with the sample size (e.g., according to the Dvoretzky-Kinn Beeieeky-KKeeee-Wae -oefeeee-ooooowiizzz. inequality). for small sample sizes the difference can be substantial, in particular if the data distribution differs from Normal distribu- tion (Kwak and Kim, 2017).
FedProx (Li et al., 2020a) is a variant of federated learning that is particularly suitable for tackling non-iid data distributions. It increases training stability by adding a momentum-like proximal term the objective functions. This increase in stability. however, comes the cost of not being privacy- preserving anymore (Rahman et al., 2021). We compare FEDDC to FedProx in Section 7.
We can reduce sample complexity by training networks only partially, c.g., by collaboratively train- ing only shared part of the model. This approach allows training client-specific models in the medical domain (Yang et al., 2021), but by design cannot train global model. Kiss and Horvath (2021) propose a decentralized and communication-efiin grates models over decentralized network and stores incoming models locally at each client until sufficiently many models are collected on each client for averaging step, similar to Gossip feder- ated learing (Jelasity et al., 2005). The variant without averaging is similar to simple daisy-chaining which compare in Section 7. FEDDC is compatible with any aggregation operator, includ- ing the Radon point (Kamp et al., 2017) and the geometric median (Pillutla et al., 2019). It can also be straightforwardly combined with approaches improve communication-effiiennn such as dynamic averaging (Kamp et al., 2018), and model quantization (Reisizadeh et al., 2020b).
variant of federated learning that mi-
3 PRELIMINARIES
We assume iterative learning algorithms (cf. Chp. 2.1.4 Kamp, 2019) A: X' y H + H. that
update a model h E H using a dataset C X x y from an input space X' and output space y, i.c.,
h++1 A(D, ht). Given a set of m N clients with local datasets DI...., D C X x y drawn
2

---

Under review as a conference paper at ICLR 2022
iid from data distribution D and loss function {: h" E H. that minimizes the risk
x:yxy x - R, the goal is to find single model
e(h) B(z,)) ((hh(v))))
In centralized learning, the datasets are pooled as D = Uie(m] D' and A is applied to D until convergence. Note that applying A on D can be the application to any random subset, e.g., as in mini-batch training, and convergence is measured in terms of low training loss, small gradient, or small deviation from previous iterate. In standard federated learning (McMahan et al., 2017), A applied in parallel for N rounds on each client locally to produce local models hy...., hm- These models are then centralized and aggregated using an aggregation operator agg 7 - H, i.e. h agg(h,,..... hm) The aggregated model is then redistributed to local clients which perform another b rounds of training using h as a starting point. This is iterated until convergence of h. In the following section, we describe FEDDC.
4 METHOD
We propose federated daisy chaining extension to federated learning and hence assume setup where we have m clients and one designated coordinator node.' We provide pseudocode our approach Algorithm 1.
The client Each client trains its local model in each round on local data (line 4), and sends its model to the coordinator every rounds for aggregation, where b is the aggregation period, and every d rounds for daisy chaining, where d is the daisy-chaining period (line 6). This re-distribution of models results in each individual model following a daisy-chain of clients, training on each local dataset. Such daisy-chain is interrupted by each aggregation round.
The coordinator Upon receiving models (line 10), in daisy-chaining round (line 11) the coor- dinator draws a random permutation т of clients (line 12) and re-distributes the model of client to client (i) (line 13), while in aggregation round (line 15), the coordinator instead aggregates all local models (line 16) and re-distributes the aggregate to all clients (line 17).
Communication complexity Communication between clients and coordinator happens in O( (fmaaz fmaz) rounds, where (maz is the overall number of rounds. Although inherently higher than in plain federated learning, the overall amount communication in daisy chained federated learning is still low. In particular, in each communication round, each client sends and receives only a single model from the coordinator. The amount of communication per communication round is thus linear in the number of clients and model size, similar to federated averaging.
&
In the following section we show that the additional daisy-chaining rounds ensure convergence for small datasets terms of PAC-like e, 8 E-guarantees.
THEORY
Next, we theoretically analyze the key properties of FEDDC in terms of PAC-like (c, 8) guaranteess. For that, we make the following assumption the learning algorithm A.
Assumption ((c, -ggaaraant.). The learning algorithm A applied all datasets drawn iid from D of size n 2 no E N produces a model E H. such that with probability ō E (0, it holds for e that
IP (E(h) c)
The sample size no is monotone function in ö and e, i.e., for fixed no is monotonically increasing with and for fixed 6 it is monotonically decreasing with (note that typically ny is a polynomial in -1 and log(6 ))).
'Tiis star-topology be extended to hierarchical networks in straight-forward manner. Federated learn- ing can also he performed in decentralized network via gossip algorithms (Jelasity al., 2005)
3

---

Under review as a conference paper at ICLR 2022
Algorithm Federated Daisy-Chaining FEDDC
Require: daisy-chaining period d. aggregation period b, learning algorithm A, aggregation operator agg, m clients with local datasets D², .',..., D
1: initialize local models hg.....hh" hn""
2: at local client in round t
3: draw random set of samples S from local dataset D
4: he + hi
5:
i % d = d or t % b = b- - then
6: send hj coordinator
7: end if
8:
9: at coordinator in round f
receive models ......, hr if t. % d = d then
11:
12:
draw permutation a of [1.m] at random for all i E [m]] send model hy to client (i) end if
13:
14:
15:
if t % b = b- 1 then Tu 4 agg .e., hT²") send hr to all clients
16:
17:
18:
end if
19:
Here e(h) the risk defined in Equation We will show that aggregation for small local datasets can diverge and that dhisi-chaining can prevent this. For this, we analyze the development of (c, 5), guarantees on model quality when aggregating local models with and without daisy-chaining.
It is open question how such an ( -ugarantee develops when averaging local models. Existing work analyzes convergence (Haddadpour and Mahdavi, 2019; Kamp et al., 2018) or regret (Kamp et al., 2014) and thus gives generalization bound. Recent work on generalization bounds for fed- crated averaging via the NTK-framework (Huang et al., 2021) is promising. but not directly com- patible with daisy-chaining: the analysis of Huang et al. (2021) requires local datasets to be disjoint which would be violated by Maass-chaining round. Using the Radon point (Radon, 1921) as aggre- gation operator, however, does permit analyzing the development of (c, In particular, it was shown that for fixed the probability of bad models is reduced doubly exponentially (Kamp al., 2017) when aggregate models using the (iterated) Radon point (Clarkson al., 1996). Here, Radon point of set of points S from a space X' ismsiar to the geometric median a point in the convex hull of with a high centrality (more precisely, a Tukey depth (Tukey, 1975; Gilad-Bach Bachhrcchhhhhh et al., 2004) of at least 2). For Radon point to exist, the size of S has to be suf- ficiently large; the minimum size of C X' is denoted the Radon number of the space X' and for X' C IRd the radon number is d + 2.
Let E N be the Radon number of H. be a learning algorithm as in assumption and be convex. Assume m 2 Th many clients with h E N. For > 0, 8 E (0, 1] assume local datasets Dı,..., Dm, of size larger than ne)(e, 8) drawn iid from D, and hı,.... Itm be local models trained on them using A. Let ti be the iterated Radon point with f iterations computed on the local models. Then it follows from Thcorem 3 in Kamp et al. (2017) that for all i [ma] it holds that
P (E(CA) > S (r 2))²
(2)
where the probability over the random draws of local datasets. This implies that the iterated Radon point only improves over the local models if ó < -1 Consequently, local models need to achieve a minimum quality for the federated learning system to converge.
Corollary 2. Given a model space H. with Radon number r E N. convex risk e, and a learning algorithm A with sample size n(e, 8)- Given 0 and any h E N. local datasets Dı...., D. with m 2 n are smaller than ni (e, (e.r-)), then federated learning using the Radon point does not improve model quality in terms of (c, 6)-guarantees.
4

---

Under review as a conference paper at ICLR 2022
0.9
nccurncy
0.8
centralized (lest)
0.8
0.7
0.7
0.5
0.5
0.4
train
test
0.4
400
500
100
220
300
400
500
rounds
rounia
(a) FEDDC with d = 1, 10.
(b) Federated learning with Radon point with b = 10.
Figure 1: Results on SUSY. We visualize results in terms of train (green) and test error (orange) for FEDDC (a) and federated learning (b), both using Radon points for aggregation. The network has 441 clients with data points per client. "Optimal performance, i.e., that central model trained on all data, is indicated by dashed line.
In other words, when using aggregation by Radon points alone, improvement in terms of (c, 5)- guarantees is strongly dependent on large enough local datasets. Furthermore, given ō > y-1, the guarantee can become arbitrarily bad by increasing the number of aggregation rounds.
Federated Daisy-Chaining as given in Algo. 1 permutes local models at random, which is in theory equivalent to permuting local datasets. This way, the amount of data visible each model is in- creased. Since the permutation drawn at random, the minimum amount of distinct local samples observed by each model can be given with high probability.
Lemma 3. Given å (0, 1]. m E N clients, and [mm]. if Algorithm with daisy chaining period d E N is run for T E N rounds with
In
T2 d
(m k 1)m
m
then each local model has seen at least distinct datasets with probability
Proof. For m clients with m local datasets, the chance of client to not see dataset after many permutations is (==1)- The probability that each of the clients is seeing m + other datasets is hence
m
m- 1 II
7(n 7(m-k+1)
m
= 771
and corresponds to the probability of each client seeing less than & distinct other datasets. The probability of all clients seeing least k distinct datasets is hence least
-++)m
7(m-k k- 1)m
1/2 2 5
$ 8
171
m
Taking the logarithm on both sides with base (m - 1)/m yields
In
7(m k + 1)m
In m-I
Multiplying with m - + and observing that many daisy-chaining rounds with period d require T= rd total rounds yields the result.
From Lm. it follows that when perform chisy-chaining with clients, and local datasets of size n, for at least dl In ((In(m - 1) In(m)) - + 1)m) rounds, each local model will with probability least 1 be trained at least kn samples.
5

---

Under review as a conference paper at ICLR 2022
Proposition 4. Given model space H. with Radon number E N. convex risk E, and learning algorithm with sample size o(e,5). Given e > (0,r-1) and any N. if local datasets Di...., Dm of size N with 2 rh, then Alg. 1 using the Radon point with
In
M d
In
nole,8) + +1)
TR
improves model quality terms of ( -ggurratteex.
Proof. The number daisy-chaining rounds before computing a Radon point ensure that with prob- abilitt 1 1-8 8 all local models are trained least kn samples with & no(e, 6)/n, i..., each model trained on at least no(e, 6) samples and thus an (e, 5) --guarrattee holds for each model. Since 5 r- this guarantee is improved as detailed in Eq. (2).
To support this theoretical result, compare FEDDC using the iterated Radon point with standard federated learning on the SUSY binary classification dataset (Baldi et al., 2014), training linear model on 441 clients with only samples per client. The results in Figure show that after 500 rounds FEDDC reached the test accuracy of a model that has been trained on the centralized dataset (ACC=0.7H 77) beating federated learning by large margin (ACC=0.66 65).. Before further investigating FEDDC empirically in Section 7, we discuss the privacy-aspects of FEDDC in the following section.
PRIVACY
A major benefit of federated learning is that data remains undisclosed on the lo- cal clients and only model parameters are exchanged. It is, however, possible infer upon local data given model pa- rameters (Ma et al., 2020). In classical federated learning there two types of attacks that would allow such inference: (i) an attacker intercepting the commu- nication of a client with the coordinator obtaining model updates infer upon the clients data, and (ii) malicious ordinator obtaining models to infer upon the data of each client. A malicious client cannot learn about other clients data, since it only obtains the average of all local models. In federated daisy- chaining there is third possible attack: (iii) a malicious client obtaining model updates from another client to infer upon its data.
0.8
0.1
FEDDC
DR-FEDDC 2, 0.01)
0.2
DE-FEDDC (8 2, 0.02)
DP-FEDDC (S
0.05)
5 10
10*
100
10
rounds
Figure Differential privacy results. We show the per- formance of FEDDC (top solid line) compared to runs with clipped parameter updates and added Gaussian noise (dashed lines) CIFARIO with 250 clients.
In the following, we discuss potential defenses against these three types of attacks in more detail. Note that we limit the discussion on attacks that aim inferring upon local data, thus breaching data privacy. For discussion of attacks that aim to poison the learning process (Bhagoji et al., 2019) or create backdoors (Sun et al., 2019) for adversarial examples, we refer to Lyu et al. (2020).
A general and wide-spread approach to tackle all three possible attack types is to add noise to the model parameters before sending. Using appropriate clipping and noise, this guarantees e, 8- differential privacy for local data (Wei et al., 2020) at the cost of a hight-to-moderatt loss in model quality.
Another approach tackle attack communication (i) is to use encrypted communication. One can also protect against a malicious coordinator (ii) by using homomorphic encryption that allows the coordinator to average models without decrypting them (Zhang al.. 2020). This, however, only works for particular aggregation operators and does not allow to perform daisy-chaining. Secure daisy-chaining in the presence of malicious coordinator (ii) can, however, be performed using

---

Under review as a conference paper at ICLR 2022
centrulized (test)
0.8
0.8
nccurncy
0.6
0.4
0.4
0.2
0
train
test
100
200
400
600
800
1,000
rounds
rounds
(a) FEDDC with d 1, 10.
(b) Federated learning with Radon point with b 10.
Figure 3: Synthetic data results. Comparison of FEDDC (left) and FedAvg (right) for training MLPs on synthetic dataset. Mean and confidence test accuracy of each client is reported in orange. where the optimal reachable accuracy, as given by centralized training, is indicated by the dashed black line.
asymmetric encryption. Assume each client creates a public-private key pair and shares the public key with the coordinator. To avoid the malicious coordinator to send clients its public key and act as a man in the middle, public keys have to be announced (c.g.. by broadcast). While this allows sending clients identify the recipient of their model, receiving client can identify the sender. Thus, inference on the origin of a model remains impossible. For a daisy-chaining round the coordinator sends the public key of the receiving client to the sending client, the sending client checks the validity of the key and sends an encrypted model to the coordinator which forwards it the receiving client. Since only the receiving client decrypt the model, the communication is secure.
In standard federated learning, malicious client cannot infer upon the data of other clients from model updates, since it only receives the average model. In federated daisy-chaining, it receives the model from a random, unknown client in each daisy-chaining round. Now, the malicious client can infer upon the membership of a particular data point in the local dataset of the client the model originated from, i.e., a membership inference attack (Shokri et al., 2017). Similarly, the malicious client can infer upon the presence of data points with certain attributes in the dataset (Ateniese et al., 2015). The malicious client, however, does not know the client the model was trained on, i... it does not know the origin of the dataset. Using random scheduling of daisy-chaining and averaging rounds at the coordinator, the malicious client cannot even distinguish between a model from another client or the average of all models. Nonetheless, daisy-chaining opens up new potential attack vectors (e.g., by clustering received models to potentially determine their origins). These potential attack vectors can be tackled by adding noise to model parameters as discussed above, since "[dJifferrntially private models are, by construction, secure against membership inference attacks" (Shokri et al., 2017). To investigate the impact of this privacy technique on FEDDC, we apply it in practice: We train small ResNet on 250 clients using FEDDC with d = 2 and b = 10. Details on the experimental setup can found in Supp. ??,??. Differential privacy is achieved by clipping local model updates and adding Gaussian noise proposed by Geyer et al. (2017). The results shown in Figure indicate that the standard trade-off between model quality and privacy holds for FEDDC well. Moreover, for mild privacy settings the model quality does not decrease. That is, FEDDC is able to robustly predict even under differential privacy.
EMPIRICAL EVALUATION
We evaluate FEDDC against the state-of-the-art in federated learning on synthetic and real world data. In particular, we compare to standard Federated averaging (FedAvg) (McMahan al., 2017), FedAvg with equal communication as FEDDC, FedProx (Li et al., 2020a), and simple daisy-chaining without aggregation. As real world applications we consider the image classification problem CI- FAR10 (Krizhevsky, 2009), publicly available MRI scans for brain tumors?, and chest X-rays for

---

Under review as a conference paper at ICLR 2022
pneumonnia (e.g., from COVID-19)* For reproducibility, provide details architectures, and experimental setup in Supp. ??,?? The implementation of the experiments is publicly available at https:: hattps://anonyneee /anonynonooss.s. 4open scwpnceeeeeeeeeeeeeeeeeee DFt.c//ee////DDBCCCCC
7.1 SYNTHETIC DATA
We first investigate the potential of FEDDC on synthetic binary classification dataset generated by the sklear (Pedregosa et al., 2011) 1) make classiffiiiicccc caaassifiiaiion function with 100 features. On this dataset, we train a simple MLP with 3 hidden layers on m 50 clients with n 10 samples per client. We compare FRDDC with d and b 200 to FedAvg with b 200. The results presented in Figure 3 show that FEDDC achieves an optimal test performance of 0.89 (centralized training on all data achieves a test accuracy 0.88), substantially outperforming FedAvg. The results indicate that the main reason is overfitting of local clients, since for FedAvg train accuracy reaches 1.0 quickly after each averaging step. the following, we investigate how these promising results translate weal-world datasets.
7.2 CIFAR10
To compare FEDDC with the state of the art on real world data, we first consider the CI-
0.8
FEDDC
RodAvg
FAR10 image benchmark. To find a suitable aggregation period for FEDDC and FedAvg, we first run search grid across periods for 250 clients with small versions of ResNet (details in Supp. ??). We report the results in Figure 4 and set the period for FEDDC to 10, and con- sider federated averaging with periods of both and 10. For our next experiment, we equip 150 clients each with ResNet18. To simulate our setting that each client has a small amount of samples, each one of them only receives 64 samples. Note that the combined amount of ex- amples only one fifth of the original training data, hence we cannot expect the typical perfor- mance on this dataset. As NNs are non-convex, Radon points are no longer suitable aggre- gation method, we instead resort to averaging. Results are reported in Table 1. We observe that FEDDC achieves substantially higher accuracy of more than 6 percentage points over federated av- eraging with the same amount of communication. Looking closer, we see that FedAvg drastically overfits, achieving training accuracies of 0.97, similar trends reported in Figure for synthetic data. We further see that daisy-chaining alone, besides its privacy issues, performs worse than FEDDC. Similarly, FedProx with b 10 and 0.1 only achieves accuracy of 0.545.
Accunicy
0.4
10
20
100
200
500
Averaging period b
Figure 4: Averaging periods on CIFARIO. For 250 clients with small ResNets and 64 samples per client, we visualize the performance (higher better) of FEDDC and FedAvg for different ag- gregation periods b.
dataset FedDC
Daisy-Chaining FedAvg(b=11)
FedAvg(b=1)
FedProx
CIFAR10 10
62.8
59.2
51.0
56.3
54.5
MRI
78.4
57.7
75.6
74.1
76.5
Pneumonia
82.5
78.8
79.0
79.9
80.0
Table 1: Results on image data, reported are test accuracies of final model.
7.3 MEDICAL IMAGE DATA
We conduct experiments on real medical image data, which are naturally of small sample size and represent actual health related machine learning tasks. Here, we observe similar trends as for CI- FARIO. For the brain MRI scans, we simulate 25 clients equipped with simple CNNs (see App. ??) and 8 samples each. The results for brain tumor prediction based these scans reported in
8

---

Under review as a conference paper at ICLR 2022
Table 1. Again, FEDDC performs best, beating both FedAvg and FedProx this challenging tasks. For pneumonia, we simulate 150 clients training ResNet18 (see again App. ??) with samples per client. The results in Table not only show that FEDDC again outperforms all baselines, but also highlight that FEDDC enables us to train a ResNet1 8 to high accuracy with as little as 8 samples per client.
DISCUSSION
Empirical evaluation shows that FEDDC drastically improves upon state-of-the-ar methods for fed- erated learning for settings with only small amounts of available data. This confirms the theoretical potential, given by the e, 6-guarantees, of improving model quality, which is unique among federated learning methods.
Using the iterated Radon point aggregation method, and given as few 2 samples per client, FEDDC matches the test accuracy of a model trained on the whole SUSY dataset, outperforming standard federated learning by over 12% points accuracy. This result shows that unlike federated learning, FEDDC does not heavily overfit and is able to learn generalized model, and is consistent with a synthetic prediction task using multi-layer perceptrons.
To study FEDDC in the context of real data, we consider both the standard image benchmark data CIFARIO, as well as two challenging image classification tasks from the health domain where only little data is available. On each of these tasks, FEDDC consistently outperforms state-of-the-art federate learning methods. Similar to before, we observe overfitting of standard federate learning methods. To rule out any effects due to increased communication, we also considered FedAvg with the same amount communication as our method, however, FedAvg shows no improvement.
Through FEDDC, we present effective solution the problem of federated learning small datasets. We further show that our method is able to robustly predict even under the effect of differ- ential privacy, and suggest effective measures based encryption as mitigations against attacks communication or malicious coordinators.
CONCLUSION
We considered the problem of learning high quality models in settings where data is inherently distributed across sites, data cannot be shared between sites, and each site only has very little data available. We propose an elegant, surprisingly simple approach that effectively solves this problem, by combining the idea of model aggregation approaches from federated learning with the concept of passing individual models around while still maintaining privacy.
We showed that this approach theoretically improves models in terms of e, 8 8.8-gurrantees, which state-of-the-att federated averaging can not provide. In extensive empirical evaluations, including challenging image classification tasks from the health domain, we further show that for settings with limited data available per site, our method improves upon existing work by a wide margin. It thus paves the way for learning high quality models from small datasets.
Although the amount of communication is not critical issue for the settings where we intend FEDDC to be used in, it does make for engaging future work to improve communication effi- ciency and hence also enable it for settings with limited bandwidth, e.g., regarding model training mobile devices. Both from a practical, as well as from a security and privacy perspective. would also be interesting study how to formulate FEDDC in decentralized setting, when no coordinator available.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Giuseppe Ateniese, Luigi V Mancini, Angelo Spognardi, Antonio Villani, Domenico Vitali, and Giovanni Felici. Hacking smart machines with smarter ones: How to extract meaningful data from machine learning classifiers. International Journal of Security and Networks, 10(3))137- 150, 2015.
Pierre Baldi, Peter Sadowski, and Daniel Whiteson. Searching for exotic particles in high-energy physics with deep learning. Nature communications, 5(1):1-9, 2014.
Arjun Nitin Bhagoji, Supriyo Chakraborty, Prateek Mittal, and Seraphin Calo. Analyzing federated learning through an adversarial lens. In International Conference Machine Learning, pages 634-643 643. PMLR, 2019.
Kenneth I. Clarkson, David Eppstein, Gary L Miller, Carl Sturtivant, and Shang-Huz Teng. Ap- proximating center points with iterative radon points. International Journal of Computational Geometry & Applications, 6(03):357-377, 1996.
Robin Geyer, Tassilo Klein, and Moin Nabi. Differentially private federated learning: A client level perspective. arXiv preprint arXiv: arXiv:1712.07557 2017.
Ran BaaaddBBachach,,, Amir Navot, and Naftali Tishby. Bayes and tukey meet at the center point. In International Conference on Computational Learning Theory, pages 549-563. Springer, 2004.
Kristin L Granlund, Sui-Seng Tee, Hebert A Vargas, Serge K Lyashchenko, Ed Reznik, Samson Fine, Vincent Laudone, James A Eastham, Karim A Touijer, Victor E Reuter, et Hyperpolar- ized mri of human prostate cancer reveals increased lactate with tumor grade driven by monocar- boxylate transporter Cell metabolism, 1155::05-1144 14, 2020.
Farzin Haddadpour and Mehrdad Mahdavi. On the convergence of local descent methods in feder- ated learning. arXiv preprint 1XXi::990011422) 2019.
Baihe Huang, Xiaoxiao Li, Zhao Song, and Xin Yang. Fl-ntk: A neural tangent kernel- frame- work for federated learning analysis. In International Conference Machine Learning, pages PMLR, 2021.
Marwa Ibrahim, Mohammad Wedyan, Ryan Alturki, Muazzam A Khan, and Adel Al-Jumaily. Aug- mentation in healthcare: Augmented biosignal using deep learning and tensor representation. Journal of Healthcare Engineering, 2021, 2021.
Márk Jelasity, Alberto Montresor, and Ozalp Babaoglu. Gossip-basee aggregation in large dynamic networks. ACM Transactions on Computer Systems (TOCS), 23(3):219 252, 2005.
Michael Kamp. Black-Box Parallelization for Machine Learning. PhD thesis, Rheinische Friedrich- Wilhelms-Unnveerstää Bonn, Universitäts-und Landesbibliothek Bonn, 2019.
Michael Kamp, Mario Boley, Daniel Keren, Assaf Schuster, and Izchak Sharfman. Communication- efficient distributed online prediction by dynamic model synchronization. Joint European Con- ference Machine Leurning and Knowledge Discovery in Databases, pages 623-639. Springer, 2014.
Michael Kamp, Mario Boley, Olana Missura, and Thomas Gärtner. Effective parallelisation for machine learning. In Advances in Neural Information Processing Systems, volume 30, pages 6480-6491, 2017.
Michael Kamp. Linara Adilova, Joachim Sicking, Fabian Hüger, Peter Schlicht, Tim Wirtz, and Stefan Wrobel. Efficient decentralized deep learning by dynamic model averaging. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 393- 409. Springer, 2018
Bingyi Kang, Zhuang Liu, Xin Wang, Fisher Yu, Jiashi Feng, and Trevor Darrell. Few-shot object detection via feature reweighting. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pages 8420-844 8429, 2019.
10

---

Under review as a conference paper at ICLR 2022
Péter Kiss and Tomas Horvath. Migrating models: A decentralized view federated learning. In Proceedings of the Workshop on Parallel, Distributed, and Federated Learning. Springer, 2021.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, Toronto, 2009.
Sang Gyu Kwak and Jong Hae Kim. Central limit theorem: the cornerstone of modern statistics. Korean journal anesthesiology, 70(2):144, 2017.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. In Conference on Machine Learning and Systems, 2020a, 2020a.
Xiaoxiao Li, Mcirui Jiang, Xiaofei Zhang, Michael Kamp, and Qi Dou. Fedbm: Federated learn- ing on non-iid features via local batch normalization. In International Conference on Learning Representations, 2020b.
Pei Liu, Xuemin Wang, Chao Xiang, and Weiye Meng. A survey of text data augmentation. In 2020 International Conference on Computer Communication and Network Security (CCNS), pages 191-195. IEEE, 2020.
Lingjuan Lyu, Han Yu, Jun Zhao, and Qiang Yang. Threats to Federated Learning, pages 3-16. Springer International Publishing, 2020. ISBN 978-3-030-63076-8
Chuan Ma, Jun Li, Ming Ding, Howard H Yang, Feng Shu, Tony QS Quek, and H Vincent Poor. On safeguarding privacy and security in the framework of federated learning. IEEE network, 34(4): 242-248, 2020.
Othmane Marfoq, Giovanni Neglia, Aurélien Bellet, Laetitia Kameni, and Richard Vidal. Feder- ated multi-task learning under mixture of distributions. In Advances in Neural Information Processing Systems. Curran Associates, Inc., 2021.
Brendan McMahan, Eider Moore, Daniel Ramage. Seth Hampson, and Blaise Aguera y Arcas. Communication--fficiee learning of deep networks from decentralized data. In Artificial Intelli- gence and Statistics, pages 1273-1282, 2017.
Corrie A Painter, Esha Jain, Brett N Tomson, Michael Dunphy, Rachel E Stoddard, Beena S Thomas, Alyssa I. Damon, Shahrayz Shah, Dewey Kim, Jorge Gómez Tejeda Zañudo, et al. The angiosar- coma project: enabling genomic and clinical discoveries in cancer through patient-partnered research. Nature medicine, 26(2):181-181 2020.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-28344 2011.
Krishna Pillutla, Sham M Kakade, and Zaid Harchaoui. Robust aggregation for federated learning. arXiv preprint 0rXiv: /3445, 2019.
Viraj Prabhu, Anitha Kannan, Murali Ravuri, Manish Chaplain, David Sontag, and Xavier Amatri- ain. Few-shot learning for dermatological disease diagnosis. In Machine Learning for Healthcare Conference, pages 532-552, PMLR, 2019.
Johann Radon. Mengen konvexer Körper, die einen gemeinsamen Punkt enthalten. Mathematische Annalen, -3-1),113-115, 1921.
KM Jawadur Rahman, Faisal Ahmed, Nazma Akhter, Mohammad Hasan, Ruhul Amin, Kazi Ehsan Aziz, AKM Muzahidul Islam, Md Saddam Hossain Mukta, and AKM Najmul Islam. Challenges, applications and design aspects of federated learning: A survey. IEEE Access, 2021.
Amirhossein Reisizadeh, Farzan Farnia. Ramtin Pedarsani, and Ali Jadbabaie. Robust federated learning: The case of affine distribution shifts. Advances Neural Information Processing Systems, volume 33, pages 21554-21566 Curran Associates, Inc., 2020a.

---

Under review as a conference paper at ICLR 2022
Amirhossein Reisizadeh, Aryan Mokhtari, Hamed Hassani, Ali Jadbabaie, and Ramtin Pedarsani. Fedpaq: A communication-efficiee federated learning method with periodic averaging and quan- tization. In International Conference Artificial Intelligence and Statistics, pages 2021-2031. PMLR, 2020b.
Nicola Rieke, Jonny Hancox, Wenqi Li, Fausto Milletari, Holger R Roth, Shadi Albarqouni, Spyri- don Bakas, Mathieu N Galtier, Bennett Landman, Klaus Maier-Hein, al. The future of digital health with federated learning. NPJ digital medicine, 3(1):1-7, 2020.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference at- tacks against machine learning models. In 2017 IEEE Symposium on Security and Privacy (SP), pages 3-18 18. IEEE, 2017.
Xiaoping Su, Xiaofan Lu, Sehrish Khan Bazai, Eva Compérat, Roger Mouawad, Hui Yao, Morgan Rouprêt, Jean-Philippe Spano, David Khayat, Irwin Davidson, et al. Comprehensive integrative profiling of upper tract urothelial careinomas. Genome biology, 22(1):1-25 2021.
Ziteng Sun, Peter Kairouz, Ananda Theertha Suresh, and Brendan McMahan. Can you really 2019.
backdoor federated learning? arXiv preprint arXiv:1911.020
Lisa Tomey and Jude Shavlik. Transfer learning. In Handbook of research machine learning applications and trends: algorithms, methods, and techniques, pages 242-264. IGI global, 2010.
John W Tukey. Mathematics and picturing data. In Proceedings of the International Congress of Mathematics, volume pages 523-531, 1975.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances in neural information processing systems, 29:3630-3633 2016.
Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris Papailiopoulos, and Yasaman Khazaeni. Federated learning with matched averaging. In International Conference on Learning Represen- tations, 2019.
Kang Wei, Jun Li, Ming Ding, Chuan Ma, Howard H Yang. Farhad Farokhi, Shi Jin, Tony QS Quek, and H Vincent Poor. Federated learning with differential privacy: Algorithms and performance analysis. IEEE Transactions on Information Forensics and Security, 15:3454-3469, 2020.
Qian Yang, Jianyi Zhang, Weituo Hao, Gregory Spell, and Lawrence Carin. Flop: Federated learning on medical datasets using partial networks. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, page 3845-3853, Association for Com- puting Machinery, 2021.
Chengliang Zhang, Suyi Li, Junzhe Xia, Wei Wang, Feng Yan, and Yang Liu. Batcherypt: Effi cient homomorphic encryption for cross-silo federated learning. In USENIX Annual Technical Conference, pages 493-506, 2020.
12


Reviews:
[{"review_id": "cA6t1za11vv", "reviewer": "Reviewer_U9P6", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Discalimer: I am not an expert on federated learning, hence I am not too knowledgeable about what has been proposed in the literature.**\n\nThe idea of increasing the sample diversity by swapping models is simple and natural. The paper deserves credit for proposing this (if indeed  this has not been considered before). On the other hand, the key downside to their approach is the privacy loss. This is what makes the federated model more challenging. In my view, this paper does not adequately analyze the privacy implications. A fair comparison to existing work in my view would look something like this:\n1. Fix a privacy budget. \n2. Prove a formal upper bound on the privacy loss incurred in the proposed model.\n3. Repeat this analysis for state of the art federated learning, and compare the resulting accuracies. \n\nIn this work, step 2 is not addressed beyond experiments, and a quotation from Shokri etal, 2017. \nIn addition, there seem to be multiple issues with the way the paper is currently written, ranging from minor to fairly serious. In its current form, I cannot recommend the paper be published.\n\n1. Assumption 1 feels a little ungainly, and circular. It feels like two things are being thrown together: one is a sample complexity bound for $H$, which is standard. The second is an assumption about the learning algorithm itself (which is what we would like to prove bounds about). Now some assumption on the algorithm is indeed called for, for instance if the aggregation function simply outputs the constant $0$ hypothesis, then the algorithm cannot work. But to sweep this under an assumption that the algorithm works properly given enough samples is somewhat tautological, it is definitely confusing. Perhaps the authors want to state an assumption on the aggregation function instead, which is indeed necessary and would not be circular.\n\n2. Equation (2) quotes a result from the literature, given an upper bound on the error incurred by \"radon point aggregation\". It is conceivable that this can be improved, unless someone proves a lower bound  that  this is the best achievable rate. Perhaps the authors have such a bound in the supplementary material (which I am unable to see)? If not, Corollary 2 is not implied by Equation (2).   Even if such a lower bound is not known, it is fair to say that the this paper can lower the error on each individual model to be aggregated, so that Equation (2) is now more useful. \n\n3. Lemma 3 is a simple corollary of the so-called coupon collector process.\n\n4. For  fair comparison, the improved generalization needs to be balanced by a more formal analysis of the privacy loss. (Perhaps ideas from the shuffle model might be useful here. )\n\n\n\n\n", "summary_of_the_paper": "This paper considers the setting of federated learning where each client holds a small subset $D_k$ of the entire dataset $D$. The goal is to learn a hypothesis with small (generalization) error. The challenge is that each dataset might be too small to guarantee convergence and/or generalization. \n\n1. The  \"obvious\" solution to this is to have multiple clients share data. But this is undesirable because of privacy considerations. \n2. Instead this paper proposes having clients swap \"models\" periodically. The paper argues that from the point of view of the model, it sees a greater diversity of client data, and thus ought to be able to train with better generalization. The paper reports that this improves over existing state of the art work in federated learning in terms of accuracy.\n3. Of course this incurs increased privacy risk. The paper proposes to tackle this through differential privacy. The results reported here in this regard are experimental. The permutation is superficially similar to the recently proposed shuffle model, but no connection is drawn in the paper. \n\n", "summary_of_the_review": "\nAs it stands, the paper seems to benefit from the gains of data aggregation among random subsets of clients, without adequately grappling with the increased privacy risk. For this and the other reasons mentioned above, in my opinion this paper should not be accepted as it stands. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "MsQoSggTmhH", "reviewer": "Reviewer_pbKL", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "It is mentioned in the introduction that \"While it can achieve good global models without disclosing any of the local data, it does require sufficient data to be available at each site in order for the locally trained models to achieve a minimum quality\". I wonder if the authors can elaborate more on this by providing relevant references and a more detailed technical discussion. \n\nA major concern in this work is the assumption of iid data. We know that in federated learning applications, the data is highly heterogenous. I wonder how well the proposed method would generalize to more realistic non-iid setting.\n\nThere is a subtle difference between the communication complexity of the two types of aggregation in the proposed method. When the coordinator aggregates the models and broadcasts the average (during aggregation period), there exist such broadcasting opportunity since all the clients receive the same model. However, during the daisy-chaining period, different models are pushed down to the clients, accounting for more communication complexity. Therefore, the characterization of communication complexity in Section 4 O(t_max/d + t_max/b) needs refinements.\n\nIt is not clear -in theory- how the performance of a federated learning algorithm is improved when using daisy-chaining technique. Proposition 4 is stated quite vaguely. Can the authors elaborate more on this?\n\n", "summary_of_the_paper": "This paper considers a federated learning setting in which the sample size at client is so inadequet that the local objectives greatly differ from the global one. This paper proposes a novel approach that intertwines model aggregations with permutations of local models. By doing so, local models are exposed to several clients' data which ultimately improves the model accuracy.", "summary_of_the_review": "In general, the technical and theoretical results of the paper are quite marginal. A clear discussion demonstrating the effect of daisy-chaining aggregate is missing. For instance, by using such method, how faster a federated learning algorithm e.g. FedAvg would converge.\n\nAnother concern is the assumption of iid data which makes the applicability of the proposed algorithm limited. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "LD40Y0cEji", "reviewer": "Reviewer_oYPW", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- A novel extension to the FedAvg training procedure which includes model sharing between the clients and aggregation of models using iterated Radon point.\n- Although the discussion on privacy concerns of the algorithm is general and may apply to other FL methods, I think that it is a nice addition to the paper.\n- Most experimental details were given. Code was also provided (for the synthetic data experiment only though).\n\nWeaknesses:\n- The comparisons in the empirical evaluation are leaking. Most comparisons are to FedAvg which is rather old and generally isn't very strong. The authors keep stating that their method is state-of-the-art, yet it wasn't really compared against recent methods in order to consider it as such (for example, [1-4]).\n- One of the main claims in favor of the proposed method is increased performance when local datasets are limited in size. Yet the paper misses important related work that addressed this issue as well (e.g., [1, 5]). I think that these studies and similar ones should be addressed in the revised version of the paper.\n- Several alternatives were proposed for aggregating models instead of the standard model averaging (e.g., [6]). I believe the authors should address this line of research as well.\n- The authors state that \"The amount of communication per communication round is thus linear in the number of clients and model size, similar to federated averaging\"; whilst the first part is true, I do not think that it is indeed similar to the communication cost of FedAvg which is sub-linear in the number of clients. At each communication round, FedDC communicates with all the clients while FedAvg (and similar methods) communicate with a small subset of them.\n- In assumption 1, can you please clarify why for fixed $\\epsilon$, $n_0$ is monotonically increasing with $\\delta$?\n- I think that background on the iterated Radon point method is missing. It cannot be expected from the average reader to be familiar with it.\n- In section 7.2 it is stated the FedDC outperforms FedAvg even though they use the same amount of communication rounds. Can you please clarify how many communication rounds were used? I think that an analysis is missing here. Perhaps FedAvg was in overfit state which resulted in decreased generalization. I think that a better approach to compare between the models would be to declare a maximal number of communication rounds and use validation-based early stopping to select the best model.\n\n\n[1] Achituve, I., Shamsian, A., Navon, A., Chechik, G., & Fetaya, E. (2021). Personalized Federated Learning with Gaussian Processes. arXiv preprint arXiv:2106.15482.  \n[2] Collins, L., Hassani, H., Mokhtari, A., & Shakkottai, S. (2021). Exploiting Shared Representations for Personalized Federated Learning. arXiv preprint arXiv:2102.07078.  \n[3] Shamsian, A., Navon, A., Fetaya, E., & Chechik, G. (2021). Personalized Federated Learning using Hypernetworks. arXiv preprint arXiv:2103.04628.  \n[4] Li, Q., He, B., & Song, D. (2021). Model-Contrastive Federated Learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 10713-10722).  \n[5] Hao, W., El-Khamy, M., Lee, J., Zhang, J., Liang, K. J., Chen, C., & Duke, L. C. (2021). Towards Fair Federated Learning with Zero-Shot Data Augmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 3310-3319).  \n[6] Yurochkin, M., Agarwal, M., Ghosh, S., Greenewald, K., Hoang, N., & Khazaeni, Y. (2019, May). Bayesian nonparametric federated learning of neural networks. In International Conference on Machine Learning (pp. 7252-7261). PMLR.  ", "summary_of_the_paper": "The paper presents a new training procedure for federated learning (FL) systems based on a daisy-chain network. Training the system has two phases, a daisy-chain phase in which models are transmitted from one client to another via a coordinator node, and the standard aggregation phase in which models are averaged according to FedAvg rule. The main motivation is reduced overfitting and improved generalization compared to the standard FedAvg training, especially with limited-sized local datasets. The paper presents PAC-like ($\\epsilon, \\delta$)-guarantees for their algorithm, and provides a discussion on privacy violation concerns of their algorithm. The method is demonstrated on several datasets and shows improved accuracy over the compared methods.", "summary_of_the_review": "The paper presents a novel procedure for learning federated learning systems. I think that the paper misses important related works and that the empirical evaluation is not sufficient. Therefore, currently, I recommend rejecting the paper. I am willing to reevaluate the paper based on the author's response to my concerns.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "BMmqgbDHal", "reviewer": "Reviewer_hTem", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n+ To raise this real-world question is a shining point in this paper. To my best knowledge, there is limited work discussing this research question in this field;\n+ They cover the key points in this paper including privacy discussion, communication cost discussion, and comparison with selected baselines.\n+ They cite appropriate related works and the presentation of their work is easy to follow.\n\nWeaknesses:\n- As each local model will be re-distributed to other clients in this mechanism, though they discuss the privacy guarantee, it will raise security concerns of model poisoning and attack propagation in the network;\n- According to the current random distribution mechanism, though authors find a substitution of sharing data by sharing models, the communication cost will increase a lot. The authors did propose the sparse matrix, but considering the whole iterative process in FL, the communication cost has not been discussed fully.\n- The non-iid setting is another point that most research papers would discuss in their work. I am wondering how this approach would perform with this setting. Also, FedAvg and FedProx are both good baselines, but I am a little bit curious about the comparison with other state-of-the-art baselines.", "summary_of_the_paper": "The authors propose a mechanism to solve the problem of small data in federated learning. Rather than directly uploading models to the server, there is an indicator to re-distribute each local model to other local models. They also discuss the conventional privacy guarantee with their approach. Finally, they provide experimental results on multiple datasets. The main contribution is to discuss how to solve the small data research question in FL. This question itself is practical and interesting. Also, I would say the analysis of the privacy guarantee is convincing.", "summary_of_the_review": "The research question itself is an interesting topic to explore. The authors develop an algorithm to solve this question by exchanging model parameters across clients rather than exchanging data. This is good preliminary work in this direction. More work needs to be done in designing a novel coordinator, discussing the communication cost, and explorations on non-iid data settings.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers were not convinced by the authors' responses to their concerns, and this paper generated little followup discussion. Some primary concerns include the privacy analysis, limited technical contribution and scope (e.g., only being applicable to iid data), and lacking comparison to suggested baselines. The authors are suggested to take the reviewer comments into account for further investigation.
Decision: Reject