Learning to aggregate: A parameterized aggregator to debias aggregation for cross-device federated learning
Authors: Tao Shen
Kun Kuang
Yaliang Li
Feng Wang
Zheqi Lv
Hongxia Yang
Chao Wu
Fei Wu
Venue: Submitted to ICLR 2023 (2023)
Keywords: Federated learning

Abstract: Federated learning (FL) emerged as a novel machine learning setting that enables collaboratively training deep models on decentralized private data. Due to the heterogeneity (non-iidness) of the decentralized data, FL methods (e.g. FedAvg) suffers from unstable and slow convergence. Recent works explain the non-iid problem in FL as the client drift, and deal with it by enforcing regularization at local updates. However, these works neglect the heterogeneity among different communication rounds: the data of sampled candidates at different communication rounds are also of non-iid distribution, and we term it as period drift, which as well as client drift can lead to aggregation bias that degrade convergence. To deal with it, we propose a novel aggregation strategy, named FedPA, that uses a Parameterized Aggregator, as an alternative of averaging. We frame FedPA within a meta-learning setting, and formulates the aggregator as a meta-learner, to learn to aggregate the model parameters of clients. FedPA can directly learn the aggregation bias and well calibrate and control the direction of aggregated parameters to a better direction towards the optimum. Experiments show that FedPA can achieve competitive performances compared with conventional baselines.

Under review as a conference paper at ICLR 2023
LEARNING TO AGGREGATE: A PARAMETERIZED AG- GREGATOR TO DEBIAS MODEL AGGREGATION FOR CROSS-DEVICE FEDERATED LEARNING
Anonymous authors Paper under double-bline review
ABSTRACT
Federated learning (FL) collaboratively trains deep models on decentralized clients with privacy constraint. The aggregation of client parameters within a com- munication round suffers from the "client drift" due to the heterogeneity of client data distributions, resulting in unstable and slow convergence. Recent works typ- ically impose regularization based on the client parameters to reduce the local aggregation heterogeneity for optimization. However, argue that they gener- ally neglect the inter-communication heterogeneity of data distributions ("period drift"), leading to deviations of intra-communication optimization from the global objective. In this work, we aim to calibrate the local aggregation under "client drift" and simultaneously approach the global objective under "period drift". To achieve this goal, we propose a learning-bases aggregation strategy, named FEDPA, that employs Parameterized Aggregator rather than non-adaptive tech- niques (e.g., federated average). We frame FEDPA within a meta-learning setting, where aggregator serves the meta-learner and the meta-task is aggregate the client parameters to generalize well on a proxy dataset. Intuitively, the meta- learner is task-specifi and could thereby acquire the meta- knowleeno Le., cali- brating the parameter aggregation from global view and approaching the global optimum for generalization.
INTRODUCTION
Federated Learning (FL) McMahan al. (2017) has been an emerging learning paradigm to collaboratively train a shared model on a decentralized manner without sharing private data. In FI., clients independently train the shared model over their private data, and the server aggregates the uploaded model parameters periodically until convergence. In FL Kairouz cct al. (202I), a key challenge hindering effective model aggregation lies in the heterogeneous data of clients Zhao al. (2018), especially in cross-device (as opposed to cross-silo) FL with large amount of clients (e.g. mobile devices). Wherein, vanilla FL algorithms, such as federated averaging (FEDAvg) McMahan al. (2017), based on averaging the parameters of candidate clients, would suffer from bad convergence and performance degradation.
machine
Existing works Hsu et al. (2019): Li al. (2020); Karimireddy et al. (2021) depict the non-iid trap as weight divergence Zhhao et al. (2018) or client drift Karimireddy et al. (00TT. To cope with it, they typically impose regularization in local optimization at each communication round such that the intra-round heterogeneity can be reduced. However, we argue that existing methods generally neglect the heterogeneity among different communication rounds, and the oound-ppeifiic regulariza- tion would inevitably fall into a local optimum. Specifically, in cross-device FL, the sampled clients to aggregated might involve different data distributions among different communication rounds. As such, the optimization direction estimated in single round might deviate from that estimated with all clients, eventually amplifying the the aggregation bias] and resulting bad convergence oscillation. For simplicity, we term this challenge as "period drift", and provide empirical evidence in real-wolrd datasets (c.f. Figure
In ecological studies, aggregation bias is the expected difference between effects for the group and effects for the individual.

---

Under review as a conference paper at ICLR 2023
Potet
1206
oecilatinng
!1!!
Local updalee
initill
Fec/ig
FedPA
W14
Calibration
Trajectory of Fes/Ag
Wp++1
Trajectory of FedFA,
(a) period drift
(b) trajectory of FL
Figure 1: Period drift in FL. In the left figure, we give a example of a 10-class classification task with label distribution skew Kairouz et al. (202T). We consider three degrees of non-iidness by setting the dirichlet hyperparameter = 170.1/0.01 Hsu et al. (2019), and display the distribution difference of five communication rounds within a
gri. The colored blocks the histogram represent the amount of data of different labels that belongs to the selected 10 clients of 100 clients. In a subfigure (i.e. within communication round), client drift exhibited by different colors, while in a column, the period drift can be presented by the length of bars. Period drift becomes more obvious as increasing the degree of non-iid (smaller a). In the right figure, we illustrates the trajectory of FL. where the direction with period drift may deviates from the global optimum, resulting in slow convergence and oscillation, and how FEDPA calibrates and controls the trajectory of FL.
In this work, we set the goal of debiasing model aggregation under client drift and period drift in unified framework. The key to this problem is to have adaptive approximation of parameter calibration towards the global objective f McMahan et al. (2077), which is, however, non-trivial to go beyond the local view and approach the optimum based on solely the intra-communication client parameters (c.f. Figure []. To bridge the gap, we introduce a learning- based framework, where a parameterized aggregator takes the cntruncommuiiiatoonn client parameters into considera- tion, and learns to calibrate the direction of aggregated parameters. Technically, we propose a novel aggregation strategy, named FEDPA, which frames the learning-ter -t-aggrreate procedure learning setting Ravi & Larochelle (2016): Andrychowicz et al. (2016). In particular, the aggregator considered meta-learner that is learning to aggregate the parameters of clients into a proxy model that could generalize well on proxy dataset. The aggregation process at each communica- tion round refers one meta-task. The meta-knowledge refers how capture the global view under the client/period drift, alleviate the aggregation bias, and calibrate the aggregated parameters towards the optimum.
meta-
2 RELATED WORK
Federated learning with non-iid data Federated Learning with non-iid Data The performance of federated learning often suffers from the heterogeneous data located over multiple clients. (Zhao jt al. 2018) demonstrates that the accuracy of federated learning reduces significantly when models are trained with highly skewed non-iid data, which is explained by weight divergence. (Li et al.) 2020) proposes FEDPROX that utilizes proximal term to deal with heterogeneity. (Li al. 202 lb) provides comprehensive data partitioning strategies cover the typical non-iid data scenarios. Fed- Nova (Wang et al.) 2020) puts insight on the number of epochs in local updates and proposes a normalized averaging scheme eliminate objective inconsistency. FedBN (Li et al. 2021c) focuses on the feature shift non-iid in FL, and proposes to use local batch normalization to alleviate the feature shift before averaging models.
2

---

Under review as a conference paper at ICLR 2023
Meta Learning Meta learning is branch of machine learning in which automated learning algo- rithms, whose major objective is to use such information to understand how automated learning may become flexible in handling learning problem, thus improving the performance of existing learning algorithms or learning the learning algorithm itself. Meta learning is to solve a problem known as learning to learn (Pramling) 2012), and has shown its effectiveness in reinforcement learning (Xu et al., 2018). few-shot learning (Nichol et al. 20188) image classification (Ravi & Larochelle| (016)) Andrychowicz al. (Andrychowicz et al.) 2016 propose to replace hand-designed update rules with a learned update rule and adopts deep neural networks to train a meta learner by means of an optimizer-optimizee setup, and each component is learnt iteratively using gradient-descent, Also, Ravi (Ravi & Larochelle, (016) proposes an LSTM meta-learner to learn an optimization procedure as a model for few-shot image classification. Finn et al. propose Model-agnostic meta-learning (MAML) method (Finn et al. 2017) that does not impose constraint on the architecture of the learner. Then, derived from MAML, Reptile (Nichol et al. 2018 simplifies the its learning process by conducting first-order gradient updates on the meta-lcarner
Federated Meta Learning Meta learning plays important roles in federated learning from differ- ent aspects, including but not limited to fast adaption, continual learning, personalization, robust- ness, and computing or communication efficiency. Jiang et al. (Jiang et al.] (219)) point out that the setting of MAML, where one optimizes for a fast, gradient-based, few-shot adaptation to a het- erogeneous distribution of tasks, has number of similarities with the objective of personalization for FL, and observe that conventional FEDAVG can be interpreted as a meta learning algorithm. Li (Li al. 2021a) proposes Meta- HHRR that train shared embedding network can generalize to any individual users, achieving robust learning and personalization. Fallah et al. (Fallah et al. (2020) studies a personalized variant of the federated learning, whose goal is find initial shared model that current or new users can easily adapt to their local dataset by performing one or few steps of gradient descent with respect to their own data. Lin (Lin et al., 2020b) designs a novel feder- ated learning framework for rating prediction (RP) for mobile environments, and employ a meta recommender (MR) module to generate private item embeddings and RP model based on the col- laborative vector. We list other methods that adopt meta learning to federated learning. Recently, Shamsian et al. 2021) proposes learning a central hypernetworks that acts on client representation vectors for generating personalized models. (Yao et al. 2019) presents FEDMETA that using proxy dataset for unbiased model aggregation by meta update on server. However, this method updates the global model by directly training proxy dataset. In our experiments, it has risks at overfitting on proxy dataset.
3 METHODOLOGY
The aim of typical federated learning is to learn shared model over decentralized data. In the federated setting, data cannot be collected in central server and should be locally fixed on various devices, to protect data privacy. FEDAVG is typical FL method that aggregates local model updates using a weighted averaging strategy, i.e.,w <- ΣK severe accuracy degradation issue in the non-iid case, i.e. P(a, v) ~ Pe(x, y) + Pj. In this section, we explore the non-iid problem of typical FEDAVG, , and propose a novel framework based on meta learning to deal with it.
w However, FEDAVG suffers from
3.1 TYPICAL FEDERATED LEARNING SETUP
FEDAVG is to learn& single shared model over decentralized data to minimize the global objective
f(w) = - Σ"= fi(w) [(w in the distributed manner. The objective is the sum of the loss function of all private data D, Prrivate. that independently generated by distinct distribution PA (z, y) from K clients. The union of decentralized private data forms the training dataset of FL. To minimize the global objective, FEDAVG starts with copying the global model parameters w E Rd by set of candidate clients, and each candidate then conducts local update that optimizes the local objective by gradient decent method for several epochs:
Fo(wF)) 7Ek SEP
Σ f((w(),
w x ww - F' (wt, D D
(1)
3

---

Under review as a conference paper at ICLR 2023
where Fl((F) is the local objective of the k-tth client, ng is the number of local samples, is the local learning rate, and F& (w() IRd is the gradient vector. After period of local updates, clients transmit local model parameters w the server, who then aggregates these parameters by weighted averaging:
K
global
nkk
wi
(2)
where wiglball is the parameters of the global model. Repeat this training process until the global model gets convergence, and the shared global model are collaboratively trained without sharing private data.
However, the expectation Ep, [w)] + f(w) since the data distribution of the k-th client may be different with that of any j-th client (Ps * P, * #Poone Poveroll), as well as the overall data distribution client drift (or weight divergence). In spite of some works Li et al.
in the non-iid setting, leading 2020); Karimireddy et al. (202)) ) that deal with this, they do not considered the problem of "period drift", that is, the data distribution of randomly selected candidate clients of the f-th communication round are different with that of the f + 1-th communication round, as well as the overall distribution (P* (z, v) # pp+1 (z, v) 4P Poverall). It will also lead to bad convergence.
3.2 PARAMETERIZED FEEDBACK AGGREGATOR
In this section, we introduce the proposed FEDPA and explain how it deals with the aggregation bias (client drift as well as period drift), and implement within a meta learning framework.
Inspired by control theory, we naturally consider federated learning as a dynamic system Haddad & Chellaboina (20TT), where we regard model parameters wy as system states. Actually, the training process of FEDAVG is autonomous system that without control, and its differential equations I 2 can written as follows
Wx++1 g(w,)
Awf).
W
where g(w;) includes the model parameters wi and the local update Auf that minimizes (wf) for several epochs, as well as parameters averaging, who determines the trajectory of model parameters W'- However, due to the non-iid data, the objective Fi((w)) of selected candidates could be an arbitrarily poor approximation
the global objective f(w:), resulting in aggregation bias (client
drift and period drift), Our idea is to control the local update approaching to the optimum by adding control variable u² for each client, intervening the trajectory of FL. We formulate the controlled system ge (we) as follows:
W++1 = (w))
1
K
(4)
(w Aw? mt))
ww Au (1 h(w,, Auf Φ)))
K
K
where we define u 1(.: Φ) as the controller of Awt, parameterized by Φi who takes the model parameters wi and Aut inputs. Now, we package the averaging operator and the controller as the aggregator, and define the aggregation function as aggr(w,, AW, Φ) > E (w, Aw h(w,, Awt d))), where AW, = [Awf) is the set of local updates of 1-th candidates. Finally, the differential equation is formulated as:
W aggr(w,, AW, Φ).
(5)
The question now how can we get an effective aggregator to debias model aggregation? We implement it within a meta learning framework. Inspired by Andrychowicz et al. (2016); Ravi &
In fact, the number of samples of clients n4 is usually unknown server, thus set Bs
4

---

Under review as a conference paper at ICLR 2023
Physical System
Dynamic System
Meta Learning
(f Zquery): y Mquery)
Target
3 Quuery
Learner
Yquery
Aipapprr Whyt
Foooback
Dicarser
Meta Learner
Controller
50%
h thaete-tarnner
Gentrol Signal
Control Var dbbllble
Feadforward Process
2 support
Model Parameters
Ysupport
Figure 2: Relations of FEDPA dynamic system and meta learning. The left figure shows the pipeline of FEDPA, including local updates, model aggregation and the aggregator of training. We explain the process FL as a dynamic process of Wt, and the trajectory of wi should be con- trolled towards global optimum due the aggregation bias (client drift and period drift). Thus, the performance of aggregated model on the proxy dataset is the target in the control loop, and the parameterized aggregator is the controller, shown in the middle figure. To achieve the training of aggregator (controller), we frame it within meta learning setting, where the aggregator meta learner that helps the aggregated model (learner) to achieve good performance the proxy dataset.
Larochelle (016)), clients in FL can be regarded learners whose their privacy data considered as support set. At the meantime, the aggregator is a meta-learner, and we utilize a set of proxy data query set. The Relations of FEDPA. dynamic system and meta learning shown in Figure 3.2
The training of aggregator is implemented by testing the performance of an aggregated model on the proxy dataset. At each communication round, the server receives and aggregates clients' parameters into a proxy model, and then evaluates the performance of proxy model on the proxy dataset. We assume that the better performance proxy model has, the better aggregator becomes. Thus, can optimize the aggregator by the objective as follows:
min of [Wprrzz D Pprozy),
where Wprozy aggr(ws AW.
For the model parameters w with m layers w [wi--m). the parameters of aggregator con- sist of two sets of input dense layers and one set of output layers ó { Q1--1 @Awiin oiur }. Considering the dimension explosion problem (o Rddd if E Ed)) we design the network with the bottleneck architecture, mapping the parameters to low-dimension space, and finally re- store the output to the original dimension. For example, we first respectively input (wi..m) and [Aw]..m fo (with dimensions d(...m into the dense layers (ør:in J and (0Awain J. and output two low-dimension hidden states hy.... A (with dimensions Pi..m = logg(dd...m) + 1). We then concatenate {hr..... &a An inal dimension, becoming the control variable uf:
} and finally input it into output dense layers {øgut } to restore the orig-
u h( wf)
dense(concal (dense (eiesssew--... (i & p...mm ) dense( CeonnseAuue 1.mm..mm @ Awain poutt
and note that all these operators are layer-wisely for each local model w.
The whole process shows in algorithm I At the training stage, just like the usual FL, 1) the server randomly samples set of candidates K, and starts clients local update and upload their updated
5

---

Under review as a conference paper at ICLR 2023
Algorithm FEDPA require the global model the proxy dataset D Pprozy server, the clients indexed by k and their local model w and private dataset DD private: the local learning rate 'tt the number of local epochs Fa and the number of epochs for training aggregator Fig. and the total number of Tr Server executes:
1: initialize the global model wp
2: for each round = 0, 2. ...... do
3: randomly sample set of candidate clients K.
4: at the meantime:
5: a) AW; ClientsUpdatt((w;, K.),
6: b) update the aggregator by equation 6 for Ey epochs}
7:
W++11 aggr(ww, AW,
8: end for
ClientsUpdate:
1: for each client k: E K. in parallel do
2: download: w <- Wt
3: for each epoch 1, 2, ..... Eg do
4:
Conduct local update: w s- w - V Fe (wt, D Dorivate) 5: end for
6: upload: AwF +- w² wi
7: end for
8: AW, = (Awt}
model parameters AW; back server; 2) the server aggregates all these parameters into global model by equation[5] and start next generation of 1); 2) at the meantime (during clients local update), the server trains the aggregator by equation 6 3) repeat these steps until FL stops. By learning the Φ, the aggregator can capture the ability calibrating and controlling the aggregated parameters in global view to better direction towards optimum, that solves aggregation bias (client drift and period drift).
4 EXPERIMENTS
4.1 SETUP
Datasets and models. We evaluate FEDPA with different state-of-the-art FL methods both CV and recommendation dataset. For CV dataset, we use FEMNIST Caldas et al. (2018), consisting of 671,585 training examples and 77,483 test samples of 62 different classes including 10 digits, 26 lowercase and 26 uppercase images with 28x28 pixels, handwritten by 3400 users. It is an image classification task, and
the lightweight model LeNet5 LeCun al. (1998). For recommen- dation dataset, we use MovieLens 1M Harper & Konstan (2013) (including 1,000,209 ratings by unidentifiable 6,040 users on 3,706 movies. is click-through rate (CTR) task, and we use the popular DIN Zhou et al. (2018) model. For performance evaluation, we follow a widely used leave- one-out protocol Muhammad et al.) (2020). For each user, we hold out their latest interaction as testset and use the remaining data as trainset, and binarize the user feedback where all ratings converted to I, and negative instances are sampled 4:1 for training and 99:1 for test times the amount of positive ones.
Federated learning settings. Note that the datasets we (both FEMNIST and MovieLens 1M) have "natural" non-iid distribution since we can splite the dataset by userid, i.e. image data are handwritten by different users and movies
rated by different users. Beside, distribution [Hsu et al. (2019) to simulate the label distribution skew setting for FEMNIST, where the hyperparameter a controls the degree of non-iidness. The smaller a, the more degree of non-iid distribution. For FL training, we set totally T = 100 communication rounds, and sample 10% of all
the Dirichlet
hhtps://gituu..co//llaaaeae http://wwwaaaalalaaa////e//tttta http://ggoonooo..oooyddaeetemmee license)
(BSD-2-Clause license)

---

Under review as a conference paper at ICLR 2023
clients per round and each client trains E; 5 epochs at local update, using Adam optimizer Kingma & Ba (2014) with learning rate 77 = 0.01. In our proposed FEDPA, we use the proxy dataset that randomly sampled 1% from training data for all tasks. As for the training of the aggregator, set Fg = 5 for Moviel and Fg = 30 for FEMNIST, with learning rate flg 0.001.
Baselines. FEDPA is server-side method that improves the model aggregation, thus we compare FEDPA with 1) the vanilla FL method FEDAVG McMahan et al. (2017). ). 2) client-side FL method FEDPROX Li et al. (2020p 3) a server-side FL method without using proxy data FEDAVGM Hsu et al. (2019). 4) side FL method without using proxy data FEDOPT Reddi al. (020), 5) a server-side FL method with proxy data FEDDF Lin et al. (2020a), 6) a server-side federated meta learning method with proxy data FEDMETA Yao et al. (2019). Note that FEDPA is to solve
the slow convergence training, thus we omit other excellent FL method involving meta learning
designed for model initialization or fast adaption Chen et al. (2018) for personalization Shamsian
et al. (021); Fallah et al. (2020).
Evaluation Metrics. For image classification task, model performance our experiments are measured by the widely used top-1 accuracy. For CTR task, model performance are measured by some popular metrics: area under curve (AUC), Hit Ratio (HR) and Normalized Discounted Cumulative Gain (NDCG):
AUC ExeEDY EMED, (11) < 5 (1o)
|D
HitRate@K
1
Σ
(Ru.g S K),
[2]
WEEl
22( SK) -
NDCG@K
Σ
U wEl
loga (1 (Ru.g. S K) +
where L/ is the user set, 1 is the indicator function, Ru.g. is the rank generated by the model for the ground truth item Iu and f is the model to be evaluated and Dr, DF is the positive and negative sample sets in testing data.
Implementation
The experiments are implemented with PyTorch. We simulate the FL environ- ment including clients and run all experiments on deep learning server with NVIDIA Tesla V100.
4.2 ANALYSIS
Visualize the impact of period drift. We conduct this experiments on the FEMNIST dataset with three degrees of non-iidness, that we the dirichlet hyperparameter a //.01/00.1. The top three figures visualize the degree of non-iidness with different a. We select 10 candidates from 3400 clients, and show the label distribution of the 10-digits labels among 62 classes for the 20 earliest communication rounds. From left to right, we show that the size of points becomes more diverse both within the same communication round and between different communication rounds, that exhibits client drift and period drift, respectively. For the bottom three figures, we show the impact of period drift by five curves for each setting. The FL training becomes more difficult as increasing the degree of non-iidness. Especially note the setting of "iid each round". To show the impact of period drift, we manually eliminate the impact of client drift by shuffling the data of selected candidates each communication round, to force the data become iid. The figures show that not only the client drift, period drift will lead to slow convergence and oscillating, and proposed FEDPA can achieve fast and steady convergence especially with smaller a.
The performance on MovieLens. We conduct this experiment on the MovieLens 1M dataset, which naturally non-iid, and evaluate it by auc, hit@5, hit@ 10, ndcg@5, ndeg@) 10. As Table most metrics. Due to the well trained
4.2 shown, our proposed FEDPA outperforms alternatives aggregator, our FEDPA has the fastest convergence than other sota methods. Methods that without proxy dataset, have low convergence because samples of CTR task is kind of strong non-iid since each user have distinct user profiles like userid, age, sex and so on. Besides, each user rates limited movies, which leads to few amount of embedding table in the model being updated. For methods
7

---

Under review as a conference paper at ICLR 2023
2-02
0-31
V- 0.33
37-
5.--
32
31
KC
150
Figure 3: The impact Period drift. This experiments on the FEMNIST dataset with three degrees of non-iidness. For the top figures, we select 10 clients and choose the 10-digits labels. In each scatter plot, the axis is twenty communication rounds and the y axis is ten labels. A specified point is specified client has samples that of specified label specified communication round, whose size means the number of samples and color means the client it belongs to. The size points becomes more diverse within the same communication round between different communication rounds, that exhibits client drift and period drift, respectively. For the bottom figures, we show the impact of period drift by five curves for each setting. As the figures show, not only the client drift, period drift will lead to slow convergence, especially with smaller a:
that use proxy dataset, comparing with FEDPA, FEDDF and FEDMETA have limited performance but have different results and for different reasons. FEDDF has advantage for the CTR task since the model is of a logistic regression that only has one output, thus FEDDF can hardly benefit from ensemble distillation on proxy dataset. FEDPA can achieve higher performance than other methods since it can get into a smaller optimum since FEDPA can deal with the period drift, which has different objectives at different communication round even around global optimum.
The performance on FEMNIST. We conduct this experiment on the FEMNIST dataset, and we four settings to evaluate the performance. As Table 4.2 shown, proposed FEDPA outper- forms alternatives. As increasing the degree of non-iidness, proposed method shows little perfor- mance degradation, comparing other FL method. FEDPA benefits from the well-trained aggre- gator that can calibrate model parameters global view, which resulting in good performance extreme non-iid setting. Compare with other FL methods, FEDPA can achieves fast and steady convergence, and a better optimum.
The advatages of learning-based FEDPA Because of the "no free lunch" (NFL) theorem, it is hard to have a perfect method that appropriate all datasets and scenarios. However, proposed FEDPA provides framework that target specific problem. The ability of the aggregator is from the proxy datasets, which is specific and adaptive to a certain task. The aggregator can learn to aggregate for different model, datasets and even the non-iidness.
The difference using proxy dataset. We compare FEDPA with the baselines that using proxy dataset, FEDDF and FEDMETA The common ground of these three methods are they all use the proxy dataset to achieve fast convergence and better performance. However, the way of using proxy dataset and the reason of how it helps FL are very different. FEDDF uses unlabeled proxy data by leveraging ensemble distillation, taking advantage of ensembling the diverse model parameters of clients as well distilling the logits to achieve consensus. FEDMETA uses proxy dataset by meta
8

---

Under review as a conference paper at ICLR 2023
updating that training global model each communication round. However, the problem of these two method is that they have the risks of overfitting on the proxy dataset, since they directly update carefully tune the hyperparameter of training epochs and regularization.
the model parameters, Instead, FEDPA does not directly update model parameters but learns aggregative bias to control the training process of FL.. Thus, FEDPA is more safe than the methods that directly updating model parameters.
Table I: The performance on MovieLens 1M
Auc
Hit@5
Hit@ 10
N
Ndeg@ 10
FEDAVG
0.7482
0.2916
0.4290
0.1901
0.2346
FEDAVGM
0.7482
0.2916
0,4290
0.1901
0.2346
FEDPROX
0.7459
0.2924
0.4298
0.1914
0.2358
FEDOPT
0.7250
0.2967
0.4419
0.1904
0.2374
FEDDF
0.7053
0.2553
0.3623
0.1701
0.2046
FEDMETA
0.7651
0.2930
0.4429
0.1919
0.2404
FEDPA
0.7878
0.3058
0.4382
0.2002
0.2431
Table 2: The performance on FEMNIST
Natural
=
0.1
a= 0.01
FEDAVG
0.6909
0.7299
0.7029
0.5427
FEDAVGM
0.6909
0.7299
0.7029
0.5427
FEDPROX
0.6990
0.7360
0.7177
0.5478
FEDOPT
0.6830
0.7295
0.7156
0.5157
FEDDF
0.6921
0.7311
0.6955
0.5271
FEDMETA
0.6967
0.7316
0.7194
0.5617
FEDPA
0.7444
0.7431
0.7261
0.7224
5 PRIVACY CONCERNS
As for the proxy dataset, understand this is denounced in some situations, since it may violate the constraint privacy because there are some scenarios that have no proxy dataset like healthcare. However, in some scenarios that allows proxy dataset like natural images recognition, item inter- actions with personal information wiped, proxy dataset can help FL to large extent Li & Wang 2019); Lin al. (2020a); Zhang et al. (202T); Yao al. (2019) etc. Instead, we encourage the use of proxy dataset in FL (if exists), since many companies like Google, Facebook remains previ- ous data the turning point of legislation for privacy, and how to use the proxy dataset is also an interesting problem.
CONCLUSION
In this work, we provide an another factor to degrade the convergence of cross-device FL., namely, the period drift. To solve the period drift as well as the client drift, we propose novel aggregation strategy, FEDPA, as an alternative of averaging. We analyze the problem view of dynamic sys- tem and frame the training procedure of FEDPA within meta-learning setting. Experiments show methods, indicating that the trained ag-
that the proposed method outperforms other gregator can well capture the ability of aggregating drifted clients' model parameters. It also has global view to alleviate the period drift, by adding parameter-we wise bias for each client to calibrate and control the aggregated parameters better direction towards the optimum.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W. Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando de Freitas. Learning to learn by gradient descent by gradient descent. arXiv: 1606. 04474 [es], 2016.
Scbastian Caldas, Sai Mcher Karthik Duddu, Peter Wu, Tian Li, Jakub Konečný, H Brendan McMa- han, Virginia Smith, and Ameet Talwalkar. Leaf: A benchmark for federated settings. arXiv preprint arXiv:1812.01097, 2018.
Fei Chen, Zhenhua Dong, Zhenguo Li, and Xiuqiang He. Federated Meta-Learning for Recommen- dation. CoRR, abs/ 802.0880..7876 2018.
Alireza Fallah, Aryan Mokhtari, and Asuman E. Ozdaglar. Personalized Federated Learning with Theoretical Guarantees: A Model-Agnostic Meta- Meta-Learning Approach. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and HHsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Anmual Conference Neural Information Pro- cessing Systems 2020, NeurlPS 2020, December 6-12, 2020, Virtual, 2020.
Advances
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-Agnostic Meta-Learning for Fast Adapta- tion of Deep Networks. urXiv:1703.03-40 [cs]. 2017.
Wassim M Haddad and VijaySekhar Chellaboina. Nonlinear dynamical systems and control. In Nonlinear Dynamical Systems and Control. Princeton university press, 2011.
Maxwell Harper and Joseph A Konstan. The movielens datasets: History and context. Acm transactions on interactive intelligent systems (tiis) 5(4):1-19, 2015.
Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effeets of non-identical data distribution for federated visual classification. arXiv preprint arXiv:/90..06335, 2019.
Yihan Jiang, Jakub Konečny, Keith Rush, and Sreeram Kannan. Improving Federated Learning Personalization via Model Agnostic Meta Learning. CoRR, abs/1909, 12488, 2019.
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Ad- vances and open problems federated learning. Foundations and Trends@ in Machine Learning. 14(1-2):1-210, 2021.
Sai Praneeth Karimireddy, Satyen Kale, Mchryar Mohri, Sashank J. Reddi, Sebastian U. Stich, and Ananda Theertha Suresh. SCAFFOLD: Stochastic Controlled Averaging for Federated Learning. 17X:7:1910.06378 [es, math, stat], 2021.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 141.::1412.69800 2014.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-2324 1998.
Chenglin Li, Niu, Bei Jiang, Xiao Zuo, and Jianming Yang. Meta-HAR: Federated Rep- resentation Learning for Human Activity Recognition. In Jure Leskovec, Marko Grobelnik, Mare Najork. Jie Tang. and Leila Zia (eds.), WWW '21: The Web Conference 2021. Virtual Event / Ljubljana, Slovenia. April 19-23, 2021. pp. 912-922 ACM / IW3C2, 2021a. doi:
10.114553442381.345500
Daliang Li and Junpu Wang. Fedmd: Heterogenous federated learning via model distillation. arXiv preprint arXiv:/9/0.03: 2019.
Qinbin Li. Yiqun Diao, Quan Chen, and Bingsheng He. Federated learning on non-iid data silos:
An experimental study. arXiv preprint arX1:22102.02079, 2021b.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated Optimization in Heterogeneous Networks. arXiv:/8/2.06/27 [cs, stat), 2020.
10

---

Under review as a conference paper at ICLR 2023
Xiaoxiao Li, Meirui Jiang, Xiaofei Zhang. Michael Kamp, and Qi Dou. Fedbn: Federated learning on non iid features via local batch normalization. arXiv preprint arXiv:2102.07623, 2021c.
Tao Lin, Lingjing Kong, Sebastian U Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning. Advances in Neural Information Processing Systems, 33:2351-2366 2020a.
Yujie Lin, Pengjie Ren, Zhumin Chen, Zhaochun Ren, Dongxiao Yu, Jun Ma, Maarten de Rijke, and Xiuzhen Cheng. Meta Matrix Factorization for Federated Rating Predictions. In Jimmy Huang, Yi Chang, Xueqi Cheng, Jaap Kamps, Vanessa Murdock, Ji-Rong Wen, and Yiqun Liu (eds.), Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2020, Virtual Event, China, July 25-30, 2020, pp. 981-990. ACM, 2020b. doi: 10.14453397271.3300081..
H. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agtiera y Arcas. Communication-Effiiinn Learning of Deep Networks from Decentralized Data. arXiv:1602.05629 [cs], 2017.
Khalil Muhammad, Qinqin Wang, Diarmuid O'Reilly-Morgan, Elias Tragos, Barry Smyth, Neil Hurley, James Geraci, and Aonghus Lawlor. Fedfast: Going beyond average for faster training of federated recommender systems. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery Data Mining, pp. 1242, 2020.
Alex Nichol, Joshua Achiam, and John Schulman. On First-Order Meta-Learning Algorithms. UrXiv:1803.000 [cs], 2018.
Ingrid Pramling. Learning to learn: study of Swedish preschool children. Springer Science & Business Media, 2012.
Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. In 5th Interna- tional Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2016.
Sashank Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečný, Sanjiv Kumar, and Brendan McMahan. Adaptive federated optimization. arXiv preprint arXiv:2003.00295 2020.
Aviv Shamsian, Aviv Navon, Ethan Fetaya, and Gal Chechik. Personalized federated learning using hypemeetorks. In International Conference Machine Learning, pp. 9489-9500 PMLR, 2021.
Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H. Vincent Poor. Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization. arXiv:2007.07481 [cs, stat), 2020.
Zhongwen Xu, Hado van Hasselt, and David Silver. Meta-Gradient Reinforcement Learning. 1805.0800.00980 [cs, stat), 2018.
Xin Yao, Tianchi Huang, Rui-Xiao Zhang, Ruiyu Li, and Lifeng Sun. Federated Learning with Unbiased Gradient Aggregation and Controllable Meta Updating. CORR, abs/1910.08234, 2019.
Jie Zhang, Song Guo, Xiaosong Ma, Haozhao Wang. Wenchao Xu, and Feijie Wu. Parameterized knowledge transfer for personalized federated learning. Advances in Neural Information Process- ing Systems, 34, 2021.
Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Federated Learning with Non-IID Data. arXiv: 00582 [cs, stat], 2018.
Guorui Zhou, Xiaoqiang Zhu, Chenru Song, Ying Fan, Han Zhu, Xiao Ma, Yanghui Yan, Junqi Jin, Han Li, and Kun Gai. Deep interest network for tlick-through rate prediction. In P Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pp- 1059-1066 2018
11


Reviews:
[{"review_id": "qCsXZAbPffE", "reviewer": "Reviewer_5zVx", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "1. The paper tackles the problem of client drift (locally) in federated learning (FL) due to heterogeneous client data distributions. Besides, they target the period drift (globally), which is the inter-communication heterogeneity of data distributions.\n2. They propose a learning based parameterized aggregator called FEDPA, debiasing model aggregation under client drift and period drift in a unified framework. Their key approach is to learn an adaptive calibration parameter to approximate the global objective.\n3. The input of the framework includes the intra-communication client parameters.\n", "summary_of_the_review": "The introduction and related work part is in good shape and I enjoyed reading it. But the quality for writing could be improved. (See the above session for more details)", "strength_and_weaknesses": "Strength\n1. The 2 levels of drift they tackle is indeed important for FL.\n2. They can achieve better training performance on the MovieLens dataset and FEMNIST dataset.\nWeaknesses\n1. Some small typos, grammar issues, inconsistency issues as stated below.\n2. The key idea should be agnostic to the dataset and types of tasks. It would be good to show the performance on other LEAF datasets.\n3. It would be good if the overhead on the server can be quantified. If the method can not be applied at scale (equation 7 seems to be iterative but more clarification would be good), it is not a perfect match for FL.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "fwKl_Hf83b7", "reviewer": "Reviewer_o7iB", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a method named FedPA to learn aggregators in federated learning. When aggregating model updates from clients, instead of uniform or weighted by number of examples as in the popular FedAvg, FedPA will feed both the global model and the client model updates to a neural network before “aggregating”/averaging. The aggregator is trained on the server with a proxy dataset. Experiments on EMNIST and MovieLens show the advantage of FedPA. \n", "summary_of_the_review": "The idea is interesting, but the draft itself needs improvement. Ablation study and experimental performance are my main concerns. ", "strength_and_weaknesses": "The general idea of “meta learning” and “learning to aggregate” makes sense. \n\nHowever, as the authors commented, though interesting, having a proxy dataset is a strong assumption in problem setting.  \n\nIn addition, the server seems to have access to the model updates of each individual client, which makes it hard to be consistent with FL privacy principles, and other privacy techniques like SecAgg and differential privacy [Federated Learning and Privacy https://queue.acm.org/detail.cfm?id=3501293]\n\nMy major concern is that the proposed method seems to be ad-hoc. It is hard for me to connect the motivation of “period shift” to the proposed FedPA method. Instead of learning neural networks for aggregation, I am wondering if there are easier approaches to use the proxy data. For example, we can simply “learn” a scalar weight to do weighted aggregating/averaging of client model updates. I would strongly suggest some more ablation studies of the proposed FedPA method.\n\nA second major concern is the experiment performance. The accuracy on FEMNIST seems to be lower than expected. For example, [Adaptive Federated Optimization https://arxiv.org/abs/2003.00295] reports >80% for natural user non-IID. \n\nI would also appreciate some more comments on hyperparameter tuning. For example, how are 100 communication rounds, 5 epochs, learning rate \\eta_l=0.01 chosen? How are training epochs (5 for MovieLens, 30 for FEMNIST) and learning reate \\eta_g chosen?\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "TU68d2sMrk", "reviewer": "Reviewer_L63q", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a method called FedPA that deals with client and period drift problems. The period drift problem is caused by the asynchronized updates of each client, leading to extra bias in model aggregation. The authors proposed a learning-based aggregation strategy, that parameterizes the aggregation function using neural network models. The models are trained under a meta-learning framework, which treats the global model as a meta-learner and each client as a specific task. Experimental results have shown that FedPA can account for the additional bias induced by both client and period drift and therefore demonstrate superior performance over other FL baselines in various tasks.", "summary_of_the_review": "This work can benefit from adding more in-depth discussion on the unique advantages of the proposed method and further polishing the writing. I am leaning toward rejecting this paper at this time.", "strength_and_weaknesses": "Strength:\n1. Drifts in FL arise in time and space, while most existing works only address the heterogeneity of client data distributions. This paper has discovered this practically important problem and proposed the notion of period drift that can facilitate further research.\n2. The authors have conducted comprehensive experiments in different settings. Results have shown that FedPA has a superior advantage over baselines in different categories.\n\nWeakness:\n1. This paper lacks an in-depth discussion on why meta-learning frameworks are particularly suited for the period drift problem. It seems like both client and period drift influence the model performance by introducing extra bias in model aggregation. In that case, why not use a regularization-based approach incorporated with a temporal dimension? Moreover, it seems like this paper [1] have studied a similar problem, the authors could consider comparing FedPA with their work as an additional baseline.\n2. The dynamic system analogy seems useless in section 3. The authors are not using significant knowledge from this area. I would recommend adding more discussions or simply removing this part to avoid confusion.\n3. From my understanding, FedPA accounts for additional bias via controlling $\\Delta w_t^k$ through $u_t^k$, then why do we need two separate neural networks for both $w$ and $\\Delta w$? The authors need to be more specific on the choice of NN architectures.\n\nMinor:\n1. Please add a discussion on FedDL in section 2.\n2. Please move the definition of $n_k$ and $n$ to the beginning of section 3.\n\n[1] Jothimurugesan et al., Federated Learning under Distributed Concept Drift, 2022", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ILzsJyUYfN", "reviewer": "Reviewer_7sGj", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a learnable aggregation scheme in the context of federated learning. The paper achieves this using meta-learning to generalize the parameters of the aggregator with a proxy dataset. The paper identifies 'period drift' in the current federated learning setup and presents the meta-learning-based aggregator as a way to overcome this issue. The paper follows up with experimental results showing increased accuracy for different methods and heterogeneity rates across two datasets.", "summary_of_the_review": "The paper proposes a meta-learning-based aggregation scheme. However, it does not show enough theoretical or experimental justification to highlight the effectiveness of the algorithm. Additionally, the paper lacks enough ablation studies on the different aspects of the algorithm like the data distribution of proxy data, the influence of the size of the aggregator model, etc.  Furthermore, the paper's concept of 'period drift' is not well defined despite being a key motivation of the algorithm.", "strength_and_weaknesses": "Strengths\n1. The paper identifies a possible source of client drift\n2. The paper proposes a novel aggregation scheme.\n\nWeaknesses\n1. The paper does not do enough to discriminate between regular client drift and the so called period drift either theoretically or through experiments.\n2. The aggregation strategy uses a proxy dataset which limits use cases. Also, it is very similar to other knowledge distillation-based techniques like FedET[1] and DS-FL[2]. A comparison of performance with these methods should be shown to justify its usefulness.\n3. There is no ablation study showing the effect of the data distribution in the proxy data on model performance.\n4. The experimental settings are not strong. The datasets and models are too simple. I suggest including results on CIFAR-100 and Stack Overflow datasets.\n\n\n[1] Cho, Y. J., Manoel, A., Joshi, G., Sim, R., & Dimitriadis, D. (2022). Heterogeneous Ensemble Knowledge Transfer for Training Large Models in Federated Learning. arXiv preprint arXiv:2204.12703.\n\n[2] Itahara, S., Nishio, T., Koda, Y., Morikura, M., & Yamamoto, K. (2020). Distillation-based semi-supervised federated learning for communication-efficient collaborative training with non-iid private data. arXiv preprint arXiv:2008.06180.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject