Few-Round Learning for Federated Learning
Authors: Younghyun Park
Dong-Jun Han
Do-Yeon Kim
Jun Seo
Jaekyun Moon
Venue: NeurIPS 2021 Poster (2021)
Keywords: Federated Learning

Abstract: In federated learning (FL), a number of distributed clients targeting the same task collaborate to train a single global model without sharing their data. The learning process typically starts from a randomly initialized or some pretrained model. In this paper, we aim at designing an initial model based on which an arbitrary group of clients can obtain a global model for its own purpose, within only a few rounds of FL. The key challenge here is that the downstream tasks for which the pretrained model will be used are generally unknown when the initial model is prepared. Our idea is to take a meta-learning approach to construct the initial model so that any group with a possibly unseen task can obtain a high-accuracy global model within only R rounds of FL. Our meta-learning itself could be done via federated learning among willing participants and is based on an episodic arrangement to mimic the R rounds of FL followed by inference in each episode. Extensive experimental results show that our method generalizes well for arbitrary groups of clients and provides large performance improvements given the same overall communication/computation resources, compared to other baselines relying on known pretraining methods.

Few-Round Learning for Federated Learning
Younghyun Park dnffkf3690kaist.acc
Dong-Jun Han" djhan938kaist.co ac. kr
Yeon Kim
dy .iikkkiist.aacc
Jun Seo
Jaekyun Moon
jwws0630@kaist.ac.
jmoonOkaist. edu
School of Electrical Engineering. Korea Advanced Institute of Science and Technology (KAIST)
Abstract
In federated learning (FL), number of distributed clients targeting the same task collaborate to train a single global model without sharing their data. The learning process typically starts from randomly initialized some pretrained model. In this paper, we aim at designing an initial model based on which an arbitrary group of clients can obtain global model for its own purpose, within only a few rounds of FL. The key challenge here is that the downstream tasks for which the pretrained model will be used are generally unknown when the initial model is prepared. Our idea is to take a meta-learning approach to construct the initial model so that any group with possibly unseen task can obtain high-accuracy global model within only R. rounds of FL. Our meta-learning itself could be done via federated learning among willing participants and based on an episodic arrangement mimic the R rounds of FL. followed by inference in each episode. Extensive experimental results show that our method generalizes well for arbitrary groups of clients and provides large performance improvements given the same overall communication/commuaali resources, compared to other baselines relying on known pretraining methods.
Introduction
Today, valuable data are being collected increasingly at distributed edge nodes such mobile phones. wearable client devices and smart vehicles/drones. Directly sending these local data to the central server for model training raises significant privacy concerns. address this issue, an emerging trend
known as federated learning (FL) [33 9 I1 20 15], where server uploading of local data is necessary, has been actively researched. FL, large group distributed clients interested in solving the same task (e.g., classification on given categories of images) collaborate in training single global model without sharing their data. While standard supervised learning uses some dataset D to find the model ő that would minimize loss function f(Ø, D), FL in comparison seeks the model Φ that minimizes k using local data Dr. The learning process typically starts from randomly initialized some pretrained model and is carried out through iterative aggregation the local model updates.
averaged version the local losses / [Ø, Dz), computed at each node
1.1 Backgrounds and Main Contributions
Motivation. Unfortunately, FL generally requires large number of communication rounds between the server and the clients for model exchange, to achieve a desired level of performance. This makes
"Equal contribution.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Preparation (Meta-training)
Deployment (Testing)
Inital model
Few staps of local updates followed by meta-update
Fast adaptation within few local updates for client
Clant1
Clinnz
(a) Personalized FL: meta- meta-training geared to few steps of local updates at any client.
Pregaration
Deployment (Testing)
(Meta-training
Metastrained model
R-rounds of fellowed by
fast adaptation within communication rounds for group clients
Group
Group [tank
(b) Proposed few-round FL: meta-training geared few rounds of FL. at any group.
Figure 1: Basic concepts of personalized FL and proposed few-round learning. In the meta-training phase (or preparation stage) of our scheme, the service provider prepares the imitial model with willing participants. Once this preparation is over, the service provider offers this initial model to any group of clients (possibly including the meta-tree training participants) who hope to perform its own task within few-round FL. In order to facilitate R-round FI. at deployment, take episodic training strategy that mimics actual deployment preceded hy R FL. rounds in the meta-training phase with set of participants chosen in each episode.
the implementation of FL significant challenge in bandwidth-limited or time-sensitive applications. Especially in real-time applications (e.g., connected vehicles or drones) where model should quickly adapt to dynamically evolving environments, the requirement many communication rounds becomes a major bottleneck.
Goal and challenge. To tackle this problem from the service provider's perspective, we aim to prepare an initial model that can quickly adapt any group (focusing on its own task) of clients within only few rounds of FL. The key challenge here is that the task of the group conducting FL (i.e., the downstream task for which the prepared model will be used) is generally not known when the service provider prepares the initial model. In the context of classification, different tasks mean classification involving different sets of classes. For example, classifying diseases A, B, (task 1) is a different task compared to classifying diseases D, E, F (task 2). Since the group conducting FL for downstream task can include classes that are unseen during preparation, existing FL. approaches cannot tackle this problem.
Key idea. Our key idea is to adopt meta-learning (which enables reliable prediction even when the task at inference is unseen when the model was meta-trained) to prepare the initial model that enables few-round FL. In other words, we aim to meta-train an initial model for few-round downstream FL. Once meta- trainnta is over, the service provider would offer the trained model to some clients who want to solve common task after collaborating through quick few rounds of FL. These clients may or may not be the participants of the earlier meta-training phase, and their classification task is generally considered unseen during meta-training.
high-level description of our idea depicted in ((b) Given small target value R, we take an episodic training approach to enable R-round FL for any group clients. In essence, we find the initial model ó that would minimize the average of local losses f ((0(()) D&). where (R) is the model to be updated from & through Rt rounds FL among future clients the deployment stage. Despite the high practical significance problem formulation, to the best of our knowledge, this is the first work to propose a meta-learning strategy geared to few-round FL. is also worth mentioning that model preparation is not real-time requirement and often be done when bandwidth demands sparse.
Fig.
this
Comparison with personalized FL. stress that our idea has a different purpose and approach relative to the recent line of works on federated meta-learning [12 4 which initiate model for personalized optimizations local clients (see Fig. ((a)). The goal of these approaches is to obtain personalized local model at each client within few steps of gradient descents, in the deployment

---

stage. To achieve this goal, in the preparation stage, few steps of local updates and meta-update are first performed at each participant independently (with its own local data), and FL (or aggregation) is adopted just to take advantage of data of various participants: these approaches seek & that minimizes the average of local losses (0), Px), where 0g (+) is the local model updated from ø through a number of gradient steps using local data DE- In contrast to personalized FL. that focuses on local client models in the deployment stage, our few-round learning inherits the ability of FL at deployment to obtain global model. Hence, for our scheme, it is natural to adopt FL in the preparation stage to mimic the R--ound FL scenario at deployment; in the preparation stage, meta-update is performed at each participant after the collaborative R FL rounds. To sum, our approach aims to prepare initial model that leads to global model within a "few rounds of FL", while personalized FL aims for an initial model leading personalized models within "few steps of local updates" based only on the local data. These are obviously two completely different problems with distinct solutions.
Main contributions. Technically, we utilize a model-agnostic meta-learning (MAML) approach to prepare the initial model via an episodie training strategy. While directly applying MAML independently to each local model leads to existing solutions personalized FL [12. 4). in our approach. It rounds of local updates and aggregations are first performed in each episode before the meta-update process. This unique episode construction compared to personalized FL methods mimics the deployment stage where actual inference is preceded by an R-round FL procedure. Another key ingredient in our solution is to adopt prototype aggregation in each FL round to construct global prototypes that serve as better class representatives compared to the locally computed prototypes, in learning embedding space. This strategy is especially effective when non-IlD (independent, identically distributed) data distribution across clients tends to induce a significantly biased model after performing local updates. The global prototypes serve prior knowledge, a form of regularization, and prevent local models from overfitting to the local data. Moreover, the global prototypes (reflecting all classes across clients) can assist the local models learn more general embedding space. We call this approach a global prototype-assisted learning (GPAL) strategy. Our main contributions are summarized as follows:
We formulate a new problem of high practical significance, namely, few-round learning, where the goal is to prepare an initial model that can quickly adapt any group of clients within only a few rounds of FL.
We propose a meta-training algorithm specifically geared to R rounds of FL followed by inference, to be performed by group of clients on a possibly unseen task.
We guarantee convergence of our meta-training algorithm via theoretical analysis.
We show via experiments that our scheme outperforms existing pretraining approaches including fine-tuning via FedAvg and personalized FL in both IID and non-III scenarios.
1.2 Related Works
Few-shot learning. Few-shot learning is an instantiation of meta-lcarning. In the context of image classification, few-shot learning typically involves episodic training where each episode of training data is arranged into a few training (support) sample images and validation (query) samples to mimic inference that uses only a few examples [9). Through repetitive exposure to series of varying episodes with different sets of image classes, the model learns to handle new tasks (classification against unseen classes) each time. Two hidely-known few-shot learning methods with different philosophical twists, which are also conceptually relevant to the present work, are MAML (5] and Prototypical Networks [1]] MAML attempts to generate an initial model from which different models targeting different tasks can be obtained quickly via just a few gradient updates. The idea is that the initial model is learned via meta-training to develop an internal representation that is close in some sense to a variety of unseen tasks. Prototypical Networks, on the other hand, learn embedding space such that model outputs cluster around class prototypes, the class-specific centroids of the embedder outputs. With episodic training, simple Prototypical Networks are surprisingly effective in learning inductive bias for successful generalization to new tasks.
We stress that our few-round learning scheme (that targets a few global rounds of FL) has different purpose and technical approach compared to the existing works on few-shot learning (that targets few shots of data sample). Nevertheless, we take advantage both concepts on MAML and Prototypical Networks to achieve our own goal: we adopt MAML in updating the initial model specifically geared to R-round Fl., and adopt both prototype aggregation and prototyp-assiited learning strategies to learn general embedding space and successfully handle the non-lID issue in FL.
3

---

Federated meta-learning. Recent research activity has focused on improving model personalization The common goal of these works is to generate an initial model based on which each new client can find its own optimized model via few local gradient steps and using only its own data. In these works, meta-learning employed during federated learning intends to enable each client to handle previously unseen tasks, in the spirit of MAML of [5]. User-specific next-word prediction at individual smartphones, for example, is a possible application. Compared to this line of work, we focus on creating an initial model that leads to a high-accurace global model, rather than personalized models. In this way, we seek to take advantage of a higher variety of data as well as the larger data volume that would be made available through collaborative learning of group of distributed nodes. A clear example is the diagnosis of broader class of diseases that would be possible through collaborative training across more examples contributed by a larger group of individuals. Personalized FL. methods (e.g., 12 especially have disadvantage in non-IlI settings where each client necessarily lacks a sufficient variety of data. The results reported in Section [
via federated meta leta-lea [12 3
One-shot FL. Another line of work recently focused one-shot FI.,, where the goal is to train global model with just one communication round between the server and the clients. The authors of [6] proposed an ensemble method to choose reliable client-specific models from given clients. In the work of (17), local clients send XOR-encoded MNIST image data to the server, and the server decodes train the global model. While the server would need certain data in advance to decode the received results, XOR operation can serve as data augmentation while preserving privacy. In the fusion learning of [8] each local client uploads both the model parameters and the distribution parameters to the server. The server generates artificial data samples from the distribution parameters to train a global model. When the data gets complex, however, is not clear whether conversion into a simple distribution would be reliable. Compared to the existing works on one-shot FL that employ some randomly initialized model, the key difference of our method is the use of meta-learning to prepare an initial model which can adapt to unseen tasks of individual groups' of clients within R rounds of FL. The advantage of our scheme compared to these methods is shown in Section 4
2 Proposed Few-Round Learning Algorithm
2.1 Problem Setup
Federated learning. Let N be the number of clients in the system. FL allows each distributed node k with a dataset D2 to participate in iterative learning of a global model 0 without having to reveal its data to anyone else including the central server. As a given round r starts, each of K participating nodes (generally chosen anew every round) downloads a global model 6" from the server and updates
it using its own local data Dr. The updated local models 05* get all uploaded to the server to be
aggregated to a new model @r+1 Σx= px0 according to the relative dataset sizes μu Σ |Di
[Pg]
The same process gets repeated. FL. generally requires significant number of such global rounds to achieve the desired accuracy, with each round taking up substantial communication resources.
Problem formulation. In preparing an initial model e for any group of clients to pursue few FL rounds, we use meta-learning based on episodic training, where each episode is constructed to mimic R FL rounds followed by inference. Once meta-training is over, in the deployment phase, the service provider offers the trained initial model o to any group of clients wishing to pursue inference on some common task (possibly unseen during meta- veta-training) after collaborating for R rounds of FL.
2.2 Meta-Training (Preparation Stage)
More precisely stated, our meta-training phase is to find é that minimizes the objective function
Σ ((0*(()) Db) K AE
F(6) EA--DCA)
where A₁ is a specific group with K participants drawn from p(A), the distribution over all possible groups, each with K participants; (²()) is the model after R rounds of FL in group A,, starting from di; and Dj. is the local dataset of participant & in group A,. In comparison, the objective function for personalized FL methods (c.g., Per-FedAvg of 44) is F()) v ΣN (()( D&)) where N is the number of clients in the system and (x(( is the model after few gradient steps at client k starting from c. We also reiterate that conventional FL aims at minimizing F(Ø) N
1 ΣN f() Dz).

---

Algorithm Proposed Meta-Training Algorithm for Few-Round Learning
Input:
Initialized model 60 Output: Model o" after T training episodes
1: for each training episode = 0,.,,., T' do
2: The server constructs a group A p(A) of participants chosen of N users.
3: Each participant & e A, splits D3 into support set Sy and query set Qi.
4: 8 6²
5:
for each communication round 0, 1,.,. R- do
6: for each participant in parallel do
7:
Download 0" and [ from the server (download only 0" when 0) for each class e E C2: do [K(c)= [S((e] ExESu(e) 90" (x:)
8:
9:
|/ Local prototype calculation with support set Sy
end for
11:
0f+1 + 0r - aV ,(0',
If Local update of 0 with support set Sy and GPAL
end for
13:
g Ek=1 Ax
// Model aggregation; Aę is relative support set size
14:
= Ek-1 {xTg(c)lc= 1,2, (tsll===12........ I.2.....No] // Prototype aggregation
end for for each participant & in parallel do
16:
17:
Download 0², p from the server.
18:
Compute local prototypes based on Q
19:
0% 8 - VV f (0 .2k) // Local meta-update 6 with query set Qk and GPAL
20:
end for
21:
d-1 K 1202
Aggregation of meta-updated models; μu is relative data size
22:
end for
Before training begins, each client k divides its local dataset into support set Sş and query set Qk- To create training environment matching the actual R-rounds of FL followed by inference at deployment, in each episode of our meta-training phase, we update the model over R federated rounds using the support set and then makes final adjustment (meta-update) using the query set. In other words, the support sets are utilized for learning how to solve the task, by performing R rounds of FL. The query sets are used for evaluating the performance on this task and performing the meta-update process. This overall process is repeated the model is exposed to a series of episodes.
The detailed procedure of our meta-training is given in Algorithm [ For a quick summary, as each episode begins, the server selects a new set K participants. The model Φ², carried over from the last episodic stage, becomes the initial model 0 for the current episode. After R rounds of FL with each round consisting
local updates via local support sets and a global aggregation, 0 evolves to 08. Before moving to the next episode, local meta-updates are done based on 0 using the local query sets to adjust the initial model g in the spirit of MAML. As these meta-updated models get aggregated dt11 at the server, the new episode can begin.
2.2.1 Rounds of Local Updates and Aggregations
In defining the loss function, we utilize the class prototypes and associated Euclidean distance metric of [18), proven method of simple yet effective learning of embedding space. For each communication round r, we not only aggregate the global model 0""1 but also the global prototypes r" (T" (c) 1, 2,2,... .... No) for all classes, where N₂ is the number of classes over all clients. The class prototype is the class-specific averaged feature for data samples and calculated as Line 9 in Algorithm
Model and global prototype download (Line 7). In the beginning of round 2 1, the server has the global model (" and the global prototypes 1 1=-1 (c)c ..,) from the previous round r- 1. Each participant l first downloads 0" and I from the server. Since there no global prototype in the first round, the participants only download the model 8 when r 0.
computed as in Line 9 using the downloaded model 0", the associated embedder outputs go corresponding to the local support samples Sx (c) labeled c. This local prototype serves as a representative of class c calculated based on the local data (support set) of client k.
Local prototype calculation (Line 9). The local prototype (c) for participant
5

---

Loss calculation from local prototypes. Let 1 be the set of all classes of prototypes at participant k: 1% (c)lec Cs}, where Cx is a set of all classes at participant Now using Sk, and T each participant & computes the local loss according to
5k 'local (0, P% (c)) Eecc, [Sx(c)
1
Σ Σ ((go(z)) + log Σ Eepp( (ep(- (z), (d00 CN zcSp(e)
d'sc
based on Euclidean distance d(-) between (c) and gx()) for x Sk(c).
Auxiliary loss from global prototypes. Relying only on the loss function of 2 based on the local prototype tends to bias the model, especially when data distributions across different clients are non-IID. This generally leads to performance degradation of the global model. To get around, we learning (GPAL) strategy, where the global prototypes serve
propose a global prototype-assiste as prior knowledge in form of regularization to prevent local models from overfitting to their local data. Moreover, the global prototypes, reflecting classes not limited to the local dataset, can assist the local model learn a more general embedding space. Given the global prototypes r--1 - (T" = 1.2,.... Nc) and {go (z) E Sk}. the auxiliary loss L (0", (0") can be computed by replacing local prototypes I with global prototypes I ["-1 in (2)
Local update based on GPAL (Line 11). Based on the local loss I (0, computed using local prototypes and the auxiliary loss L (0, 1 based on global prototypes, the objective function becomes
f(6", SR)) = = local (0", + (1 (1-7) "00",,00",T"" (0",
where y is a balancing coefficient. For r = O, we have "(t",Sk)= Lie ( since the global prototype is not defined in the first global round. Line 11 of Algorithm [ accordingly, where is the learning rate. We call this strategy GPAL.
performs local update
In FI., the clients can perform multiple local updates, say F. times. Hence, the process of local prototype computation in Line 9 of Al Algorithm] T loss computation in and local update of Line can be repeated E' times obtain 0%+1
Model and prototype aggregations (Lines 13~14), After performing local updates, each participant k sends its updated local model 0f+1 and the computed local prototypes 1% to the server. Then the server aggregates them according to Lines 13 and 14 in Algorithm where the weighting factor S.p reflects the relative support set sizes.
A
Σ [8y]
The above local update and global aggregation processes are repeated for R global rounds (r 0, 1,.., R 1), and the server obtains 0 and p in a given episode.
2.2.2 One-Round Local Meta-Update and Aggregation (Lines 16-21)
Towards the end of each episode processing stage, the participants download g² and I from the server. Each participant i uses its query set Q& to compute the local prototypes k as in as in Line 9. The query loss J [O" Qx) is calculated similar to (3) based on Q2, 0 PR-1 and r The meta-update would call for taking the derivative of this loss with respect to 8": f Q+)
But one would need the double derivatives from other user locations as well, which highly inconvenient. Ignoring the double derivative terms, we simply replace f Qx) with Fo f(0 Qx). as in Line 19, This is the same as making first-order approximation done in the implementation of MAMI. variants including the original work of [5]. All our reported experimental results as well as convergence analysis in the present paper reflect this choice. The server finally aggregates the meta-updated models from all participants. The next episode begins as the server selects a new set of K participants.
[OR Qx)x
a0 ahet o ((RR, Qu) x [ Σ/< > JØ
a
the MAML-like meta-update, as often
2.3 Testing (Deployment Stage)
In the actual deployment or test phase, given a group of clients, the server sets 60 -T and then leads R rounds of FI. obtain 0 and r fh- Now given test sample, make prediction based on g and p the model output is first computed using g and then comparison is made with the distances from all global prototypes in R-1 to reach decision.

---

Convergence Analysis
We provide theoretical analysis guarantee a certain convergence behavior for our meta-training algorithm for nonconvex loss functions (x(()) := f'Ø, D&). We need the following assumptions commonly made in convergence analyses of FL. involving meta-learning, e.g. 12 4.
Assumption 1. For all i, Si is L-smooth, i.e., [[Vi) ((()a)) L (| - for any Φ1, 62-
Assumption 2. Let (;(d;;) x) be the loss function for a single data point x E D; of participant i. For all - 2.., N. the variance of the loss gradients across data samples at a given participant is Va for any @.
bounded, i.e.
Assumption 3. Let f(Ø)
# Ei f(() be the average local loss of all participants in the system. The variance of the gradient of loss fo across participants is bounded, i.e. 1 N Σ [Vf vf(o)l s
for any ø.
Two key lemmas and theorem below establish the convergence of our method. All proofs are in Supplementary Material.
Lemma 1. Assume that the learning rate or is in the range (0, 1/.] Then, the global loss function F()) in ① Lp-smooth, where LE L2
Lemma 2. Define the local loss of our scheme Fx( := (x())())) at participant k. For a group A with K clients, define the loss averaged within that group FA(o) == t Esea Fo((). Assume e (0, 1/L]. Then, the variance of the gradient of (6) across groups is bounded as
LAI Σ -
& V
(4)
where A is the set of all possible groupings of K participants drawn from a pool of N individuals. Theorem 1. Suppose Assumptions 7 hold and e (0, 1/L.. Let [D] be the mini-batch size at the meta-update processes of all participants. Then, Algorithm guarantees the following upper bound on the loss gradient associated with our learned model o²
T-1 Σ E[LIfFO'DIPE 4(F(0)) - -F)))) 1-0
cB3, R, K)
(5)
BT
where d² is the optimal solution of and (3, DI, K) 3L2 R (V VI- VyK-
As the number of episodes T increases, the upper bound of (5) settles to e. For a given smoothness L, assumed loss gradient variance bounds (Va. Vp)) and targeted number of FL rounds R, the error term e is controlled by the meta-update learning rate 3, the mini-batch size [ and the per-episode number of participants K. For any reasonable value R. practical choices of B, [D] and K can make sufficiently small, as discussed in in Supplementary Material using representative parameter values.
Experiments
We validate our algorithm on CIFAR-100 [10, minilmageNet [19], FEMNIST Following the data splits in [14), for CIFAR-100 and minilmageNet, 100 classes are divided into 64 train, 16 validation and test classes. For FEMNIST, we divide classes into 52 alphabet (upperease, lowercase) and 10 digit classes. For each class of FEMNIST, we sort the images by its name and choose first 600 samples. After all, we have 600 samples for each class every dataset. 52 alphabet classes are set to train classes, while 10 digit classes are set to test classes. The train classes are utilized for the preparation stage, and the test classes are utilized at deployment to model the unseen tasks.
Comparison schemes. First, initialized model is trained for R FL rounds at deployment. The preparation stage is not considered for this scheme. Thus, direct performance comparison would be obviously unfair for FedAvg, but we just want to show what kind of performance improvement is possible by meta-learned initialization versus random initialization. Second, we consider a FedAvg-based fine-tuning, where the model first pretrained by conducting FedAvg in each episode during preparation, and then fine-tuned with new clients for R FL rounds via FedAvg deployment. For example, in minilmageNet, a way classifier model is first pretrained in the preparation stage. Next, the last linear layer is replaced by
simplest baseline, we consider FedAvg [3] where randomly
7

---

Table 1: Performance with only unseen classes at deployment in an IID setup.
Methods
CIFAR-100
miniImageNet
FEMNIST
FedAvg
51,55 = 0.38%
38,80 = 0.26%
74.76 0.35%
Fine-tuning via FedAvg
63.18 + 0.41%
61.58 0.47%
91.95 0.28%
Fine-tuning via one-shot FL 6
64.71 # 0.37%
65.23 0.43%
93.62 0.26%
FRL: Linear classifier (Ours)
67.32 # 0.37%
67.75 = 0.35%
94.86 0.13%
FRL: Distance-based classifier (Ours)
69,74 # 0.31%
68.05 = 0.34%
95.07 0.10%
FRL: Distance-based classifier + GPAL (Ours)
72.93 = 0.32%
69.31 + 0.33%
96.61 0.09%
Xavier-initialized layer, and then the overall model is fine-tuned to the group at deployment. We also consider fine-tuning based on one-shot FL (6) where the local models are sampled and aggregated by ensemble cross-validation. We allow a larger number of available clients (in the deployment stage) for this scheme to accommodate user sampling. The model is first pretrained via FedAvg during preparation, and then fine-tuned based on the scheme of 6 for R rounds at deployment. Finally, although comparison with personalized FL [4] is tricky the goal is different, a global model can still be trained by repeating local updates and aggregations for R FL rounds starting from the initialized model geared to client personalization. Comparison results with this "forced" global model are given in Supplementary Material. For our few-round learning (FRL), we try both a linear classifier and distance- based classifier [8] for comparison. For the linear classifier, connect an additional classifier utilizes prototypes instead of using the linear layer. For the distance-based classifier that utilizes prototypes, we observe the effect of our GPAL strategy. Although we utilize FedAvg for the model aggregation at the server, adopting other aggregation methods that outperform FedAvg can further improve the performance of our method and other basclines.
linear layer behind CNN layers, as in other baselines. The distance-
Preparation stage. We assume N = 64 participants in the system in the preparation stage for CIFAR- 100 and minilmangeNet. We assume N 52 for FEMNIST. For every dataset, following 13. training data samples are prepared into 2.V shards of 300 samples each, such that each shard corresponds one image class. Each participant is given two shards, and these two shards may belong to either a common class or two distinct classes. This models non-IID data distributions across participants. To construct each episode, the server then randomly selects K 10 out of N participants. Each participant uses one half of its local data from each class as support samples, and the remaining half as query samples. We typically set the target number of global rounds R 3. Each episode of our scheme requires 4 global rounds in the meta- veea-training phase: 3 rounds of local updates and aggregation, and I round of local meta-update and aggregation. For a fair comparison, we let all baselines to consume the same amount of communication resources in the preparation stage: up to 40, 000 communication rounds between the server and participants (other than FedAvg that employs no preparation rounds). Hence, our scheme is meta-trained over up to 10, 000 episodes, taking 4 rounds in each episode. We also reiterate that model preparation at the service provider is not a real-time requirement and can be done when bandwidth demands are sparse; this offers an even more favorable performance/complexiy tradeoff options for the proposed scheme.
Deployment stage. At deployment, we distribute the initial model obtained in the preparation stage to new group of clients. To measure the performance, we obtain the average test accuracy with a 95% confidence interval over 1000 different groups (with K 10 clients in each group) after R rounds of FL. For testing, in one case we randomly sample 7 classes from test classes that have not been seen during preparation and distribute across K 10 clients. In the other case, we randomly sample Tu classes from the unseen test classes and r classes from the train classes seen during meta-learning. We consider both IID and non-IllD distributions. In the IID setup, the data samples from each class are equally distributed to K = 10 clients. In the non-IlI setup, we distribute data as in the preparation stage. The support set is utilized for R FL rounds and the server calculates test accuracy with the global model and the gathered query sets of all clients. For the one-shot FL scheme, we allow 20 clients and the server samples K = 10 of them to aggregate. We focus on a 5-way setup (i.e., 7 = 5) in the main paper with the T 10 case reported in Supplementary Material.
Implementation details. The structure of the model follows the setting of [5] and [18), containing 4 consecutive 3 convolutional layers with 64 filters. Successively, each CNN output goes through batch normalization, ReLU, and 2 2 max pooling. In the case of CIFAR- 100 where the size of image 32 x 32, the last two max pooling layers are omitted to up-seale the feature map. We adopt the SGD optimizer with learning rate of 3 = 0.001 for the leaa-ee and a learning rate of 0.0001 for
8

---

Table 2: Performance with only unseen classes deployment in non-IID setup.
Methods
CIFAR-100
miniImageNet
FEMNIST
FedAvg
34.85 + 0.27%
29.74 = 0.22%
59.22 0.18%
Fine-tunie Fine-tuning via FedAvg
44.33 = 0.37%
33.39 = 0.41%
58.23 = 0.65%
Fine-tuning via one-shor FL 6
35.11 = 0.46%
27.16 0,42%
57.88 0.67%
FRL: Linear classifier (Ours)
52.98 = 0.42%
53.51 0.43%
85.14 0.44%
FRL: Distance-based classifier (Ours)
63.85 = 0.43%
61.07 = 0.41%
88.60 0.42%
FRL: Distance-based classifier + GPAL (Ours)
66.87 # 0.40%
63.41 = 0.39%
92.42 0.32%
70
Teet seouracy
60
500
PRI Dicanis 6PN Duan Fias tuging ua Fachun
80
Diearos (0um)
Pessoaabee Deraiio
-0
FRL Daparoo, GPAL (0us)
FRL: Darance 10umt
Pattorlladd FL. Dasarc
ep Falkin
Liear (Dum))
Pesoralired 1ine
Pire tert preahct
3.5
3 Commurication rounds preparation
Communiaztion rounds proparation stago
x10
(a) CIFAR-100
(b) minelmageNNtt
Figure 2:
Final test accuracy at deployment, with varying numbers of communication rounds (proportional to the number of episodes for FRL and to the number of pretraining rounds for fine-tuning in the preparation stage.
the learner. We set the mini-batch size to 60 and the number of local epochs at each client to E = 1. All methods are implemented using Pytorch and trained with single GeForce RTX 2080 Ti.
Results with unseen classes at deployment. Tables [ and 2 show test accuracies averaged over 1000 different groups after R = 3 global rounds at deployment, where the goal of each group is to classify - 5 classes that were unseen during preparation. First, it can be seen that FedAvg yields significantly lower accuracy compared others, as expected, since it uses randomly initialized model for training. By pretraining the model, FedAvg-based fine-tuning gives significant performance gains compared to naive application of FedAvg, underlying the importance of initialization efforts. The fine-tuning scheme based on one-shot FL shows further performance improvements in the IID setup. However, since K 10 clients sampled from 20 clients for this method, there possibly exist some unseen classes when building the global model in the non-IID setup, which lowers the performance compared to distance-basee classifier showing better accuracy compared to the linear classifier. The relative gains of our methods for non-Ill) are partieularly strong. It can be also seen that the performance of the global model can be further improved by our GPAL strategy. Fig. 2 shows how the final test accuracy (after 3 fixed FL rounds at deployment) improves with the number of communication rounds in the preparation stage. The overall results in Tables 2 and Fig. confirm the ad advantage of exploiting meta-learning and global prototype-assisted learning facilitate few-round FL.
FedAvg. Our FRL algorithm performs the best, with the
Results with both unseen/seen classes at deployment. In Table 3 we report test accuracies with both unseen/seen classes deployment; the goal of each group is to classify 5 classes, 2 from the unseen classes and from the seen classes. Since the tasks also handle classes already seen during preparation, the accuracies are generally higher than in Tables [ The trend is consistent with the results in Tables [ and confirming the advantage of the proposed algorithm.
Effect of global we visualized t-SNE of the embedding space at a client in Fig. 3 CIFAR-100 considered with
learning. To understand the effect of our GPAL method further,
Table 3: Performance with both unscen/seen classes at deployment.
CIFAR-100
miniImageNet
Methods
Non-lID
IID
Non-1ID
FedAvg
50.03 4 0.42%
34.82 =: 0.31%
42.17 0.36%
30.37 +: 0.26%
Fine tueen-unnng via FedlAv
66.73 0.36%
44.46 0.36%
63.82 0.49%
36.18 0.42%
Fino-tuning via one-shot FL o
69.84 +: 0.39%
35.33 = 0.46%
67.05 0.44%
29.12 +: 0.43%
FRL: Linear classifier (Ours)
68.22 à 0.38%
53.62 = 0.45%
69.02 0.39%
55.18 0.46%
FRL: Distunce bused classifier (Ours)
70.49 4: 0.36%
65.13 0.43%
70.39 0.38%
62.42 #: 0.43%
FRI.: Distance based classifier GPAL (Qurs)
73.68 + 0.37%
0.44%
0.34%
65.33 + 0.42%

---

(a) Before update
(b) After update without GPAL (c) After update with GPAL
Figure 3: 1-SNE visualization of embedding space at client. The local data samples of the client is illustrated with diamond O, and the data points of all other clients represented by circle O. The global prototypes of each class are shown with t. GPAL. prevents the model from being biased to local data and enables to learn more general embedding space. This leads to performance improvements as seen in Tables 3 and Fig. 2
each client having two classes in its local data in non-IlD setup. used for training as in Fig. ((b) it can be seen that the two classes of the client form clusters without considering the data samples of other clients (but still sell-separated). By considering the global prototypes (reflecting classes of all participants), in Fig. 3(c) the data points in the local client form clusters while staying away from all other global prototypes. clearly desirable feat. This prevents the local model from being biased to its local data and enables the local model to learn more general embedding space compared the case Fig. consiidriig only the local prototypes.
When only local prototypes
Other experimental results. Additional results other settings including higher-way classification, larger group size and mismatched R are reported in Supplementary Material. Comparison with the "foreced" global model based on the personalization scheme is also shown in Supplementary Material.
Conclusion
We proposed meta-learning strategy to prepare initial model geared to few-round federated learning. Given a group of clients with a new task, our meta-trained model generalizes well within only few FL rounds. Convergence of our meta-training guaranteed through theoretical analysis. Extensive experimental results confirm significant advantages of our idea over different baselines such FedAvg-based fine-tuning and personalized FL in various setups. Our solution offers a promising direction for FL in practice. where minimizing training time and communication resources required real-time is among key challenges.
Acknowledgments
This work was supported by IITP fund from MSIT of Korea (No. 2020-0-00626) and by National Research Foundation of Korea (No. A2A226111
References
[1] Keith Bonawitz, Hubert Fichner, Wolfgang Grieskamp, Dzmitry Huba, Alex Ingerman, Vladimir Ivanov, Chloe Kiddon, Jakub Konecny, Stefano Mazzocchi, H Brendan McMahan, et al. Towards federated learning at scale: System design. arXiv preprint 1rXiv:1902.01000 2019.
[2] Sebastian Caldas, Sai Meher Karthik Duddu, Peter Wu, Tian Li, Jakub Konečný, H Brendan McMahan, Virginia Smith, and Ameet Talwalkar. Leaf: A benchmark for federated settings. arXiv preprint arXivv 2018.
[3] Fei Chen, Mi Luo, Zhenhua Dong, Zhenguo Li, and Xiuqiang He. Federated meta-learning with fast convergence and efficient communication. arXiv preprint arXiv: 1/XXvv/802.07876 2018.
[4] Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Personalized federated learning with theoretical guarantees: A model-agnostic meta-learning approach. Advances in Neural Information Processing Systems, 33, 2020.
[5] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adapta- tion of deep networks. In ICML, 2017.
10

---

[6] Neel Guha, Ameet Talwalkar, and Virginia Smith. One-shot federated learning. arXiv preprint arXiv:1902.11173 2019.
[7] Yihan Jiang, Jakub Konečný, Keith Rush, and Sreeram Kannan. Improving federated learning personalization via model agnostic meta learning. arXiv preprini arXiv: 12488, 2019.
[8] Anirudh Kasturi, Anish Reddy Ellore, and Chittaranjan Hota. Fusion learning: A shot federated learning. In International Conference on Computational Science, pages 424-436. Springer, 2020.
[9] Jakub Konecny, H. Brendan McMahan, Felix X. Yu, Ananda Theertha Suresh, and Dave Bacon. Federated learning: strategies for improving communication efficiency. In NIPS Workshop on Private Multi-Party Machine Learning, 2016.
[10] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[11] Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. Federated learningg Chal- lenges, methods, and future directions. arXiv preprint arXiv:1908.0787.. 2019.
[12] Sen Lin, Guang Yang. and Junshan Zhang. A collaborative learning framework via federated meta-learning. arXiv preprint (tt::2000.0322233 2020.
[13] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efiie effliccenf learning of deep networks from decentralized data. In Artificial Intelligence and Statistics, pages 1273 1282, 2017.
[14] Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. 2017.
[15] Amirhossein Reisizadeh, Aryan Mokhtari, Hamed Hassani, Ali Jadbabaie, and Ramtin Pedarsani. Fedpaq: A communication-ccii -efficiiinnnn federated learning method with periodic averaging and quantization. arXiv preprint 1909:190913000 2019.
[16] Felix Sattler, Simon Wiedemann, Klaus-Robert Müller, and Wojeiech Samek. Robust and communication-effcceent federated learning from non-iid data. IEEE transactions on neural networks and learning systems, 2019.
[17] MyungJae Shin, Chihoon Hwang, Joongheon Kim, Jihong Park, Mehdi Bennis, and Seong-Lyun data augmentation for one-shot federated learning. arXiv
Kim. Xor mixup: preprint 01:22000000144, 2020.
[18] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning, In Advances neural information processing systems, pages 4077-4087, 2017.
[19] Oriol Vinyals, Charles Blundell, Timothy Lillicrap. Daan Wierstra, et al. Matching networks for one shot learning. In Advances in neural information processing systems, pages 3630-3638, 2016.
[20] Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Federated learning with non-iid data, arXiv preprint arXiv:1806.00822. 2018.


Reviews:
[{"review_id": "yMTRwTMR6D4", "reviewer": "Reviewer_NDpP", "summary": "This paper studies a new problem setting called few-round learning in federated learning, and proposes a meta-learning based method to solve the problem. Experiments on real datasets are used to verify the effectiveness of the proposed method. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper studies a new problem setting called few-round learning in federated learning, and proposes a meta-learning based method to solve the problem. Experiments on real datasets are used to verify the effectiveness of the proposed method. \n\nThe few-round learning problem seems to be interesting and useful in federated learning, because it can reduce the cost in federated learning. The proposed meta-learning based method seems to be reasonable. Furthermore, the proposed meta-learning itself can be completed via federated learning. Experiments on real datasets show that the proposed method can outperform baselines for arbitrary groups of clients. \n\nThe experiments can be improved. The datasets used for evaluation are those for meta-learning evaluation which are relatively small, but this paper is for federated learning rather than meta-learning. Hence, adopting datasets suitable for federated learning can improve the convincingness of the proposed method. Furthermore, the paper does not introduce the hardware and platform of the experiments. Experiments on a platform of real applications (or similar settings) can also make the results more convincing. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "bldblwNbqtM", "reviewer": "Reviewer_wVqa", "summary": "This paper presents a method for using meta-learning to train an initial model, which is intended to achieve optimal performance on an arbitrary downstream tasks when trained for R rounds of federated learning (for some small R). The paper presents aa meta-learning-flavored algorithm for conducting the pretraining step, and provides theoretical bounds on the gradient of the resulting loss (Theorem 1). The paper includes some empirical results on CIFAR100, miniImageNet, and FEMNIST, with comparison to some other baselines (although no baselines exist from other works for this particular task, according to the authors).", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall: In general, I found the paper to address a task which seems relevant to the quickly-growing area of federated learning. This seems to be an extremely relevant task to introduce to the field, and the paper's approach is logically presented. I found the theoretical analysis somewhat lacking, since Theorem 1 only bounds the loss gradients (and not the loss), which does not seem to translate into a useful measure of model accuracy (only convergence to some value of \\epsilon). It is difficult to evaluate the empirical results given the baselines which are not really baselines (since they are optimized for entirely different scenarios), but the empirical results seem good. In general, I think that the paper would be a candidate for acceptance if the feedback below is satisfactorily addressed.\n\nMajor Comments:\n\n* The authors note that no prior work has addressed this task. However, this makes it difficult to interpret the results of the empirical experiments, which essentially compare to three techniques not applicable to the \"few-round\" task, plus three versions of the proposed algorithm; effectively the other \"Methods\" (three versions of FRL) amount to an ablation study. I would encourage the authors to explore better baselines -- perhaps something from the meta-learning literature, at the very least, to give a sense for how those approaches may compare. This would give a dual \"FL only\" and \"meta-learning only\" perspective on why the combined FL+meta-learning approach is superior.\n\n* The \"non-IID\" scenario described at L295 is a very weak model of non-IID-ness, so much so that I hesitate to refer to it as such. I would recommend to either find a better non-IID scenario (examples abound in the FL literature), clarify why this is comparable to other non-IID FL tasks, or remove the \"non-IID\" framing.\n\n* It would be useful to provide some motivation for why meta-learning in particular is motivated in this scenario, as opposed to some form of unsupervised pretraining which has performed quite well for a large variety of downstream tasks e.g. in the NLP literature (for example, the recent work on T5).\n\n* The authors make significant efforts to differentiate their task of so-called \"few round learning\" from existing FL tasks such as personalized learning. The difference is clear; however, the authors could do more to motivate the significance of this task. While I think there are clear arguments for this, it would be useful to make them explicit in the paper.\n\n* I don't think the term \"few-round learning\" in the paper, nor the title itself, accurately represent the work described in this paper; the existing characterization make it sound like \"doing FL in the fewest rounds possible\". I do not have a great suggestion, but something more like \"Meta-Learning for Few-Round Downstream FL\", while less elegant, seems more accurate.\n\n* The paper as-is is not self-contained; it relies on class prototypes and distance metrics only described in [18]. Please clearly describe the necessary components in the paper directly.\n\nMinor Comments:\n\n* The confusing phrase \"the task of the group conducting FL\" is used in both abstract and intro (L38). I would recommend something like \"the downstream task for which the pretrained model will be used\" instead.\n\n* Should the equation f(...) on L52 also depend on the task, or somehow indicate that this is over all downstream tasks?\n\n* The terms \"query set\" and \"support set\" are never formally defined in the paper, nor are the methods for assembling them clearly described, to my reading. Similarly, I do not think that \"class prototypes\" are clearly described. Please clarify.\n\n* I thinkSection 2.2.1 could be better titled and organized. It is essentially a walk-through of Algorithm 1 (which could be better reflected in the title); also, each bolded section should refer to specific relevant lines in the Algorithm.\n\n* The assumptions given in Section 3 are quite strict. Are these met in the experiments, or can we approximately compute them? It seems these could also create opportunities for an adversary to break the guarantees by injecting samples/clients which deliberately create large variance or large gradient variance. This could be worth at least a brief discussion or clarification.\n\n* It isn't clear to me how Fig 3(c) has more \"clearly desirable\" (L354) properties than 3(b). Please clarify.\n\nTypos etc.\n\n* Fig 1 caption: \"hope to classify an arbitrary task\" --> hope to perform an arbitrary classification task\n* What is meant by \"actual inference\" in Fig. 1 caption? Please clarify.\n* L74: \"via episodic\" --> via an episodic\n* L125: \"diseases that would\" -> diseases than would\n* Line 11 of Algorithm 1 should refer to Equation (3).\n\n**Update**: I have read the author response. This largely addresses my main concerns with the paper, enough so that I will increase my rating from 6 to 7. I do hope the authors will consider further clarification regarding the \"downstream\" target of the FL in the title and/or abstract; it is my opinion that readers will benefit from this clarification.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2hrs"}, "extra_text": {}}, {"review_id": "Fe4L-wosy_", "reviewer": "Reviewer_1tsf", "summary": "The paper introduces a problem that is based on both meta-learning and federated ideas.  They call this few-round learning, where the goal is to prepare an initial model that can quickly adapt to any group of clients within only a few rounds of FL. The problem is in my opinion significant and has a lot of practical application. The introduce a meta-learning algorithm using few rounds of FL followed by inference, to be performed by a group of clients on a possibly unseen task. The also show experiments that the scheme outperforms  existing  pre-training  approaches including fine-tuning via FedAvg and personalized FL in both IID and non-IID scenarios. I think the paper has core value for the FL field. ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper reads well and contributes a new solution to a unique and interesting problem. The authors have explained the problem and the importance of it in the field if federated learning very well. The main contribution in my opinion is  the fact that they build a global model that can be re-used for several tasks with few learning episodes at the local clients. In their words, \"The global prototypes serve as prior knowledge, a form of regularization, and prevent local models from overfitting to the local data. Moreover, the global prototypes (reflecting all classes across clients) can assist the local model to learn a more general embedding space\". So basically we have a system that learns the general features of the data at the global layer and relies on local models for specific features in the local data. More over these local models can benefit from the meta-models to have better generalization behavior. I think authors can do a bit of a better prior work summarization on the FL field and introduce comparisons with other FL techniques that outperform FedAvg. \n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I am not quite sure why there is a section comparing the results with the personalized FL while the goal here is obviously (and as stated in the paper) different. It would have been better if other FL methods apart from FedAVG were considered for the comparison instead of the focus on comparison with personalized FL. Personalized FL is in my opinion the opposite problem to the problem at hand so not sure of the value of the comparison although we see improvement on the personalization part. \n\nA few papers worth adding in references and comparisons:\n\nhttps://proceedings.neurips.cc//paper_files/paper/2020/hash/fb2697869f56484404c8ceee2985b01d-Abstract.html\nhttps://openreview.net/forum?id=BkluqlSFDS\n\n", "review_text": "", "extra_scores": {"ethical_concerns": "NA", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "AkVt02ebYge", "reviewer": "Reviewer_7HAh", "summary": "The paper designs an initial global model which adapts to few round federated learning for possibly unseen tasks.  A meta-learning based approach with a global prototype-assisted learning strategy is built to tackle this problem. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Although the authors take advantage of known techniques from MAML and prototype assisted learning strategies, I would say that this paper would be of interest for the federated learning community as the studied problem is really practical. It can be seen as a way to reduce the high communication cost introduced by the classic federated learning if every time the algorithm needs to be rerun for new tasks. The authors initiate the study of this problem and propose an efficient method with theoretical guarantees.   \n\nThe paper is written quite well and I like the precise description step by step for their algorithm. However in the paragraph of “key idea” of Introduction, lines 51-53 are not really clear for who starts reading from the beginning. The notations are introduced in a sudden without precise explication such as D_k (but I got the idea of this part after reading the algorithm). This is just a minor suggestion for the flow of the paper. The experiments are conducted for image classification tasks, and the proposed method is compared with multiple baselines which are quite fair. The description of the details of the experiments are clear and well organized. The performances of the proposed method are quite convincing.\n\nJust have a few questions related to the communication cost of the novel method:\n1. For GPAL, at the deployment stage, the clients will need to train $\\Gamma^{R-1}$ as well (e.g. the client needs to send the local prototypes). Will this add too much communication budget to few-round federated learning when the number of the classes is large? If the number of clients in the deployment stage is large, this may worsen the communication bottleneck.\n2. To save the communication cost in total for deployment (wrt the number of rounds to reach a given accuracy), which value of $R$ is more interesting? Will there be an optimal one? I saw the supplementary material part of the dismatch R part and Target R=5, which somehow suggests that R=3 may be better. I would like to have the authors' insight on it.\n\nIn total, I would recommend this paper to be accepted. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6 "}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a meta-learning approach for federated learning which allows arbitrary groups of clients can obtain a good global model in a few communication steps.

The reviews were quite positive and the author response (which contained additional experimental results) helped to further consolidate the scores. Overall, the problem of few-round FL and the proposed approach were deemed to be nicely motivated, interesting and useful in practice.

Therefore, the paper is accepted.
Decision: Accept (Poster)