On Bridging Generic and Personalized Federated Learning for Image Classification
Authors: Hong-You Chen
Wei-Lun Chao
Venue: ICLR 2022 Spotlight (2022)
Keywords: federated learning
personalization
image classification

Abstract: Federated learning is promising for its capability to collaboratively train models with multiple clients without accessing their data, but vulnerable when clients' data distributions diverge from each other. This divergence further leads to a dilemma: "Should we prioritize the learned model's generic performance (for future use at the server) or its personalized performance (for each client)?" These two, seemingly competing goals have divided the community to focus on one or the other, yet in this paper we show that it is possible to approach both at the same time. Concretely, we propose a novel federated learning framework that explicitly decouples a model's dual duties with two prediction tasks. On the one hand, we introduce a family of losses that are robust to non-identical class distributions, enabling clients to train a generic predictor with a consistent objective across them. On the other hand, we formulate the personalized predictor as a lightweight adaptive module that is learned to minimize each client's empirical risk on top of the generic predictor. With this two-loss, two-predictor framework which we name Federated Robust Decoupling (Fed-RoD), the learned model can simultaneously achieve state-of-the-art generic and personalized performance, essentially bridging the two tasks. 

Published as a conference paper at ICLR 2022
ON BRIDGING GENERIC AND PERSONALIZED FEDERATED LEARNING FOR IMAGE CLASSIFICATION
Hong- You Chen The Ohio State University, USA
Wei-Lun Chao The Ohio State University, USA
ABSTRACT
Federated learning is promising for its capability to collaboratively train models with multiple clients without accessing their data, but vulnerable when clients' data distributions diverge from each other. This divergence further leads to dilemma: "Should we prioritize the learned model's 's generic performance (for future use at the server) or its personalized performance (for each client)?" These two, seemingly competing goals have divided the community to focus on one or the other, yet in this paper we show that it is possible to approach both at the same time. Concretely, we propose a novel federated learning framework that explicitly decouples a model's 's dual duties with two prediction tasks. On the one hand, we introduce a family of losses that are robust to non non-identical class distributions, enabling clients to train a generic predictor with a consistent objective across them. On the other hand, we formulate the personalized predictor lightweight adaptive module that is learned to minimize each client's 's empirical risk on top of the generic predictor. With this fwo-loss, !wo-predictor framework which we name Federated Robust Decoupling (FED-RoD), the learned model can simultaneously achieve state-of- the-art generic and personalized performance, essentially bridging the two tasks.
INTRODUCTION
Large-scale data are the driving forces for modern machine learning but come with the risk of data privacy. applications like health care, data are required to kept separate to enforce ownership and protection, hindering the collective wisdom (of data) for training strong models. Federated learning (FL), which aims to train model with multiple data sources (i.e., clients) while keeping their data decentralized, has emerged
popular paradigm to resolve these concerns (Kairouz et al., 2019).
The standard setup of FL seeks to train a single "global" model that can perform well on generic data distributions (Kairouz et al., 2019), e.g., the union of clients' data. As clients' data are kept separate, mainstream algorithms like FEDAVG (McMahan et al., 2017) take a multi-round approach shown in Figure 1. Within each round, the server first broadcasts the "global" model the clients, who then independently update the model locally using their own (often limited) data. The server then aggre- gates the "local" models back into the "global" model and proceeds to the next round. This pipeline is shown promising if clients' data are IID (i.e., with similar data and label distributions) (Stich, 2019; Zhou & Cong, 2017), which is, however, hard to meet in reality and thus results in a drastic performance drop (Li et al., 2020b; Zhao et al., 2018), Instead of sticking to a single "global" model that features the generic performance, another setup of FL seeks to construct "personalized" model for each client to acknowledge the heterogeneity among clients (Dinh et al., 2020; Hanzely et al., 2020; Smith et al., 2017). This latter setup (usually called personalized FL) is shown to outperform the former (which we name generic FL) regarding the test accuracy of each client alone.
So far, these two seemingly contrasting FL setups are developed independently. In this paper, we however found that they can be approached simultaneously by generic FL algorithms like FEDAVG.
Concretely, algorithms designed for generic FL (G-FL) often discard the local models {wm } after training (see Figure 1). As result, when they are evaluated in a personalized setting (P-FL), it is the global model u being tested (Arivazhagan et al., 2019; Dinh et al., 2020; Fallah et al., 2020; Li et al., 202 Liang et al., 2020; Smith et al., 2017: Zhang al., 2021). Here, we found that if we instead keep {w,)) } and evaluate them in P-FL, they outperform nearly all the existing P-FL algorithms. In other words, personalized models seem to come for free from the local training step of generic FL.

---

Published as a conference paper at ICLR 2022
At first glance, this may not be totally surprising: local training in G-FL algorithms is driven by the client's empirical risk, which is what personalized model strives to optimize'. What really surprises us
Local training
W m
Client
is that even without an explicit regularization term imposed by most P-FL algorithms (Dinh et al., 2020; Smith al., 2017), the local models of G-FL algo- rithms can achieve better generalization performance. We conduct a detailed analysis and argue that global aggregation taking average over model weights indeed acts like a regularizer for local models. More- over, applying advanced G-FL. algorithms (Acar et al., 2021) to improve the G-FL accuracy seems to not hurt the "local" models' P-FL accuracy.
Local data
Server
Broadcast
Global aggregation
G-FL Acc
A-c
MY my
Round
Figure I: The multi-round generie FL pipeline (top). The dashed arrow indicates that local mod- els or statisties may be carried to the next round. Here we apply FEDAVG (McMahan al., 2017) on CIFAR-10 with 20 non-IID clients (see section 5), and show that personalized models come free from generic FL (bottom). The global model w outperforms local models W.o the bottom- generic accuracy (G-FL), yet Wm outperforms w on the bottom-right personalized accuracy (P-FI.). The accuracy is computed at the end of cach round.
Building upon these observations, we dig deeper into generic FL. Specifically for classification, the non- IID clients can result from non-identical class dis- tributions non-identical class-conditional data distributions. One way to mitigate their influences make the local training objectives more aligned among clients. While this can hardly be achieved for the latter case without knowing clients' data, we can do so for the former case by setting a consistent goal among clients the learned local models should classify every class well, even if clients' data have different class distributions. We realize this by viewing each client's local training as an independent llass-imbalanced problem (Cui et al., 2019; He & Garcia, 2009) and applying objective functions dedicated to it (Cao et al., 2019; Ren et al., 2020). As will be shown in section 5, these among clients, making the resulting "global" model more robust non-Ill) conditions.
objectives lead to much consistent local training
The use of balanced objectives, nevertheless, degrades the local models' P-FL performance. This is because the local models are no longer learned to optimize clients' empirical risks.
To address this issue, we propose a unifying frame- work for G-FL and P-FL which explicitly decouples a local model's 's dual duties: serving as the personalized model and the ingredient of the global model. Con- cretely, we follow the FEDAVG pipeline and train the local model with the class-balanced loss, but on top of the feature extractor, we introduce a lightweight personalized predictor and train it with client's em- pirical risk (see Figure 2). With this two-loss, two- predictor framework which we name Federated Ro- bust Decoupling
Feature Extractor
G-llend
9G
Balanced risk
P-Head [Mypeenel]
9r
Empirical ris
Clien's class distibulion
Figure 2:
Local training of FED-ROD. Yel- low/blue boxes are the models for G-FL/P-FL. Green ellipsoids are the learning objectives. The red area means what to be aggregated at the server.
-DD, the resulting global model can be more robust to non-identical class distributions; the personalized predictor can lead to decent P-FL accuracy due to the implicit regular- ization and the empirical loss. Specifically for the personalized predictor, we propose to explicitly parameterize it with clients' class distributions via a hypernetwork (Ha et al., 2017). That is, we learn shared meta-model that outputs personalized predictors for clients given their class distribu- tions. This not only enables zero-shot model adaptation to new clients (without their data but class distributions), but also provides better initialization to fine-tune the models given new clients' data.
We validate FED-ROD on multiple datasets under various non-III settings. FED-ROD consistently outperforms existing generic and personalized FL algorithms in both setups. Moreover, FED-ROD is compatible with and can further improve advanced generic FL algorithms like FEDDYN (Acar et al., 2021) whenever non-identical class distributions occur. Our contributions are three-folded:
Unlike most of the previous works that focus on either generic FL or personalized FL, we propose FED-ROD to excel on both at the same time. We validate FED-ROD with extensive experiments.
'oweever when G-FI. algorithms are tested on the P-FI. seFup the literature does not use their local models.
2

---

Published as a conference paper at ICLR 2022
We show that strong personalized models emerge from the local training step of generic FL algorithms, due to implicit regularization. We further show that baasn-bl objectives are effective for improving the generie FL performance when clients have different class distributions. FED-ROD enables zero-shot adaptation and much effective fine-tuning for new clients.
RELATED WORK (A DETAILED VERSION IS IN APPENDIX A)
Generic federated learning. FEDAVG (McMahan et al., 2017) is the standard algorithm, and many works are proposed to improve it, either in the global aggregation step (Chen & Chao, 2021;
Hsu et al., 2019; Lin et al., 2020; Reddi et al., 2021; Wang et al., 2020a; Yurochkin et al., 2019)
or local training step (Malinovskiy et al., 2020; Wang et al., 2020b; Yuan & Ma, 2020; Zhao et al., 2018). For example, to reduce local models' drifts from the global model, FEDPROX (Li et al., 2020a) and FEDDYN (Acar et al., 2021) employed regularization toward the global model; SCAFFOLD (Karimireddy et al., 2020a) leveraged control variates to correct local gradients. We also aim to reduce local models' drifts but via a different way. We apply objective functions in class-imbalanced learning (He & Garcia, 2009), which are designed to robust to class distribution changes. The closest to ours is (Hsu et al., 2020), which used a traditional class-imbalanced treatment named re-weighting, We show that more advanced techniques can be applied to further improve the performance, especially under extreme non-IID conditions where re- weighting is ineffective.
Personalized federated learning. Many approaches for personalized FL (Kulkarni et al., 2020) are based on mulli--task learning (MTL) (Ruder, 2017; Zhang & Yang. 2017). For instance, Smith et al. (2017) encouraged related clients to learn similar models; Dinh et al. (2020); Hanzely et al. (2020); Li et al. (a) regularized local models with learnable global model. Our approach is inspired by MTI as well but has notable differences. First, we found that global aggregation in generic FL already serves as a strong regularizer. Second, instead of learning for each client a feature extractor (Bui et al., 2019; Liang et al., 2020) or an entire model, FED-ROD shares a single feature extractor among clients, inspired by Caruana (1997); Zhang et al. (2014). This reduces the total parameters to be learned and improves generalization. Compared to (Arivazhagan et al., 2019; Collins et al., 2021) which also learned a shared feature extractor, FED-ROD simultaneously excels in both FL setups.
Instead of designing specific algorithms for personalized FL,, Cheng et al. (2021); Wang et al. (2019); Yu et al. (2020) showed that performing post-processing (e.g., fine-tuning) a generic FL model (e.g., w in FEDAVG) leads to promising personalized accuracy. We further showed that, the local models {wm} learned in FEDAVG and other generic FL algorithms are strong personalized models.
We note that, while many personalized FL algorithms also produce global model, it is mainly used to regularize or construct personalized models but not for evaluation in the generic setup. In contrast, we learn models excel in both setups via single framework without sacrificing either of them.
recent work PFEDHN (Shamsian et al., 2021) also applies hypemnetorrs (Ha et al., 2017) but in a very different way from FED-ROD. PFEDHN learns hypernetwork the server to aggregate clients' updates and produce entire models for them for the next round. In contrast, we learn the hypernetwork locally to construct the personalized predictors, not the entire models, for fast adaptation clients.
3 PERSONALIZED MODELS EMERGE FROM GENERIC FEDERATED LEARNING
In this section, we show that personalized FL (P-FL.) models emerge from the training process of generic FL (G-FL) algorithms. To begin with, we review representative G-FL and P-FL algorithms.
3.1 BACKGROUND
Generic federated learning. a generic FL setting with M clients, where each client has a data set [mml the optimization problem to solve can be formulated as
Dm, (x;, w
M [Dml
min L(w) = Σ
L. (w), where f. D
Σ C(a,, Ue: w)
(1)
i
m==
Here, wi is the model parameter; D = Dm is the aggregated data set from all clients; L. (w) is the empirical risk computed from client m's data; e is a loss function applied each data instance.
3

---

Published as a conference paper at ICLR 2022
Federated averaging (FEDAvG). As clients' data separate, Equation 1 cannot be solved directly. A standard way to relax it is FEDAvG (McMahan et al., 2017), which iterates between two steps, local training and global aggregation, for multiple rounds of communication
M
1 D
Local:
Wm
arg min fm (w)) initialized with wi;
Global:
V
Wm
(2)
m==1
The local training is performed at all (or part of) the clients in parallel, usually with multiple epochs of SGD to produce the local model Wm. The global aggregation by taking element-win average over model weights. Since local training is driven by clients' empirical risks, when clients' data are non-IID, Wm would drift away from each other, making to deviate from the solution of Equation 1.
Personalized federated learning. Personalized FL learns for each client model Wm. whose goal is to perform well on client m' 's data. While there is no agreed objective function so far, many existing works (Dinh et al., 2020; Hanzely & Richtárik, 2020; Hanzely et al., 2020; Li & Wang, 2019; Li et al., 2021a; Smith et al., 2017) define the optimization problems similar to the following
(Dml [D]
min
Lm Lm (wm) R(2, w1, WM),
{92, M. 3
where R a regularizer; 52 introduced to relate clients. The regularizer is imposed to prevent wm from over-fitting client m's 's limited data. Unlike Equation 1, Equation 3 directly seeks to minimize each client's empirical risk (plus a regularization term) by the corresponding personalized model wm.-
In practice, personalized FL algorithms often run iteratively between the local and global steps well, so as to update according to clients' models. One example et al., 2020; Hanzely & Richtárik, 2020; Hanzely et al., 2020; Li et al., 2021a), e-g., by taking average over clients' models, and apply an La regularizer between and each Wm. The corresponding local training step thus could generally be formulated
define as global model (Dinh
X ||w - 12112, initialized with will
Local: ((++1) arg min L. (w) +
(4)
where with denotes the local model after the t-th round; À is the regularization coefficient. is worth noting that unlike Equation 2, w in Equation 4 is initialized by whi not by Z (or wT).
Terminology. Let us clarify the concepts of "global" vs. "local" models, and "generic" vs. "person- alized" models. The former corresponds to the training phase: local models are the ones after every round of local training, which are then aggregated into the global model at the server (Equation 2). The latter corresponds to the testing phase: the generic model is used at the server for generic future test data, while personalized models are specifically used for each client's 's test data.
3.2 LOCAL MODELS OF GENERIC FL ALGORITTMS ARE STRONG PERSONALIZED MODELS
Building upon the aforementioned concepts, we investigate the literature and found that when generic FL algorithms are evaluated in the P-FL setup, it is their global models being tested. In contrast, when personalized FL algorithms are applied, it is their local models (c.g., Equation 4) being tested. This discrepancy motivates to instead evaluate generic FL algorithms using their local models.
Figure 1 summarizes the results (see section 5 for details). Using local models of FEDAVG (i.e., Equation 2) notably outperforms using its global model in the P-FL setup. At first glance, this may not be surprising, as local training in FEDAVG is driven by clients' empirical risks. What really surprises us, will be in section is that FEDAVG's 's local models outperform most of the existing personalized FL algorithms, even if no explicit regularization is imposed in Equation 2.
3.3 INITIALIZATION WITH WEIGHT AVERAGE IS A STRONG REGULARIZER
To gain further understanding, we plot FEDAYG local models' accuracy on clients' training and test data. We do so also for state-of-the thrt personalized FL. algorithm DITTO (Li et al., 2021a), whose local training step for producing personalized models is similar to Equation As shown in Figure FEDAVG has lower training but higher test accuracy, implying that FEDAVG's 's local training is more regularized than Equation 4.
4

---

Published as a conference paper at ICLR 2022
We attribute this effect the initialization in Equation 2. Specifically, by initializing w with w, we essentially impose an Lg regularizer & 1| 1 with À - oo at the beginning of each round of local training, followed by resetting À to be O. We found that this implicit regularization leads to smaller value of ||w sot--rr) u// 2 at the end of each local training round, compared to Equation 4. Due to the page limit, we leave additional analyses in the appendix. We note that, advanced generic FL algorithms like SCAFFOLD (Karimireddy et al., 2020b) and FEDDYN (Acar et al., 2021) still apply this initialization and learn with the empirical risk during local training. Thus, their local models are strong personalized models as well.
1.0
0.8
Myle
P-FL Acc
Ditto train
0.6
FedAvg train
Ditto test
0.4
FedAvg test
50
100
Rounds
Figure 3: Comparison of the
training and test accuracy in the P- FL setup. FEDAVG's local mod- els achieve lower training accu- racy but higher test accuracy.
4 FEDERATED ROBUST DECOUPLING (FED-RoD)
The fact that personalized models emerge from generie FL. algorithms motivate us to focus more on how to improve the latter, especially when clients have non-III data distributions.
4.1 IMPROVING GENERIC FL WITH BALANCED RISK MINIMIZATION (BRM)
We first analyze what factors may lead to non-III conditions. Suppose the data instance (ac, y) of client is sampled from client-specific joint distribution Pm, (ar, y) = Pm (ar Py) (v), the non-lll) distributions among clients can result from non-identical class distributions P,,. (x) y), non-identical class-conditional data distributions Pn (y). or both. All these cases can make C. (w) deviate from [(w) in Equation 1, which is the main cause of degradation in generic FL (Li et al., 2020a;b).
One way to mitigate the influence of non-IID data is to make (w) align with each other. This can be challenging to achieve if clients have different P... (z) (xly): without knowing clients' data". it is hard to design such an aligned C. (w). However, when clients have different Pm) i.e., different and hence imbalanced class distributions, we can indeed design a consistent local training objective by setting shared goal for the clients- the learned local models should classify all the classes well. is worth noting that setting such goal does not require every client to know others' data.
Learning a classifier to perform well on all classes irrespective of the training class distribution is the main focus of class-imbalanced learning (He & Garcia, 2009; Japkowicz, 2000; Johnson & Khoshgoftaar, 2019). We therefore propose to treat each client's local training as a class-imbalanced learning problem and leverage techniques developed in this sub-field. Re-weighting and re-sampling (Buda et al., 2018) are the most fundamental techniques. Denote by Nm. the number of training instances of class for client m, these techniques adjust L... (wr) in Equation into
1 Nm.
BBl ( x Σ Un: ((x;,, yi; w), where 4y, is usually set as
or
(5)
m.
Namely, they mitigate the influence of Pm (v) by turning the empirical risk C. into balanced risk C such that every client solves a more consistent objective that is robust the class distributions. Recently, many class-imbalanced works proposed to replace the instance loss E (e.g., cross entropy) with class-balanced loss (Cao et al., 2019; Kang et al., 2020; Khan et al., 2017; Ren al., 2020; Ye et al., 2020), showing more promising results than re- re-weighting or re- re-sampling. We can also define CBR using these losses, e.g., the balanced softmax (BSM) loss (Ren et al., 2020) m
x Σ BBSM (x,, yo w)) where (BSM (x. y: w) = - log No exp(9p (zz; (2))
CBR (BW( (w)
(6)
Ceec No. exp(g.(x; w))
Here, (ar; w) is the logit for class c, C is the label space, and ry is hyper-parameter. The BSM loss an unbiased extension of softmax to accommodate the class distribution shift between training and testing. It encourages minor-class instance to claim larger logit gy w) in training to overcome feature deviation (Ye al., 2020) in testing. We list other balanced losses the appendix.
CCientts having different Pm (mly) is related to domain adaptation (Gong ct al., 2012) and generaliza- tion (Muandet ct al., 2013), which require knowing the distributions of all/some clients for algorithm design. "Tis is indeed the main cause of non-IID data distributions in the literature of FT. (Hsu al., 2019; 2020).
5

---

Published as a conference paper at ICLR 2022
Personalized model
Aggregated at the server
model
Festure Extractor
G-Hteed
Empiriaal
nisk
Fasture Estractor
G-Hfeed
Balanced
(a) Empirical risk minimization (FRM)
(b) Balanced risk minimization (BRM)
Feature Extractor
GiHead
Sa
Balanced
Festure Extractor
G-Head
fa
Balanced
P-Head (Linear)
9p
Empirical nikk
P-teed (Hypormet)
9p
Empincal risk
Clint's class
distribtion
(c) FED-ROl (linear)
(d) FED-ROD (hyper)
Figure 4: Comparison of local training strategies and model architectures. G: generie; P: personalized. Yellow boxes correspond the models for G-FL; green boxes, for P-FL. Boxes that covered by the red background are sent back the server aggregation (e.g., weight average), and ro-initialized at Green ellipsoids correspond to the learning objectives. ÿ stands for the predicted logits (of all classes); &a and ÿP come from the G-head and P-head, respectively. (a) local training with ERM; (b) local training with BRM; (c) FED- FED-ROD (lincar): learning with both BRM (for G-FL) and ERM (for P-FL) using the two-predictor (head) architecture; (d) FED-ROD (hyper):
round.
(c), but the P-head constructed by shared hypernetwrk.
We take advantage of these existing efforts by replacing the empirical risk f... in Equation 2 with balanced risk C which either takes the form of Equation 5 or applies class- Hlas-bbllanced loss (e.g., Equation 6), or both. We note that, a variant Equation 5 has been used (Hsu et al., 2020). However, experiments show that it is less effective than class-balanced losses in extreme non-llD cases. Interestingly, we found that L FEDDYN (Acar et al., 2021), because these algorithms are agnostic to the local objectives being used.
BR can easily be incorporated into advanced FL algorithms like
4.2 LOCAL TRAINING AND LOCAL MODEL DECOUPLING WITH ERM AND BRM
The use of balanced risk CB in local training notably improves the resulting global model wZss generic performance, will be seen in section 5. Nevertheless, inevitably hurts the local model Wn.3 's personalized performance, since it is no longer optimized towards client's empirical risk L..
To address these contrasting pursuits of generic and personalized FI., we propose unifying FL. framework named Federated Robust Decoupling (FED-ROD), which decouples the dual duties of local models by learning two predictors on top of a shared feature extractor: one trained with empirical risk minimization (ERM) for personalized FL (P-FL) and the other with balanced risk minimization (BRM) for generic FL (G-FL). Figure 4 (c-d) illustrates the model and local training objective of FED-ROD. The overall training process of FED-ROP RoD follows FEDAVG, iterating between local training and global aggregation. As mentioned subsection 4. other generic FL algorithms (Acar al., 2021; Karimireddy et al., 2020b; Li et al., 2020a) can easily be applied to the BRM branch to further improve the generic performance. Without loss of generality, we focus on the basic version built upon FEDAVG. We start with the model in Figure (c).
We denote by f(x; 0) the shared feature extractor parameterized by O. whose output is We denote by A ; 2b) and h 1 (z; Φm)) the generie and personalized prediction heads parameterized by ab and Φi.. respectively: both are fully-connected (FC) layers. In short, our generic model is parameterized by {0, v/) ); our personalized model for client m parameterized by (0, s/, Φm)
Notations.
Predictions.
perform =f [ax; 0), followed by ÿG 12 personalized prediction, we perform f(æ; 0). followed by ÿP = (2; v/) + 1/"((: Φm). That is, h" is an add-on to A providing personalized information that is not captured by the generic head.
For generic prediction,
(z; ub). For
The overall objective. FED-ROD learns the generic model with the balanced risk CBB and the personalized predictor with the empirical risk Lm- That is, different from Equation 1, FED-ROD aims solve the following optimization problems simultaneously
Dml
min C((0, 0.40
Σ
CBR (10,
4b)) and
min min (m0 w. dm Vm e [M]. (7) 4t..
D
We note that, L. is only used to learn the personalized head parameterized by Φm.

---

Published as a conference paper at ICLR 2022
Learning. Equation 7 cannot be solved directly in federated learning, FED-ROD follows FEDAVG to learn iteratively between the local training and global aggregation steps
Local:
0 2 arg ming, Lm ((0, 2b)),
initialized with 0, xb.
(8)
Φnn arg min min....
Lm (m 2/, (m,)m.) })
initialized with d'm: (9)
M
M + m
Dmll D
D mm [D]
Global: 0
2 m
(10)
m==1
where Sn is learned from the previous round, similar to wn in Equation 4. That is, the person- alized head will not be averaged glohally but kept locally. In our implementation, Equation 8 and Equation 9 are solved simultancously via SGD, and we do not derive gradients w.r.t. and &b from (10 3 &m }). The and & in Equation 9 thus come dynamically from the SGD updates of Equation 8. In other words, Equation
not merely fine-tuning on top of the generic model. In the end of federated learning, we will obtain 0 and &p (Equation 10) for generic predictions and {0% ti &in J.M (Equation and Equation 9) for personalized predictions, respectively. Please be referred to the appendix for the pseudocode.
4.3 ADAPTIVE PERSONALIZED PREDICTORS VIA HYPERNEETWORKS
In subsection 4.2, the parameter Φn of the personalized predictor is learned independently for each client and never shared across clients. In other words, for a new client not involved in the training phase, FED-ROD can only offer the global model for generic prediction. In this subsection, we investigate learning shared personalized predictor that can adapt new clients. Concretely, propose to learn a meta-model which can generate Øm for a client given the client's class distribution. We denote by H (am: v) the meta-model parameterized by v, whose output is Φm- Here, am E | the |C| (CI-dimensional vector that records the class distribution of client m; i.e., the c-th dimension am [c]
Nm,c Nm.e
Accordingly, the local training step of Φn in Equation is replaced by
M
D [D]
Local: v = -rg min, L. (10, sb v)), initialized with v; Global: D
vin (11)
m==
We implement P by a lightweight hypernetwork (Ha et al., 2017) with two fully-connectie layers. With the learned the meta- model H can locally generate a... based am, making it adaptive to new clients simply by class distributions. The parameter Φm. can be further updated using clients' data. We name this version FED- ROD (hyper); the previous one, FED-ROD (linear). Please Figure (c-d) for an illustration. We include more details in the appendix.
5 EXPERIMENT (MORE DETAILS AND RESULTS IN THE APPENDIX)
Datasets, models, and settings. We use CIFAR- 10/100 (Krizhevsky et al., 2009) and Fashion- MNIST (FMNIST) (Xiao al., 2017). We also include a realistic EMNIST (Cohen et al., 2017) dataset, which collects hand-written letters of thousands of writers. To simulate the non-III data distributions on CIFAR and FMNIST, we follow Hsu et al. (2019) to create a heterogeneous partition for M clients: an M -diimensional vector qe is drawn from Dir(a) for class c, and we assign data of class e client m proportionally to [m]] The resulting clients have different numbers of total images and different class distributions. With < 1, most of the training examples of one class are likely assigned
small portion of clients. Similar to Lin et al. (2020), we use M 100 clients for FMNIST and M = 20 for
and sample 20% /40% clients at every round, respectively. For EMNIST, we use the digit images, follow Caldas et al. (2018) to construct 2, 185 clients (each writer), and sample 5% clients every round. We use ConvNet (LeCun et al., 1998) similar to (Acar et al., 1021; McMahan et al., 2017). It contains convolutional layers and fully-connected layers. We train every FL algorithm for 100 rounds, with 5 local epochs in each round.
We report the mean accuracy of five times of experiments with different random seeds. We evaluate the generic performance (G-FL) using the generic model (GM) the standard generic test set. For FMNIST and CIFAR- 100100, we evaluate the personalized performance (P-FL) using personalized models (PM) the same set, but re-weight the accuracy according to clients' class distributions
7

---

Published as a conference paper at ICLR 2022
Table Results in G-FL accuracy and P-FL accuracy (%). methods with no G-FL models and we combine their P-FL models. 3: official implementation. Blue/bold fonts highlight the best baseline/our approach.
Datasct
EMNIST
FMNIST
CIFAR-10
CIFAR-100
Non-IID
Writers 1 D)
Dir(3.)
Dir(0.1) I
Dir(0.))
Dir(0.1) I Dir(0.3)
Test Set
[G-F
P.FL [G-FL P.FL [G-FL) P.FL
[G-FLI
P-FL [G-FLI P.FL
|G-FL
P-FL
[G-FL] P-FL
Method / Model
GM
[GM PM | GM JGM PM|
GM
IGM
PM GM
GM PM | GM [GM PM
GM
JGM
PMI
GM
GM
PM
FEDAYG FEDPROX
97.0
96.9 97.2
81.1
81.0 91.5
83.4
83.2 90.51
57.6
57.1 90.5
68.6
69.4 85.
41.8
41.6
70.2
46.4
46.2 61.7
97.0
92.0
92.0
82.2
82.3
91
84.5
84.5
89.7
58.7
58.9
89
09.9
09.8
84
41.6
70.4
46.5
46.4
61
SCAFFOLD
97.1
97.0
13.1
83.0
89.0
85
85.0
90.4
61.
60.8
90
71.
71.
42.1
70.4
46.5
46.5
61-7
FEDDYN 3
97.3
97.3
97.3
83.2
83.2
90.7
86.1
86.1
91.5
63.4
63.9
92.1
72.5 73.2
85.4
43.0
43.0
72.0
47.5 47.4
62.5
85.6
16.1
87.3
53.1
78.3
12.7 90.6
10.8
49.9
FEDAYG 88
54.8
54.5
89.5
66.8
29.5
28.8
90.8
46.7
82.4
23.4
34.5
55.4
FEDPER
93.3
93.1
97.2
74.5
74.4
91
799
79.9
90.4
50.4
50.2
89
64.4
64.5
84.9
37.6
37.6
71.0
40.3
40.1
62.5
FEDAVG
PFEDME
96.3
96.0
76.7
76.7
83.4
79.0
79.0
81.4
50.6
50.7
76.6
62.1
61.7
70.5
38.6
38.5
414
41.1
514
DITTO TEDFOMO
97.0
97.0
97.4
81.5
81.5
89.4
81.3
83.2
90.1
58
58
86
69.7
69.8
81.5
41.7
41.8
68.5
46.4
46.4
58.6
80.5
80.4 95.9
34.5
34.3
90.0
70.1
69.9
89.6
30.5
31.2
90.5
45.3
45.
83.4
35.4
35.1
68.9
39.6
39.3
58.4
FEDREP "B
95.0
95.1
97.5
79.5
80.1
91.8
80.6
80.5
90.5
56.6
56.2
91.0
67.7
67.5
85.2
40.7
40.7
71.5
46.0
46.0
62.1
Local only
64.21
85.9
85.01
87.41
75.71
40.0
32,5
FED-ROD (linear)
97.5
83.9
86.3
68.5
76.9
72.2
48.5
FED-ROD (hyper)
83.9
92.9
86.3
68.5
76.9
76.8
86.8
48.5
FEDDYN
85.9
85.7
95.3
94.6
68.2
68.2
92.7
74.6
85.6
46.2
48.4
(y) and average the weighted accuracy across M clients as M Σ... the instance index. This evaluation is more robust (essentially the expectation) than assigning each client a specific test set. For EMNIST, each client has its own test set with the same writing style.
Σ, (.
Σ. (m,))) Here, i is
Our variants. We mainly use Equation 6 with 7 version (cf. subsection 4.3). Table 2 provides the ablation study.
f. and report the FED-ROD (hyper) the L
Baselines. For G-FL methods including FEDAVG (McMahan et al., 2017), FEDPROX (Li et al., 2020a), SCAFFOLD (Karimireddy et al., 2020b), and FEDDYN (Acar et al., 2021), we use their global models v for G-FL evaluation; their local models (i.e., wm in Figure 1) for P-FL evaluation. For P-FL methods, to evaluate their G-FL performance, we use the available global models in PFEDME (Dinh et al., 2020) and DITTO (Li al., 2021a) or average the final personalized models for MTL (Smith et al., 2017), FEDPER (Arivazhagan et al., 2019), LG-FEDAVG (Liang et al., 2020), FEDFOMO (Zhang et al., 2021), and FEDREP (Collins et al., 2021).
To illustrate the difference between applying GMs and PMs in setting, we also evaluate the P-FL performance using GMs, which how FEDAVG has been applied to P-FL in literature.
5.1 RESULTS
FED-ROD bridges G-FL, and P-FL and consistently outperforms all generic and personalized FL methods. Table summarizes the results. In terms of G-FL. accuracy, advanced local training (i.e., SCAFFOLD, FEDPROX, and FEdDYN) outperforms FEDAVG and personalized methods, and our FED-ROD can have further gains by using balanced risk minimization (BRM). We also investigate combining FED-ROD and FEDDYN (Acar et al., 2021), using the latter to optimize the generic model with BRM, which outperforms either ingredient in many cases. We report the G-FL accuracy of personalized FL algorithms mainly to investigate if they have similar properties like FEDAVG: an algorithm designed for one setup can also construct models for the other setup.
In terms of P-FI. accuracy. by using PMs most methods outperform the baseline of local training with individual client's data without communication (i.e., local only). justifying the benefits of federated collaboration'. For generic FL methods, using PMs (i.e., local models {wm)) clearly outperforms using GMs (i.e., w), which supports our claims and observations in Figure and subsection 3.2. It is worth noting that the local models from generic FL methods are highly competitive or even outperform personalized models produced by personalized FL methods. This provides generic FL methods with an add-on functionality to output personalized models by keeping the checkpoints on clients after local training. Our FED-ROD achieves the highest P-FL accuracy and we attribute this to (a) the shared feature extractor learned with the balanced risk and re-initialized every round to benefit from implicit regularization; (b) the personalized head learned with clients' empirical risks.
'PMs of "local only" could outperform the GM of FEDAVG on the P-FL. accuracy, especially when the non-III condition becomes severe (e.g., Dir (0.1)) it is hard train single GM perform well in P-F...
8

---

Published as a conference paper at ICLR 2022
Table 2: Ablation study on vari- ants of TED-ROD. FT. tie-tuning
Table 3: FED-ROD with dif-
Table 4: P-FL. accuracy on future non-IID clients (Dir(0.3) for all datasets). Each cell is before/after local fine-tuning.
ferent balanced losses. *: BSM
Test Set
[G-FLI P-FL
Method/ Model
I GM JGM PM
Test
[G-FLI P.FL
Method
JMNIIST CIFAR- 10 CIFAR-IO 100
Centralized
I 85.4 85.4
Loss / Model
IGM
GM PM
FEDAVO BEDDYN PER-FEDAYG
3/7.22 56.2/76.0
18.9/54. 40.1/572
61.7762 12.1/89.6
FEDAyG (BRM)
68.6
69.4 85.1
entropy
69.4 85.1
60.07998
37.6/5.66
2020)
FEDAYG (BRM. FT)
76.8
76.7 84.5
(Coo 2019)
FED-ROD (linar) 88 5/1.1
62 4/80.2
40.0/58.
FED-RoD BRM)
76.9
76.8
2020)
75.2
8
FED-ROD hyyer
98.991
75.
40.7/59.0
FED ROD (hyper. BRM)
76.9
76.8
86.8
(Ren al., 2020)
76.9
76.8 86.5
+YEDDYN 89.2/9 77. 1/8.5
41,4/59.5
BRM effectively reduces the variance of G-FL accuracy and local gradients. To understand why FED-ROD improves G-FL, we visualize the global model w's s and each local model Wm's 's G-FL accuracy on CIFAR-10 (Dir(0.3)), in Figure 5 (upper). FED-ROD not only learns better global model for G- FL. but also has a smaller variance of accuracy across the local models' generic heads (as their objectives are more aligned). We also show how Wtn deviates from w after local training in Figure 5 (lower). FED-ROD has smaller variance. This coincides with the study in (Kong et al., 2021): lower variances of the local gradients could imply better generic performance.
FedAvg -ftter
FedAvg- before
Fed-RoD- after
Fed-RoD before
70-
GFF Acc
M Amm
18
10
Rounds
1.0
redAvg
redDyn
FED-ROI ROD benefits from decoupling, We compare several variants of FED-ROD (cf. Figure 4), with one head (reduced to FEDAVG) or different networks (linear/hyper). We evalu- ate on CIFAR-IO (Dir(0.3)). As shown in Table 2, FEDAVG with BRM significantly improves G-FL but degrades in P-FL. FED-ROD remedies this by training a decoupled personalized head. We note that, FED-ROD does not merely fine-tune the global model with clients' data (cf. subsection 4.2). We also compare different balanced losses in Table 3: advanced losses outperforms importance re-weighting (Hsu et al., 2020).
Gradtent Variance
0.8
FedProx
Fed-RoD
SCAFFOLD
0.6
0.2
20
40
60
80
100
Hounds
Figure 5: Upper: G-FL test accuracy along the training rounds before/after av- eraging the local models. Lower: vari- ances of wm.
across clients.
FED-ROD (hyper) benefits future clients. To validate the gencralizability to new clients, we build on the Dir(0.3) non- IID setting for FMNIST and CIFAR- ing data into 100 clients (50 are in training; 50 are new). We train on the 50 training clients for 100 rounds (sampling 20 of them every round). We then evaluate the 50 new clients indi- vidually, either using the global model directly or fine-tuning it with clients' data for several steps. Table 4 and Figure 6 shows the averaged accuracy on new clients. Without fine-tuning, FED-ROD (hyper) can already generate personalized models, and outperforms others methods stably with fine-tuning.
100
FedAvg
FndDyn
but split the train-
FedProx
Per FedAug Fod-RoD
90-
P- Acc.
SCAFFOLD
80
70-
40
60
90
100
Local Steps
Figure 6: The average P-FL accuracy on future clients, with local training.
More results and analyses in the Appendix. The Appendix includes studies with iidalaiimbl global distributions and we show that FED-ROD still performs well. BRM can be further improved with meta-learned hyper-parameters. We validate that re-initializing the local models by the global model at every round (i.e., Equation 2) does lead to much smaller regularization loss than Equation to support our claim in subsection 3.3. More comprehensive results regarding more clients, deeper backbones, compatibility with other methods, robustness against adversaries, etc, are also provided.
6 CONCLUSION
Most of the existing work in federated learning (FL. has been dedicated to either learning better generic model or personalized models. We show that these two contrasting goals can be achieved simultaneously via novel two-loss, two-predictor FL framework FED-ROP Concretely, we show that strong personalized models emerge from the local training of generic FL algorithms, due to implicit regularization; imposing class-balanced objectives further improves the generic FL accuracy when clients have non-IID distributions. FED-ROD seamlessly incorporates these two observations to excel in both FL settings, and further enables fast adaptation to new clients via an adaptive module.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
This research partially supported by NSF IIS-2107077, NSF OAC-2118240 NSF DAC-2112600 and the OSU GI Development funds. We are thankful for the generous support of the computational resources by the Ohio Supercomputer Center and AWS Cloud Credits for Research.
REPRODUCIBILITY STATEMENT
We report the results with the average over runs of different random seeds. We exhaustively provide the information about the hyperparameters, datasets, evaluation, and other details in section and Appendix C, which should be comprehensive for reproducibility. We also provide our code in pectps://gite www.com//ccc..eeeee.ecc htmp/hongyouc/Feeee (eaeyooue//edRDD.....
REFERENCES
Durmus Alp Emre Acar, Yue Zhao, Ramon Matas, Matthew Mattina, Paul Whatmough, and Venkatesh Saligrama. Federated learning based on dynamic regularization. In ICLR, 2021. 2, 3, 5, 6, 7, 16, 17, 18, 23, 24, 29, 30
Alekh Agarwal, John Langford, and Chen-Y Yu Wei. Federated residual learning, arXiv preprint arXiv: 12880, 2020. 17
Kartik Ahuja, Karthikeyan Shanmugam, Kush Varshney, and Amit Dhurandhar. Invariant risk minimization games. In ICML, 2020. 17, 22, 23
Manoj Ghuhan Arivazhagan, Vinay Aggarwal, Aaditya Kumar Singh, Sunav Choudhary. Federated learning with personalization layers. arXiv preprint arXiv:1912.008/5, 2019. 1, 3, 8, 17,
Martin Arjovsky. Léon Bottou, Ishaan Gulrajani, David Lopez-Paz. Invariant risk minimization. arXiv preprint 1907: 2019. 17, 22
Keith Bonawitz, Hubert Eichner, Wolfgang Grieskamp, Dzmitry Huba, Alex Ingerman, Vladimir Ivanov, Chloe Kiddon, Jakuh Konečný, Stefano Mazzocchi, Brendan McMahan, et al. Towards federated learning at scale: System design. orXiv preprint arXiv: 0e0//992200000 2019. 16
Mateusz Buda, Atsuto Maki, Maciej A Mazurowski. A systematic study of class imbalance problem convolutional neural networks. Neural Networks, 106:249-259, 2018.
Due Bui. Kshitiz Malik, Jack Goetz, Honglei Liu, Seungwhan Moon, Anuj Kumar, and Kang Shin. Federated user representation learning. arXiv preprint 19Xiv: 12535, 2019. 3, 17
Sebastian Caldas, Peter Wu, Li. Jakub Konečný, Brendan McMahan, Virginia Smith, and Ameet Talwalkar. Leaf: A benchmark for federated settings. arXiv preprint arXiv:1812.01097, 2018.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with label-distribution -warr margin loss. Conference on Neural Information Processing Systems, 2019, 2, 5, 9, 18, 20, 21, 28
Rich Caruana. Multitask lcarning. Machine learning, 28(1):41-75, 1997. 3
Soravit Changpinyo, Wei-Lun Chao, Boqing Gong, and Fei Sha. Synthesized classifiers for zero-shot learning. In CVPR, 2016.
Soravit Changpinyo, Wei-Lun Chao, and Fei Sha. Predicting visual exemplars of unseen classes for zero-shot learning. In JCCV, 2017.
Soravit Changpinyo, Wei-Lun Chao, Boqing Gong, and Fei Sha. Classifier and exemplar synthesis for zero-shot learning. IJCV, (28(1):166-2)))) 2020.
Fei Chen, Mi Luo, Zhenhua Dong, Zhenguo Li, and X. He. Federated meta-learning with fast convergence and efficient communication. arXiv: Learning, 2018. 17
Hong-You You Chen and Wei-Lun Chao. Fedbe: Making bayesian model ensemble applicable federated learning. In ICLR, 2021. 3, 16, 18
Gary Cheng, Karan Chadha, and John Duchi. Fine-tuning is fine in federated learning. arXiv preprint arXiv:2 2021.
10

---

Published as a conference paper at ICLR 2022
Gregory Cohen, Saeed Afshar, Jonathan Tapson, and Andre Van Schaik. Emnist: Extending mnist to handwritten letters. IJCNN, 2017.
Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai. Exploiting shared representations for personalized federated learning. In ICMI,, 2021. 3, 8,
Luca Corinzia and Joachim M Buhmann.
Variational federated multi-task learning.
arXiv preprint
arXiv: (1t6://90000668, 2019. 17
Yin Cui, Menglin Jia, Tsung- Yi Lin, Yang Song, and Serge Belongic. Class-balanced loss based on effective number of samples. In CVPR, 2019. 2,20
Yuyang Deng, Mohammad Mahdi Kamani, and Mehrdad Mahdavi. Adaptive personalized federated learning. arXiv preprint arXiv:2003.1346/ 2020a. 17
Yuyang Deng, Mohammad Mahdi Kamani, and Mehrdad Mahdavi. Distributionally robust federated averaging.
NewrlPS, 33, 2020b. 17
Canh T Dinh, Nguyen H Tran, and Tuan Dung Nguyen. Personalized federated learning with moreau envelopes. In NeurIPS, 2020. 1, 2, 3, 8, 17, 24, 29
Moming Duan, Duo Liu, Xianzhang Chen, Renping Liu, Yujuan Tan, and Liang Liang, Self-balancing federated learning with global imbalanced data in mobile systems. IEEE Transactions Parallel and Distributed Systems, 92(5)19977,, 2020. 17
An Evgeniou and Massimiliano Pontil. Multi-task feature learning. In NeurlPS, 2007. 17
Theodoros Evgeniou and Massimiliano Pontil, Regularized multi-task learning. In KDD, 2004. 17
Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Personalized federated learning: A meta-learning approach. In NewrlPS, 2020. 1, 17, 23, 29, 31
Muhammad Ghifary, Bastiaan Kleijn, Mengjic Zhang, and David Balduzzi. Domain generalization for object recognition with multi-task autoencoders. In ICCV, 2015. 17
Boqing Gong, Yuan Shi, Sha, and Kristen Grauman. Geodesic flow kernel for unsupervised domain adaptation. In 2012 IEEE conference on computer vision and pattern recognition, pp. 2066-2073. IEEE, 2012. 5
Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmentation. In CVPR, 2019. 18
David Ha, Andrew Dai, and Quoe Lc. Hypemeeworrs. In ICLR, 2017. 2, 7, 17, 18, 22
Farzin Haddadpour Mehrdad Mahdavi. On convergence local descent methods federated learning. arXiv preprint arXiv: 14425, 2019.
Filip Hanzely and Peter Richtárik. Federated learning of mixture of global and local models. arXiv preprint arXiv:2002.0553 2020. 4, 17
Filip Hanzely, Slavomír Hanzely, Samuel Horváth, and Peter Richtárik. Lower bounds and optimal algorithms for personalized federated learning. In NeurlPS, 2020. 1, 3, 4, 17
Chaoyang He, Murali Annavaram, and Salman Avestimehr. Group knowledge transfer: Federated learning of large cnns at the edge. NeurlPS, 2020. 16,
Haibo He and Edwardo Garcia. Learning from imbalanced data. IEEE Transactions on knowledge and data engineering. 2169):126312284 2009. 2, 3,
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016. 31
Tzu-Ming Harry Hsu, Hang Qi. and Matthew Brown. Measuring the effects of non non-identical data distribution for federated visual classification. arXiv preprint (XXix1909.06333 2019. 3. 16,
Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Federated visual classification with real-world data distribution. In ECCV, 2020. 5, 6, 9, 27
Yutao Huang, Lingyang Chu, Z. Zhou, Lanjun Wang. J. Liu,, Jian Pei, and Yanxin Zhang. Personalized cross-silo federated learning on non-iid data. In AAAI, 2021. 17

---

Published as a conference paper at ICLR 2022
Laurent Jacob, Francis Bach, and Jean-Philippe Vert. Clustered multi-task learning: A convex formulation. In NewrIPS, 2009, 17
Nathalic Japkowicz. The class imhalance problem: Significance and strategies. In Proc. of the Int'l Conf. on Artificial Intelligence, 2000. 5
Yihan Jiang, Jakub Konecný, Keith Rush, and S. Kannan. Improving federated learning personalization via model agnostic meta learning, ArXiv, abs/1909. 12488, 2019. 17
Justin M Johnson and Taghi M Khoshgoftaar. Survey on deep learning with class imbalance. Journal of Big Data, 6(1):27, 2019. 5
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mchdi Bennis, Arjun Nitin Bhagoji, Keith Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems federated learning. arXiv preprint arXiv: 2019.
Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Alhert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition. ICI.R, 2020. 5, 20
Sai Praneeth Karimireddy, Martin Jaggi. Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian U Stich, and Ananda Theertha Suresh. Mime: Mimicking centralized stochastic algorithms in federated learning. arXiv preprint arXiv:2008. 03606, 2020a. 3, 16, 17
Sai Pranceth Karimireddy, Satyen Kalc, Mchryar Mohri, Sashank Reddi, Schastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In ICMI,, 2020b. 5, 6, 8, 16, 18, 29, 30
Khaled, K Mishchenko, and P Richtárik. Tighter theory for local sgd identical and heterogeneous data. In AISTATS, 2020.
Salman Khan, Munawar Hayat, Mohammed Bennamoun, Ferdous A Sohel, and Roberto Togneri. Cost- sensitive learning of deep feature representations from imbalanced data. IEEE transactions on neural networks and learning systems, 29(8):3573-356) 2017. 5, 20
M. Khodak, Maria- Florina Balcan, and Ameet Talwalkar. Adaptive gradient- based meta- leetalearming methods. In NewrlPS, 2019. 17
Jakub Konečný, H Brendan McMahan, Felix X Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency. arXiv preprint 1r000:1610 2016.
Lingjing Kong, Tao Lin, Anastasia Koloskova, Martin Jaggi, and Sebastian U Stich. Consensus control for decentralized deep learning. In JCML, 2021. 9
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Kulkarni, Milind Kulkarni, and Pant. Survey personalization techniques for federated learning. 2020 Fourth World Conference on Smart Trends in Systems, Security and Sustainability (WorldS4), pp. 794-797, 2020. 3, 17
Christoph H Lampert, Hannes Nickisch, and Stefan Harmeling, Attribute-based classification for zero-shot visual object categorization. TPAMI, 36(3):453 465, 2013. 18
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEFF, 86(1) )1:27882324, 1998. 7, 23,
Daliang Li and Junpu Wang. Fedmd: Heterogenous federated learning via model distillation. arXiv preprint arXiv:1910.03581, 2019. 4. 17
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smithy. Feddane: A federated newton-type method. In 2019 53rd Asilomar Conference on Signals. Systems, and Computers. 2019. 16
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Amect Talwalkar, and Virginia Smith. Federated optimization in heterogencous networks. In MLSys, 2020a. 3. 5, 6, 8, 16, 18, 23, 29, 30
Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. Ditto: Fair and robust federated learning through. In ICML, a021a. 1, 3, 4, 8, 17, 26, 29,
12

---

Published as a conference paper at ICLR 2022
Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang. and Zhihua Zhang. On the convergence of fedavg on non-iid data. ICLR, 2020b. 5, 16,
Xiaoxiao Li, Meirui JIANG, Xiaofei Zhang, Michael Kamp, and Dou. Fed(bn): Federated learning on [ features via local batch normalization. In ICLR, 2021b.
Yu Li, Tao Wang, Bingyi Kang. Sheng Tang, Chunfeng Wang, Jintao Li, and Jiashi Feng. Overcoming classifier imbalance for long-tail object detection with balanced group softmax, In CVPR, 2020c.
Paul Liang, Terrance Liu, Liu Ziyin, Ruslan Salakhutdinov,
Phiil-ph Morency. Think locally, act glohally: Federated learning with local and global representations. arXiv preprint arXiv:2001.0153,, 2020. 1, 3, 8, 17, 29
Xianfeng Liang, Shuheng Shen, Jingchang Liu, Zhen Pan, Enhong Chen, and Yifei Cheng. Variance reduced local sgd with lower communication complexity. arXiv preprint 1912:19912.12844 2019. 16
Tao Lin, Lingjing Kong, Sebastian U Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning. In NewrlPS, 2020. 3, 7, 16, 18, 24
Grigory Malinovskiy, Dmitry Kovalev, Elnur Gasanov, Laurent Condat, and Peter Richtarik. From local sgd to local fixed-point methods for federated learning. In ICMI., 2020. 3, 16
Yishay Mansour, Mehryar Mohri, Jae Ro, and Ananda Theertha Suresh. Three approaches for personalization with applications to federated learning. arXiv preprint arXiv:2002. 10619, 2020. 17
H Brendan McMahan, Fider Moore, Daniel Ramage, Seth Hampson, et al. Communication-effitii learning of deep networks from decentralized data. In A/STATS, 2017. 2, 7, 16, 19, 20, 23, 29, 30, 32
Mehryar Mohri, Gary Sivek, and Ananda Theertha Suresh. Agnostic federated learning, In /CML, 2019. 17
Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. Domain generalization via invariant feature representation. In ICML, 2013. 5, 17
Reese Pathak and Martin Wainwright. Fedsplit: An algorithmic framework for fast federated optimization. In NewrlPS, 2020. 16
Daniel Peterson, Pallika Kanani, and Virendra J Marathe. Private federated learning with domain adaptation. arXiv preprint arXtv:19/2.06733 2019. 17
Tomaso Poggio, Ryan Ritkin, Sayan Mukherjee, and Alex Rakhlin. Bagging regularizes. Technical report, MASSACHUSETTS INST OF TECH CAMBRIDGE ARTIFICIAL INTELLIGENCE LAB, 2002.
Sashank Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečný, Sanjiv Kumar, and Brendan McMahan. Adaptive federated optimization. In ICLR, 2021. 3,
Amirhossein Reisizadeh, Aryan Mokhtari, Hamed Hassani, Ali Jadbabaic, and Ramtin Pedarsani. Fedpaq: communication-effiienn federated learning methad with periodic averaging and quantization. arXiv preprint (r99v:1909.130// 2019. 16
Matthias Reisser, Christos Louizos, Efstratios Gavves, and Max Welling. Federated mixture experts, 2021. URL
not forum? Lorumedd-xrdddateBB00 17
Jiawei Ren, Cunjun Yu, Shunan Sheng. Xiao Ma, Haiyu Zhao Shuai Yi, and Hongsheng Li. Balanced meta-softmax for tailed visual recognition. In NewrIPS, 2020. 2, 5, 9, 18, 20, 21, 22
Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep
learning. In ICML, 2018. 8. 22
Schastian Ruder. An overview of multi-task learning in deep neural networks. arXiv preprint 17X.v:1706.000098 2017. 3, 17
Aviv Shamsian. Aviv Navon, Ethan Fetaya, and Gal Chechik. Personalized federated learning using hypernet- works. In ICML, 2021. 17
Jun Shu, Qi Xie, Lixuan Yi, Qian Zhao, Sanping Zhou, Zongben Xu, and Deyu Meng. Meta-weight-net Learning an explicit mapping for sample weighting. In NeurlPS, 2019. 22
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICI.R, 2015. 31
13

---

Published as a conference paper at ICLR 2022
Marina Skurichina and Robert PW Duin. Bagging for linear classifiers. Pattern Recognition, 31(7):909-930. 1998.
Virginia Smith, Chao-Kat Kai Chiang. Maziar Sanjabi, and Ameet S Talwalkar. Federated multi-task learning. In NewrlPS, 2017. 1, 2, 3, 4, 8, 17,
Scbastian Stich. Local sgd converges fast and communicates little. ICLR, 2019.
and Junjie Yan. Equalization Jingru Tan, Changhao Wang. Buyu Li, Quanquan Li, Wanli Ouyang, Changqing Yin, and
loss long-tailed object recognition. In CVPR, 2020. 18
TensorFlow team. Tensorflow convolutional neural networks tutorial. http:///wwwwtenssflloo...oo settorials/dcpp...o 2016. 23
Grant Van Hom Pietro Perona. devil the tails: Fine-grained classification wild. arXiv preprint UrXiv:/709.01450 2017. 18
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In CVPR, 2018. 18
Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris Papailiopoulos, and Yasaman Khazaeni. Federated learning with matched averaging. In ICLR, 2020a. 3, 16
Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H Vincent Poor. Tackling the objective inconsistency problem in heterogeneous federated optimization. In NeurlPS, 2020b. 3, 16
Kangkang Wang, Rajiv Mathows, Chloé Kiddon, Hubert Eichner, Franccoise Beaufays, and D. Ramage. Federated evaluation of on-device personalization. ArXiv, abs/1910.10220 2019. 3, 17
Lixu Wang, Shichao Xu, Xiao Wang, and Qi Zhu. Addressing class imbalance in federated learning. In AAAI, 2020c. 17
Yongqin Xian, Christoph H Lampert, Bert Schicle, and Zeynep Akata. Zero-shot learning- comprchensive evaluation of the good, the bad and the ugly. TPAMI, (19)):255-2225, 2018. 18
H. Xiao, K. Rasul, and Roland Vollgraf. Fashion-mnist. novel image dataset for benchmarking machine learning algorithms. ArXiv, abs/1708.007477, 2017.
Miao Yang. Akitanoshou Wong, Hongbin Zhu, Haifeng Wang, and Hua Qian. Federated learning with class imbalance reduction, arXiv preprint urXiv:2011. 2020. 17
Yao, Tianchi Huang, Rui-Xiao Zhang, Ruiyu Li, and Lifeng Sun. Federated learning with unbiased gradient aggregation and controllable meta updating. arXiv preprint 1rxiv:1910.082344 2019. 16
Han-Jia Ye, Hong- You Chen, De-Chuan Zhan, and Wei-Lun Chao. Identifying and compensating for feature
deviation in imbalanced deep learning. arXiv preprint arXiv:2001.01885, 2020. 5, 9, 18, 20, 21
Tao Yu, Eugene Bagdasaryan, and Vitaly Shmatikov. Salvaging federated learning by local adaptation. arXiv
preprint 2rXiv:2002.04720 2020. 3, 17
Honglin Yuan and Tengyu Ma. Federated accelerated stochastic gradient descent. In NeurlPS, 2020. 3, 16
Mikhail Yurochkin, Mayank Agarwal, Soumya Ghosh, Kristjan Greenewald, Trong Nghia Hoang, and Yasaman Khazaeni. Bayesian nonparametric federated learning of neural networks. In ICML, 2019. 3, 16
Edvin Listo Zec, Olof Mogren, John Martinsson, Leon René Sütfeld, and Daniel Gillblad. Federated learning using mixture of experts. arXiv preprint arXiv:2010.02055 2020. 17
Michael Zhang, Karan Sapra, Sanja Fidler, Serena Yeung, and Jose M. Alvarez. Personalized federated learning with first order model optimization. In ICLR, 2021. URL http:s/////ppeeeeeeeeeeeeeeeeeee id=
ehJeJJokcww. 1, 8, 17, 29, 30
Yu Zhang and Qiang Yang. A survey multi-task learning, arXiv preprint arXivv 14, 2017. 3, 17
Yu Zhang and Dit-Yan Yeung. A convex formulation for learning task relationships in multi-task learning, 2010.
17
Zhanpeng Zhang, Ping Luo, Chen Change Loy, and Xiaoou Tang. Facial landmark detection by deep multi-task learning. In European conference on computer vision, pp. 94-108. Springer, 2014. 3
14

---

Published as a conference paper at ICLR 2022
Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Federated learning with non-iid data, arXiv preprint
2018. 3, 22, 29, 30
Fan Zhou and Guojing Cong. On the convergence properties of step averaging stochastic gradient descent algorithm nonconvex optimization. arXiv preprint arXiv:1788.01011 2017.
Yanlin Zhou, George Pu, Xiyao Ma, Xiaolin Li, and Dapeng Wu, Distilled one-shot federated learning. arXiv preprint arXiv:2009, 07999, 2020. 16, 18
15

---

Published as a conference paper at ICLR 2022
Appendix
We provide details omitted in the main paper.
Appendix A: additional comparison to related work (cf. section and section 3 of the main paper).
Appendix B: additional details of FED- RoD (cf. section and section 4 of the main paper).
Appendix C: details of experimental setups (cf. section 5 of the main paper).
Appendix D: additional experimental results and analysis (cf. section 3 and section of the main paper).
A COMPARISON TO RELATED WORK
A.1 FOLLOW-UP WORKS OF FEDAVG
Several recent works (Karimireddy et al., 2020a;b; Zhao et al., 2018) have shown that, with multiple steps of local SGD updates, the local model Wm would drift away from each other, leading to degenerated global model w that deviates from the solution of Equation of the main text.
One way to mitigate this is to modify the local training objective (cf. Equation 2 of the main text). For instance, FEDPROX (Li et al., 2020a) introduced a regularizer with respect to w
X
min Lm (w)
2
FEDDyN (Acar et al., 2021) further added a dynamic term based on the local model of the previous round w'
min C. (w) + (VLm (7mm(wn))) w) +
w|
These regularizers aim stabilize local training and align the objectives among clients. Some other works did not change the objectives but introduced control variates or momentum to correct the local gradient (Karimireddy al., 2020a;b), designed new optimizer more suitable for decentralized learning (Malinovskiy et al., 2020; Pathak & Wainwright, 2020; Yuan & Ma, 2020).
It is worth mentioning. in most of these works, the empirical risk C. (w) still plays an important role in driving the local model update. Since L. (w) directly reflects the (non-IID) client data distribution, the learned local models are indeed strong candidates for personalized models.
A.2 GENERIC FEDERATED LEARNING
FEDAVG (MeMahan et al., 2017) is the standard algorithm, which involves multiple rounds of local training and global aggregation. Many works have studied its convergence (Haddadpour & Mahdavi, 2019; Khaled al., 2020), robustness (Bonawitz al., 2019), communication (Konečný et al., 2016; Reisizadeh et al., 2019), especially for non-llI) clients (Li et al., 2020a;b; Zhao et al., 2018). Many other works proposed to improve FEDAVG. terms of global aggregation, (Wang et al., 2020a; Yurochkin et al., 2019) matched local model weights before averaging. (Chen & Chao, 2021; He al., 2020; Lin et al., 2020; Zhou al., 2020) replaced weight average by model ensemble and distillation. (Hsu et al., 2019; Reddi et al., 2021) applied server momentum and adaptive optimization to improve the global model update. In terms of local training, (Liang al., 2019; Malinovskiy et al., 2020; Pathak & Wainwright, 2020; Yuan & Ma, 2020) improved the optimizer. To reduce local models' drifts from the global model, (Zhao et al., 2018) mixed client and server data in local training: FEDPROX (Li et al., 2020a), FEDDANE (Li et al., 2019), and FEDDyN (Acar et al., 2021) employed regularization toward the global model; SCAFFOLD (Karimireddy et al., 2020a) MIME (Karimireddy et al., 2020b) leveraged control varieties and/or server statistics correct local gradients; (Wang et al., 2020b; Yao et al., 2019) modified the local model update rules. For most of them, the empirical risks clients' data the major forces to drive local training.
We also aim to reduce local models' drifts but via a different way. We directly bypass the empirical risks that reflect clients' data distributions. Instead, we apply objective functions in class-imbalanced
16

---

Published as a conference paper at ICLR 2022
learning (He & Garcia, 2009), which are designed to be robust to the change of class distributions. Our approach is different from (Duan et al., 2020; Wang et al., 2020c; Yang et al., 2020), which monitored tackled it at the clients. Our approach is also
and resolved class imbalance from the server while different from agnostic FL. (Deng et al., 2020b; Mohri et al., 2019), whose local training is still built on empirical risk minimization. The closest to ours is (Hsu al., 2020), which used traditional clas-imbalaanced treatment, re-weighting, to mitigate non-identical class distributions. We show that more advanced techniques can be applied further improve the performance, especially under extreme non-llI conditions where re- re-weighting is less effective. Moreover, our method is compatible with existing efforts like FEDDYN (Acar al., 2021) and SCAFFOLD (Karimireddy et al., 2020a) to boost the generic performance.
A.3 PERSONALIZED FEDERATED LEARNING
Personalized FL (Kulkarni et al., 2020) learns customized model for each client. Many approaches are based on multi-task learning (MTL. (Evgeniou & Pontil, 2007; 2004; Jacob et al., 2009; Ruder, 2017; Zhang & Yang. 2017; Zhang& & Yeung, 2010) leveraging the clients' task relatedness to improve model generalizability. For instance, (Smith et al., 2017) encouraged related clients to learn similar models; (Corinzia & Buhmann, 2019; Dinh et al., 2020; Hanzely & Richtárik, 2020; Hanzely et al., 2020; Li & Wang, 2019; Li et al., 202 regularized local models with a learnable global model, prior, or set of data logits. (Arivazhagan et al., 2019; Bui et al., 2019; Li et al., 2021b; Liang et al., 2020) designed the model architecture to have both personalized (usually the feature extractor) and shareable components. (Huang et al., 2021; Zhang al., 2021) constructed for each client an initialized model or regularizer based on learnable bases. Our approach is inspired by MTL as well but has several notable differences from existing works. First, we found that the global aggregation step in generic FL. already serves as a strong regularizer. Second, instead of learning for each client a personalized feature extractor (Bui et al., 2019; Liang al., 2020) or an entire independent model that can operate alone (Dinh et al., 2020; Hanzely et al., 2020; Smith et al., 2017), FED-ROD shares a single feature extractor among all clients, inspired by invariant risk minimization (Ahuja et al., 2020; Arjovsky et al., 2019) and domain generalization (Ghifary et al., 2015; Muandet et al., 2013). This reduces the total parameters to be learned and improves model's generalizability. Compared to FEDPER (Arivazhagan et al., 2019) and FEDREP (Collins et al., 2021) which also learned shared feature extractor, FED-ROD simultaneously outputs a single, strong global model to excel in the generic FL. setup.
Some other approaches are based on mixture models. (Agarwal et al., 2020; Deng et al., 2020a; Man- sour et al., 2020; Peterson al., 2019; Zec et al., 2020) (separately) learned global and personalized models and performed a mixture of them for prediction. (Reisser et al., 2021) learned a sets of expert models and used them to construct personalized models. Meta-learning is also applied to learn good initialized model that can be adapted to each client with few steps of local training (Chen et al., 2018; Fallah al., 2020; Jiang et al., 2019; Khodak et al., 2019).
Instead of designing specific algorithms for personalized FL, (Wang et al., 2019; Yu et al., 2020) showed that performing post-processing (e.g.. fine-tuning) to generic FL model (e.g.. t learned by FEDAVG) already leads to promising personalized accuracy. In this work, we further showed that, the local models wn, learned in FEDAVG and other generic FL algorithms are indeed strong personalized models.
We note that, while many personalized FL algorithms also produce global model, is mainly used to regularize construct personalized models but not for evaluation in the generic setup. In contrast, we learn models to excel in both the setups via a single framework without sacrificing either of them.
PFEDHN (Shamsian et al., 2021) also applies hypernetwrrks (Ha et al., 2017) but for very different purpose from FED-ROl Specifically, PFEDHN learns hypernetwork at the server to aggregate clients' model updates and produce their entire models for the next round. In contrast, we learn the hypernetwor locally to construct the personalized predictors, not the entire models, for fast adaptation to clients.
17

---

Published as a conference paper at ICLR 2022
A.4 AVERAGING MODEL WEIGHTS A REGULARIZER
In subsection 3.3, demonstrate that taking the average model weights indeed acts regularizer for local modely to improve their individual personalized performance.
In more traditional machine learning, the regularization effects of averaging multiple independently- trained models have been observed in some techniques like bagging (Poggio et al., 2002; Skurichina & Duin, 1998). Indeed, in several recent works of FL (Chen & Chao, 2021; He al., 2020; Lin et al., 2020; Zhou et al., 2020), the authors replaced weight average by bagging/model ensemble to improve the generic performance on the global test set. That is, they found that performing the model ensemble over clients' models can yield more robust predictions on the global test set than the global model, which is generated by averaging the client models' weights.
Here, we however study different regularization effect, in personalized FL on local test sets. As reviewed in subsection 3.1, personalized FL algorithms often impose a regularizer on the lo- cal/personalized models to overcome the fact that clients usually have limited data (please see Equa- tion 3 and Equation 4 and the surrounding text). What we claim is that even without such an explicit regularizer, the model weight average before local training (Equation 2) already serves as an im- plicit regularizer to the local models for their individual personalized performance, as we discussed in subsection 3.3 (Figure 3) and empirically verified in subsection D.2 and Figure 7.
A.5 SYSTEMATIC OVERHEAD
FED-ROD has similar computation cost, communication size, and number of parameters as FEDAVG. We discuss the difference between FED-ROD and existing generic FL methods from a system view. FEDPROX (Li et al., 2020a) proposes a proximal ferm to prevent client from diverging from the server model, which is more robust to the heterogeneous system. SCAFFOLD (Karimireddy et al., 2020b) imposes a gradient correction during client training. Maintaining such a correction term, however, doubles the size of communication. FEDDyN (Acar et al., 2021) resolves the communication cost issue by introducing a novel dynamic regularization. However, it requires all users to maintain their previous models locally throughout the FL process, which is not desired when users have memory and synchronization constraints.
A.6 CLASS- LEARNING
Class-imbalanced learning attracts increasing attention for two reasons. First, models trained under this scenario using empirical risk minimization perform poorly on minor classes of scarce training data. Second, many real-world data sets class-imbalanced by nature (Gupta et al., 2019; Van Hom & Perona, 2017; Van Horn et al., 2018). In this paper, we employ a mainstream approach, cost- sensitive learning (Cao al., 2019; Li et al., 2020c; Ren al., 2020; Tan et al., 2020; Ye et al., 2020), which adjusts the training objective to reflect class imbalance so as to train a model that is less biased toward major classes.
A.7 ZERO-SHIOT LEARNING
Our design choice of parameterizing the personalized prediction head with clients' class distributions reminiscent of zero-shot learning (Changpinyo al., 2016; 2017; 2020; Lampert et al., 2013; Xian et al., 2018), whose goal is to build an object classifier based on its semantic representation. The key difference that we build entire fully-connected layer for FL., not just single class vector. We employ hyperneworkss (Ha et al., 2017) for efficient parameterization.
B ADDITIONAL DETAILS OF FED-ROD
B. ADDITIONAL BACKGROUND (CE. SUBSECTION 3.1 OF THE MAIN PAPER)
In the generic federated learning (FL) setting, the goal is to construct a single "global" model that perform well for test data from all the clients. Let denote the parameters of the model, for
18

---

Published as a conference paper at ICLR 2022
classification problem whose label space is C, commonly used loss is the cross entropy,
(a,, yr w) = -log explgy(2; w))
(12)
Eoec (xp()e(z;w))
where (z; w) is the model's output logit for class
We note that, the concepts of global vs. local models and generic vs. personalized models should be confused. No matter which task (generic or personalized) an FL algorithm focuses on, as long as has the local training step, it generates local models; as long as has the global aggregation step (of the entire model), it generates a global model. For instance, FEDAVG (McMahan et al., 2017) aims for generic FL but creates both the global and local models.
B.2 OVERVIEW OF FED-ROD
For generic predictions, FED-ROD performs feature extraction z = f(x 0). followed by h (z; v)).
For personalized predictions, FED-ROD performs & = f(æ; 0), followed by ( Φb) + h² (z; Φm). The element-wise addition is performed at the logir level. That is, (e w) in Equation 12 can be re- re-written as
( (2 Φb) h zz; vb) + (z; Φm) Personalized model,
Generic model,
ge (0, sb, 2ml))
(13)
where = = f(x; 0) is the extracted feature.
The overall training process of FED-ROD iterates between the local training and global aggregation steps. In local training, FED-ROD aims to minimize the following objective
C CBR ((0, + ((0 2b. Hm
(14)
The empirical risk L. (wm = (0, 2b, bm is defined as P... Σ, E( wm), where Dm
((x;; (Pul is the training data of client We will introduce options of the balanced risk CBR in subsection B.3. We optimize Equation 14 via stochastic gradient descent (SGD). We updates 0, tb, and om in a single FEDAVG. For L. (10, 2b, &m }), do not derive gradients w.r.t. O and sb.
pass, which consumes almost the same computation cost
We emphasize that, according to subsection 4.2 of the main paper, the finally learned parameters of FED-RON RoD (linear) are 0, 4b. and { JM - We then plug them into Equation 13 for predictions.
19

---

Published as a conference paper at ICLR 2022
In algorithm 1 and algorithm provide pseudocode of our FED-ROD algorithm.
Algorithm 1: FED-ROL RoD (linear) (Federated Robust Decoupling)
Server input :initial global model parameter and 1²;
Client mn's input
:initial local model parameter Φi.. local step size 17, local labeled data Dms
for 1 to R. do
Sample clients SC f1,,.,,,H
Communicate and vb to all clients m E S:
for each client m E S in parallel do
Initialize 0
vb 4- vb, and ob.. 4- Dii
{0m 2i 2.n } t Client local training( (0, v, Jm J. Pm, 7): Communicate oth and im to the server; end
[Equation and Equation 9]
Construct 0 0-E = mEs Emies pm. i [Dml 2
[Pml
Construct 2/- = Emes Enn' cs (Pmll
end
Server output :0 and :b;
Client m''s output: (0'm 1oo, }
Algorithm 2: FED-ROD (hyper) (Federated Robust Decoupling)
Server input Client m'' input :local step size 1,, local labeled data Dm,
:initial global model parameter θ, &b and D;
for
do
Sample clients [1,...,NV]
Communicate 0, vb. and to all clients m E S: for each client m S in parallel do
Initialize 0
ab 1b, and v + Vs (0m., Vin. ] t- Client local training(()) (00, sb, v), Dm, r)):
[Equation 11]
Communicate O ii and vn to the server:
end
Construct 0 -Emes Emies t D 12
ml
Construct 2b = Emes Emies ml
Construct =E Emes Emies
[Dml V
15 ml
end
Server output :0, Tb, and D (for personalized model generation).
Client m output: : (0'm so v.
B.3 BALANCED RISK MINIMIZATION (BRM)
To learn generic model, standard federated learning (e.g., FEDAVG (MeMahan al., 2017)) aims to optimize [(w) in Equation 1 of the main paper. In theory, the overall objective L(w) is equal to the expected objective B[C.. (w)] for client m, if client m's data D,, are IID partitioned from D. Here, the expectation is over different Dm partitioned from D. In reality, L. could diverge from L due non-IlD partitions of the aggregated data D into clients' data. That is, E[Lm( * E[(mm that EE (w)) Re B[ (w)] x L(w).
L(w). We mitigate the non-lID situation by directly adjusting L. such
Essentially, C.. (w)) is the client's empirical risk, which could be different among clients if their class distribution Pm (y) are different. We, therefore, propose to turn the empirical risk L. into a class-balance risk LBR by replacing l in Equation 12 with class-balanced loss (Cao et al., 2019; Cui et al., 2019; Kang et al., 2020; Khan et al., 2017: Ren et al., 2020; Ye et al., 2020). The class-balanced loss attempts to make the learned model robust to different training class distributions, such that the learned model can perform well for all the test classes. In other words, the balsneb loss is designed with an implicit assumption that the test data will be training data may not be. Table summarizes some popular class- balanced losses. We also include some extensions with meta-learning. See subsection B.5.
even though the
20

---

Published as a conference paper at ICLR 2022
Table 5: Balanced risk and loss functions. We ignore normalization in C.. (wr). Red highlights modifications by the balanced losses. Blue highlights the terms learned with meta-learning (see subsection B.5).
Method
f( w)
L. L
exp expl9x (m:w))
Cross entropy
log Eeec capluo
E; 8( Ues us)
exp (m:w))
Dvee Nma f(Tri yes Nmius Σ,
(Hsu et al., 2020)
log Ecec (xp1)
LDAM (Cao et al,, 2019)
(u(=:
( tuned with validation)
log
Ecec,ett (xplac su(
-N NNii
CDT (Ye et al., 2020)
Nmm y ay (amzo))) mas
exp
(y tuned with validation)
log
E, w)w
Eoec exp
Nm (Nn, mire
17 Ta (a)ww)
BSM (Ren et al., 2020)
Nal.y exp ou (anpu)) Ecec No, (x:1))
(
fixed)
log
E. w);
Meta-BSH ( fixed,
No. (x: Eege Ain (xp(sc((:.))
fm.y, meta-leamnd)
log
Eg Um,v, E(xi, yes w)
Meta-BSM
Nm, 7m exp (a:w)) log Eoec Nm. The explaa. (=)w)
(7/m; 4m.s, meta-learned)
Ee Yon.n, Elai, w)
Table 6: # parameters ConvNets EMNIST/FMNIST and CIFAR-10/100
Module
EMNIST/EMNIST
CIFAR-10
CIFAR-
Feature extractor
92,646
1,025,610
1,025.610
Generic head Total
500
640
6400
93,146
1,026,250
1,032,010
Hypernetworks
8,160 (+8.8%)
20,800 (+2.0%) 104,000 (+10.0%)
One may wonder what if the global distribution is iaaslimmbalnnced? Will BRM still be beneficial to FL? In subsection D.5, we perform experiments to show that FED-ROD with BRM can still improve on FEDAVG since BRM seeks to learn every class well. Even though global distribution might be skewed, BRM provides novel alternative mitigate the non-lID problem by making every client optimize a more consistent objective as we discussed above. Designing better losses for BRM in FL will be interesting future work.
B.4 ON FEDERATED LEARNING FOR THE PERSONALIZED HRAD WITH HYPERNETWORKS
One drawback of existing personalized methods is that the personalized models only available for clients involved in training or with sufficient training data. When new clients arrive in testing, is it possible for the federated system to provide corresponding personalized models?
To this end, instead of learning specific prediction head &n. for each client m, we propose to learn a meta- model H" (am; (a v)) with a shared meta-parameter z. The input to H P is a vector a., E RCI, which records the proportion of class C in client m's data. The output of H is Omn for I.P In
21

---

Published as a conference paper at ICLR 2022
other words, HP can adaptively output personalized prediction heads for clients given their local class distributions am-
We implement the meta- modell Hº by hypernetwork (Ha et al., 2017), which can be seen lightweight classifier generator given am- This lightweight hypernetwork not only enables clients to collaboratively learn a module that can generate customized models, but also allows any (future) clients to immediately generate their own personalized predictors given their local class distribution am as input, even without training. We construct the hypernetwork by two fully-connected (FC) layers (with a ReLU nonlinear layer in between). Table 6 summarizes the number of parameters of each part in FED-ROD, Hypernetworks add only a small overhead to the original model.
B.5 EXTENSION WITH META-LBARNING FOR THE IMPROVED BSM LOSS
FED-ROD incorporates balanced loss to learn the generic model. Here we study more advanced way to derive such balanced loss with meta-learning Inspired by (Ren et al., 2018; Shu al., 2019) and the FL scenario proposed by (Zhao et al., 2018), we seek to combine the BSM loss and re-weighting as E; qm..y (BSM (zi y6; w)) where qm.y; is meta-learned with a small balanced meta dataset D meta provided by the server. (See Table 5 for a comparison.) The D meta should have similar distribution to the future test data. We implement this idea with the Meta- Weight Net (MWNet) (Shu et al., 2019) with learnable parameter Ç.
In addition, we notice that the original BSM loss (BSM = log E Lege N (x:))) has a hyper- parameter y which is be 1 via validation (Ren al., 2020). However, in federated learning can be hard to tune such a hyperparameter due to the large number of non-llD clients. Therefore, we propose to learn client-specific Y' with meta-learning for eBSM More specifically, given meta-learning rate 7) the meta-learning process involves the following iterative steps:
NI. m.n expLn, (xp(a((a:)))
1. Compute the Meta-BSM loss with a mini-batch B Dmn's Le., V( E B, compute (BSM (x, (x, W.))
2. Predict the example weights with Jn.y = MWNet (BSM (ar, y W Sm). V(E, y) B. JBSM
3. Re-weight the Meta-BSM loss: _ (w.)) Emny)e Qm.y "Tm (x) y/s Wm). and per- form one step of gradient descent to create duplicated model in Wm V w... m.
4. Computes the loss on the meta dataset D Dmeta using the duplicated model: CBR
(im)
Ducy
E(myy)Donn (BS (x. y, Wm)) followed by updating 7' + Y'm - 17 V 7m mn, Dass
CBR
and
Sm + Sm 1/ c... D. Duzu"
LBR
CBR 5. Update the model: Wm + Wm 77 V wm mm B Wm)
Throughout the federated learning process, You and Qm.y are dynamically learned with meta-learning for different clients and rounds.
Results of FED-ROD with Meta-BSM We sample 10 images for each class (only 0.2% of the overall training set) from the training set as the meta set. We compare to (Zhao et al., 2018) that concatenates the meta set clients' local data. The results in Table 10 and Table 12 are encouraging. With very small meta set, FED-ROD outperforms (Zhao et al., 2018) by 1% to 14% on accuracy across different settings, validating the importance of balanced losses and how to set them up dynamically via meta-learning
B.6 CONNECTION TO INVARIANT RISK MINIMIZATION GAMES (IRMG)
FED-ROD inspired by a recently proposed machine learning framework Invariant Risk Min- imization (IRM) (Arjovsky et al., 2019) and its extension Invariant Risk Minimization Games
(IRMG) (Ahuja et al., 2020).
Suppose that the whole dataset is collected from many environments, where data from each envi- ronment is associated with its characteristic, IRM introduces the concept of learning an invariant predictor. (Note that, in IRM the learner for an FL setting.) Given the training data partition, IRM aims to learn an invariant feature extractor f(x; 0) and classifier h(2; {b) that achieves the minimum risk for all the environments.
access data from all the environments; thus, it is not
22

---

Published as a conference paper at ICLR 2022
The concept of environments can be connected to clients' private local data in FL which are often non-lID. That is, given M environments, we can re-write IRM in a similar expression to Equation 7 in the main paper
M
min (IRM(O, b) 0.4
Σ Lm (b)
(15)
m=1
2²¹), s.t +b E arg min Lm (0,,)))) Vm E [M].
(16)
Unfortunately, IRM is intractable to solve in practice given the constraint that every environment relies on the same parameters (Ahuja et al., 2020). IRMG relaxes it by reformulating the classifier sb as an ensemble of environment-specifee classifiers (by averaging over model weights) Φ M Σm Pm:
M
min C [IRMG (0, σ) 0.Φ
Σ (0, Φ),
(17)
s.t Om E arg min pm'mm Cm, fm 1.M =1); Vn E [M.]
(18)
IRMG proved to optimize the same invariant predictor of IRM when it converges to the equilibrium in game theory, and it holds for a large class of non-linear classifiers. IRMG is solved through iterative optimization: (1) training the feature extractor with centralized data (i.e., aggregated data from all environments), (2) training the cetrromment-speccif classifiers Φm on the data of each environment Dm, and (3) updating the main classifier through weight averaging o- M Σm Φm.
We highlight the similarity between IRMG and FED-ROD: both are training a strong generic feature extractor and set of personalized classifiers. For predictions on data of client (environment) m in Equation 18, IRMG uses ý
37 (Ø = + Em 2m=2) z); FED-ROD's personalized model is y G 2: v/) + h (z: pm). We can connect IRMG to FED-ROD by re-writing its prediction as (z; Φ) =- Φ
M Em d' m and (z; Φn) == A ( Hm z), where o'm is the client m's model in the previous round/iteration of learning.
IRMG can not be applied directly to federated learning for the following reasons. First, centralized training of the feature extractor is intractable since clients' data are not allowed to be aggregated to the server. Second, to perform the iterative optimization of IRMG, the clients are required to communicate every step, which is not feasible in FL. due to communication constraints.
C IMPLEMENTATION DETAILS
Implementation. We adopt ConvNet (LeCun et al., 1998) following the existing works (Acar et al., 2021; McMahan et al., 2017; TensorFlow team, 2016). For EMNIST/FMNIST, it contains 2 Conv layers and 2 FC layers. The Conv layers have 32 and 64 channels, respectively. The FC layers are with 50 neurons as the hidden size and 10 neurons for 10 classes outputs, respectively. For (IFAR-10/10) it contains Cony layers and 2 FC layers. The Conv layers have 32, 64, and 64 channels, respectively. The FC layers are with 64 neurons as the hidden size and 10/100 neurons for 10/100 classes as outputs, respectively. To implement hypernetworks in FED-ROD, we use simple 2-FC ReLU network with hidden size 16 for EMNIST/FMINISTCCIFRR 100 and 32 for CIFAR-10.
We use standard pre-processing, where E.MIIT/FMNIST and CIFAR-10/100 images are normalized. EMNIST/FMIIST is trained without augmentation. The 32 x 32 CIFAR-IO1 1O/R-10/10000 images are padded 2 pixels each side, randomly flipped horizontally, and then randomly cropped back to 32 x 32.
We train every method for 100 rounds. We initialize the model weights from normal distributions. As mentioned in (Li et al., 2020b), the local learning rate must decay along the communication rounds. We initialize it with 0.01 and decay it by 0.99 every round, similar (Acar et al., 2021). Throughout the experiments, we use the SGD optimizer with weight decay 1e-5 and 0.9 momentum. The mini-batch size 40 (16 for EMNIST). each round, clients perform local training for epochs. We report the mean over five times of experiments with different random seeds.
For FEDPROX (Li et al., 2020a), the strength of regularization À selected from [1c- 2, 1c 1c-3,,3 1c For FEDDYN (Acar et al., 2021), the strength of regularization À is selected from [le-1, 1, 1e-3
suggested in (Acar et al., 2021). For PER FFDAVG (Fallah al., 2020), the meta-learning rate
23

---

Published as a conference paper at ICLR 2022
Table EMNIST FMNIST results G-FL accuracy and P-FL accuracy (%). x: methods with no G-FL models and we combine their P-FL models. g: official implementtion.
Dataset
EMNIST
FMNIST
Non IID
Writers
Dir(0.1)
Dir(O.3)
Test Set
G-FL
P-FL
G-FL
P-FL
G-FL
P-FL
Method / Model
GM
GM
PM
GM
GM
PM
GM
GM
PM
FEDAVO SCAFFOLD FEDDYN
07040006
140.11 81.040.14
83.44.15
97.0-10.5
84.5-10.17
19.74..19
97.1=0.11
03.0
80000052
85.0--0229
00.4
97.320.12
97.340.10
97 3:0000
83.2
83.240.16
86.1
86.120.17
91.5+0.19
MTL *
15.42-0.85
75.0:10.778
85.610.77
36. 16.1
36.010666
0733-0.75
53
0.70
53.410.69
78.34-8.0
LG--EEDAG "g
80.120.34
60.0:1::24
95...0.15
548.041
54510.44
0965
66.8
0.40
6688:0.42
84.40055
PEDPER
93.11:0.0
74410.25
79.910.22
PER-FEDAVG
05.1=0.24
97
60
82
06.7
PFEDME
96.3+0.11
%.00
1
74033
26
834
79.0
0.
79.04-035
45.4
DITTO
97.020005
97.0--0.6
47.00
51
81540.27
99440.41
83.240222
90 00.11:030
PEDFOMO
80.5
80.410.78
95.900677
34
34341.59
59.9-0.55
89.6:0.70
FEDREP 8
95.0100.8
95.110.11
97.51005
79.5
80.1
91.81029
80.6
80.5.1034
90.5
Local only
64.2+0088
85.9=0.69
85.0+
FED-ROD (lincar)
97.3--0.10 10
97.3+-009
97.5+000
23
8394021
02
86.3
0.16
86.300
94.5-0200
FED-ROD (hyper)
97.32-0.1
97310
97.50008
13
83910.18
92.9 026
86.3
0.17
86.310011
948
+ FRDDYN
97.42.008
97410.11
97..00111
85.9
85.710.22
95.3 0.36
87.5
0.26
87.5.1026
94.6
selected from [1c-2,1c-3 1c 1c-41. For PFEDME (Dinh al., 2020), the strength of regularization À is selected from [15, 20, 30). FED-ROD introduces no extra hyperparameters on top of FEDAVG.
For the generic and personalized heads of FED-ROD, we study using 1 4 FC layers but do see notable gain by using more layers. We attribute this to the well-learned generic features. Thus, for all our experiments on FED-ROD, we use a single FC layer for each head.
We run our experiments on four GeForce RTX 2080 Ti GPUs with Intel 19-9960X CPUs.
Evaluation. Both datasets and the non-IID Dirichlet simulation are widely studied and used in literature (Acar et al., 2021; Hsu et al., 2019; Lin et al., 2020). We use the standard balanced test set Dies for evaluation on generic FL (G-FL):
1 [Drest
G-FL accuracy:
Σ1(y: 1 (r,G).
(19)
where G here is the predicted label (i.e., arg max over the logits). For evaluation on personalized FL (P-FL), we still apply Drest but weight instances w.r.t. each client's 's class distribution:
1 M
I
E, Pan (3: (y:))(( = -.P)
P-FL accuracy:
(20)
E. (m)
We do so instead separating Drest into separate clients' test sets in order to avoid the variance caused by how we split test data (except the EMNIST dataset that each client has its own test set with the writer's styles). What we compute is essentially the expectation over the splits. We have verified that the difference of the two evaluation methods is negligible.
In Table of the main paper and some other tables in the appendix. we evaluate G-FL by an FL algorithm's ''s generic (usually the global) model, denoted G.. We evaluate P-FL by an FL algorithm's personalized models (or local models of a G-FL algorithm), denoted as PM. For P-FL, also report the generic model's 's accuracy following the literature to demonstrate the difference.
Due to the space limit of the main paper, we provide the standard deviations of the results of Table in Table 7, Table 8, and Table 9 here.
24

---

Published as a conference paper at ICLR 2022
Table 8: CIFAR-10 results in G-FL accuracy and P-FL accuracy (%). methods with no G-FL models and we combine their P-FL models. S: official implementation.
Non-IID
Dir(0.1)
Dir(0.3)
Text Set
G-FI.
P-FT.
G.FI.
P-F.
Method Model
GM
GM
PM
GM
GM
PM
57.6110.3
57.1..0.2
694410.41
58.71021
58.900.5
SCAFFOLD
61.240556
00.30
90.1 20.65
11.16
71:5=0000
8484067
FEDDYN
63,41::44
63.91:.38
92.420.5
72.540.37
73.220.39
85.4:0.44
MTL
12.14:3.55
12.74:3788
90.6=0.98
13.5+1189
13.7=199
80.24-1011
L.G FEDAVG
29.5.1.1.6
28.8.1..46
90.11061
46.7.10.45
46.2.10.7
82.41065
50.4::0.77
50.21:0.88
64.5:046
8491055
PER-PEDAVG
60.74:0.7
82.7+141
70.50088
8024-122
PEEDMR
50.6:10.66
50.71-0.88
76.620.60
62.110.0
61.7-057
70.540066
DITTO FEDFOMO
58.11:0.9
58 58.31.0.7
$6.8106
69.710.44
69
81.510.99
30.5+:7722
31.2-1.4
90.50085
45.341.00
16.1=166
83.440811
3
Local only
87.400.69
75.74:0.7
TED-ROD (linerr)
68.5-10.35
68.5.00335
92.7-0.54
76.9
76.8-033
86.4110.49
FED-ROD (hyper)
68.5.10.35
68.5.10.39
00.55
76.910.34
76.8
10335
86.810..55
+ FEDDYN
0.42
68.2.10.4
92.710.57
74.6
74.6
0.43
83.6.10.58
Table 9: CIFAR- 100 results in G-FL accuracy and P-FL accuracy (%). methods with no G-FL models and combine their P-FL models. S: official implementation.
Non-lID
Dir0.11)
Dir(0.3)
Test Set
G.FL
P.FL
G.FL
P.F.
Method Model
GM
GM
PM
GM
GM
PM
41.8:16.7
41.610.71
70.210.66
46.410.44
46.2-0.41
61740440
FEDPROX
417:1051
41.610.54
16.5-048
464-0.41
SCAFFOLD
42.31-03
42.17
10440.69
46.510.6
46.5-0665
6171066
FEDDYN
43.01...99
43.010.47
72.020.38
47.510.41
47.410.4
62.510.35
MTL
9,54655
9.345598
60.7+1455
10.8+8.1
10.7-6.78
49.04-2.33
LG-FEDAVG
23.5:12.50
23.41:2.14
66.71:100
34512.56
33.9-3301
55.4.1.11
FEDPER
17.61:0.55
37.6:10.63
71.0=0.55
62.41055
PER FEDAGG
39.0-10.89
66.61
44510.79
58.91130
PREDME
18.610667
38.5.0..5
63.01080
41410.71
41.1 1068
$3.40.700
DITTO
41.7100.6
41810.54
68.5
46410.4
4644
46.4
58.10.38
FEDPOMO
35.41-200
35.37
68.920.98
39.1
39
14
58.44115
40.710.51
40.71:0.5
Local only
40.0-11.03
32.5:10.99
RED-RRDD (linear)
45.9..0.44
45.4
72.2
18.50..3
48.5 0.38
60.3
FED-ROD (hyper)
45.9.1041
1..39
72.3 1.48
1..42
48. 0.45
62.51.0.22
+ FEDDYN
4..2:00.0
46.2:10.51
72,5
48.490.49
0.47
62.5
25

---

Published as a conference paper at ICLR 2022
Empirical risk
Regularization
30
1.00
Ditto ERM
FedAvg ERM
0.75
20
Ditto reg.
0.50
FedAvg reg.
10
0.25
Awm
50
100
0
25
50
75
100
Rounds
Rounds
Figure 7: Comparison of the empirical risk and regularization between personalized models of DITTO and local models of FEDAVG. The dataset is CIFAR-10, 10, with Dir(O.3).
D ADDITIONAL EXPERIMENTS AND ANALYSES
Here we provide additional experiments and analyses omitted in the main paper. We validate our claims in the main paper and the designs of our proposed FED-ROD via the following experiments:
subsection D.I: personalized models emerge from local training of generic federated learning (cf. subsection 3.2, subsection 3.3, and subsection 5.1 in the main paper).
subsection D.2: balanced risk minimization (BRM) improves generic-Fi -L performance (cf.
subsection 5.1 the main paper).
subsection D.3: the roles of FED-ROD's generic and personalized heads (ef. subsection 4.2 in the main paper).
subsection D.4: personalization with hypernetworks (cf. subsection 5.1 in the main paper).
subsection D.5: robustness to class-imbalanced global data.
subsection D.6: compatibility of FED-ROD with other G-FL algorithms (ef. subsection 5.1 in the main paper).
subsection D.7: comparison to personalized FL algorithms (cf. subsection 5.1 in the main paper).
subsection D.8: ablation studies and discussions on FED-ROD (cf. subsection 5.1 in the main paper).
D.1 PERSONALIZED MODELS EMERGE FROM LOCAL TRAINING OF GENERIC FEDERATED LEARNING
As mentioned in section 3 in the main paper, personalized FL algorithms usually impose an extra regularizer (ef. Equation 3 and Equation of the main paper) during local training, but do not re-initialize the local models by the global models at every round. In contrast, generic FL algorithms like FEDAVG do not impose extra regularization
re-initialize the local models at every round.
12 wm - w |2 (cf. Equation 3 and Equation of the main paper), for FEDAVG and state-of-the-ar personalized FL algorithm Ditto (Li et al., 2021a) at the end of each local training round. (Ditto does include the L2 regularizer in training the personalized models.) Ditto achieves a lower empirical risk (L.e, the first term), likely due to the fact that does not perform re-initialization. Surprisingly, FEDAVG achieves much smaller regularization term (i.c. the second term) than Ditto, even if it does not impose such a regularizer in training. We attribute this to the strong effect regularization by re- re-initialization:
Here in Figure 7, we monitor the two loss terms, Σm [Daal
[Tl L. (w,) and Σn, [2
mentioned in subsection 3.3 of the main paper, re- re-initialization is equivalent to setting the regularization coefficient as infinity. We note that, the reason that the regularization term of Ditto increases along the communication rounds because time the global model w
26

---

Published as a conference paper at ICLR 2022
FMNIST
CIFAR-10
70
FedAvg
FedAvg
80
FrdProx
FedProx
60
75
SCAFFOLD
SCAFFOLD
50
70
redDyn
FedDyn
65
FodRoD
D
60
30
Client
Client
Figure 8: The G-FL accuracy by the local models wm difffrent generic methods. There are 100/20 clients for FMNIST/CIFAR 10, respectively, Both datasets use Dir(O.3).
FMNIST
CIFAR-10
0.20
1.0
FodProx
FodAvg
FedDyn Fed-RcD
Gradient Variance
0.15
SCAFFOLD
Gradient Jariance
FedProx
redDyn
SCArTOLD
0.6
0.10
FedAvg
m
Fed RoD
0.4
0.05
0.2
20
40
60
80
100
20
40
60
100
Rounds
Rounds
Figure 9: Variances local model updates w.r.t. the global model. For both datasets, we use Dir(O.3).
is updated, it moves sharply away from the local model wm- Thus, even if the regularization term added into local training, cannot be effectively optimized. This analysis suggests that the local models of generic FL algorithms are more regularized than the personalized models of personalized FL algorithms. The local models generic FL algorithms thus strong candidates be evaluated in the personalized FL setting.
D.2 BALANCED RISK MINIMIZATION (BRM) IMPROVES GENERIC-FL PERFORMANCE
To understand why FED-ROD outperforms other generic methods the G-FL accuracy, we visualize each local model u.... 's G-FL accuracy after local training in Figure 8 (both datasets with Dir(O.3)). Methods rely on ERM suffer as their local models tend diverge. Figure further shows that the variances of local weight update Awn, Wm across clients smaller for FED-RoD, which result from a more consistent local training objective.
In Figure 10, we further compare the G-FL accuracy among FEDAVG, FED- RoD with the original BSM loss, and RED-ROD with the Meta-BSM loss introduced subsection B.5 along the training process (i.e., training curve). The local models of FEDAVG tend to diverge from each other due the non-lID issue, resulting in high variances and low accuracy of G-FL. The global aggregation does improve the G-FL accuracy, validating its importance in federated learning. The local training in FED-ROD (BSM) not only leads to better global model, but also has smaller variances and higher accuracy for the local models (as their objectives are more aligned). With the help of meta dataset and meta-lcarning, FED-ROD (Meta-BSM) yields even better G-FL performance for both global models and local models, and has much smaller variances among local models' performance, demonstrating the superiority of using meta-learning to learn a balanced objective.
D.3 THE ROLES OF FED-ROD's GENERIC AND PERSONALIZED HEADS
To demonstrate that FED-ROD's two heads learn something different, we plot in Figure 11 every local model's 's generic prediction and personalized prediction on its and other clients' data (i.e., P-FL accuracy). The generic head performs well in general for every client's 's test data. The personalized head could further improve for its own data (diagonal), but degrade for others' data.
27

---

Published as a conference paper at ICLR 2022
90
sug.
avg]
Tedeftut MetwiISM (ether sugg
Pwo-loff NatwecM (belorw wags
Giobal Accurdcy
70-
M
50-
30-
Rounds
Figure 10: Training curves of different FL algorithms. We show the G-FL accuracy along the training process, using models before (i.e., local models) and after global aggregation. The dataset is CIFAR-10 Dir(0.3).
90
Client model ID
Local test set ID
Figure 11: P-FL accuracy of G-head (left) and G-head P-head (right) using the local models of FED-ROD. evaluated each client's test data. Here we CIFAR-IE Dir(O.3) with clients.
D.4 PERSONALIZATION WITH HYPERNETWORKS
FED-ROD (hyper) learns the personalized head with hypernetworks introduced in subsection B.4. The goal is to learn hypernetwork such that it can directly generate personalization prediction head given client's 's class distribution, without further local training. Figure 12 shows the training (convergence) curves on CIFAR- 10 Dir(..3). The hypernetwok (globally aggregated, before further local training) can converge to be par with that after local training. In the main paper (cf. Figure 6), we also show that servers as a strong starting point for future clients - it can generate personalized models simply with future clients' class distributions. That is, the clients may not have labeled data, but provide the hypernetwork with their preference/prior knowledge. It can also be used as the warm-start model for further local training when labeled data are available at the clients.
Table provides the P-FL results for the new 50 clients studied in subsection 5.1 and Figure 6 in the main paper. Except for FED-ROD (hyper), the accuracy before local training obtained by the global model. The best personalized model after local training is selected for each client using validation set. FED-ROD (hyper) notably outperforms other methods before after local training.
D.5 CLASS-IMBALANCED GLOBAL DISTRIBUTIONS
In the real world, data frequency naturally follows a long-tailed distribution, rather than a class- balanced one. Since the server has no knowledge and control about the whole collection of the clients' data, the clients data may collectively be maass-imllanned. This adds an additional challenge for the server to learn fair and blassbbalanced model. We follow the setup in (Cao et al., 2019) to transform FMNIST and CIFAR- 10 training sets into iaas-iibalancee versions, in which the sample sizes per class follow exponential decay. The imbalanced ratio (IM) is controlled as the ratio between
28

---

Published as a conference paper at ICLR 2022
100
80
P-FL P-FLACC Acc
GM
PM (hype) after local training
40
PM (hype) before local training
20
40
60
80
100
Rounds
Figure 12: P-FL accuracy of hypernetwork before and after local training.
Table 10: Class-imbalanced global training distribution. methods with no global models and we combine their P-FL. models. Gray rows: meta-learning with 100 labeled server data.
Dataset
FMNIST
CIFAR-10
Non-lID/ Imbalance Ratio
Din(0.3). IM10 (Dir(0.3), IM100 Dir(0.6). IM10
[Dir(0.6). IM100
Test Set
IG-FL
P-FL
[G-FL]
P-FL
G-FL|
P-FL
[G-FL]
P-FL
Method Model
GM
|GM
PM
GM
|GM
PM
GM
GM
PM
GM
GM
FEDAYG (McMahan et al. 2017)
80.2 80.2 85.1
71.6
71.5
86.9
50.9
50.2
76.5
40.1
40.0
78.2
FEDPROX (Li et al., 2020a)
81.0
81.0 82.3
70.4
70.2
87.0
58.6
58.7
76.5
37.6
38.0
76.6
SCAFFOLD (Karimireddy ot al., 2020b)
81.1
81.1 82.2
72.0
71.8
86.9
58.7
58.7
76.6
38.4
38.4 77.6
FEDDYN (Acar et al., 202[)
83.3
83.2
86.4
77.2
77.1
87.5
62.5
62.3
80.4
46.6
46.5
80.9
MTL (Smith et al., 2017)*
45.3 45.4
75.61
47.7
47.5
77.5
17.9
18.2
63.4
14.4
14.9
65.9
LG-FEDAVG (Liang et al., 2
62.8 62.4
82.4
74.0
74.1
83.2
31.5
31.5
62.8
24.9
24.8
66.3
PER-FEDAVG (Fallah et al., 2020)
80.1
82.5
72.0
78.5
46.3
77.2
31.7
74.3
PFEDME (Dinh et al., 2020)
78.9 78.9
81.6
69.3
69.2
71.6
46.2
46.2
54.2
31.7
31.8
50.6
DIrTO (Li el al., 2021a)
81.0 81.0
83.7
71.8
71.6
86.5
$1.0
50.9
73.1
40.3
40.2
75.4
FEDFOMO (Zhang et al., 2021)*
65.5 65.2
89.5
64.5
64.4
90.1
42.7
42.6
76.6
23.6
23.8
76.7
Local only
76.1
79.8
72.1
74.5
RED-ROD (BSM) RED-ROD (BSM) FEDDYN
81.3 81.3
89.5
76.8
76.8
89.8
63.3
63.3
80.1
48.3
48.3
81.1
84.5
84.5
80.0
79.9
88.6
65.5
65.5
79.7
49.6
81.0
FEDAYG + Meta (Zhao et al., 2018)
80.5
80.5
85.
71.8
71.8
86.9
51.5
51.3
77.4
40.4
40.2
84.0
FED-ROD (Meta-BSM)
86.5 86.5
90.2
82.5
82.5
90.9
72.3
72.1
82.1
61.9
60.9
83.5
sample sizes of the most frequent and least frequent classes. Here we consider IM= 10 and IM= 100. The generic test set remains dlaasballanced.
Table 10 shows that FED-ROD remains robust on both generic accuracy and client accuracy con- sistently. We see that FEDDYN also performs well, especially on FMNIST of which the setup has more clients (100) but a lower participation rate (20%). By combining FEDDyN with FED-ROD, achieve further improvements.
Essentially, the generic FL methods (except for RED-ROD) are optimizing toward the overall class- imbalanced distribution rather than the balanced distribution. In Table 11, we further examine the G-FL accuracy training set. FED-ROD still outperforms other methods, demonstrating that FED-ROD learns a robust, generic, and strong model.
class- dlas-immallnced test set whose class distribution is the same the global
D.6 COMPATIBILITY OF FED-ROD WITH OTHER G-FL ALGORITHMM
As mentioned in the main paper, other G-FL algorithms like FEDDYN (Acar et al., 2021) can be incorporated into FED-ROD to optimize the generic model (using the balanced risk). We show the results in Table 12, following Table 1 of the main paper. Combining RED-ROD with SCAF- FOLD (Karimireddy et al., 2020b), FEDDYN (Acar et al., 2021), and FEDPROX (Li ct al., 2020a) can lead to higher accuracy than each individual algorithm along in several cases.
29

---

Published as a conference paper at ICLR 2022
Table 11: G-FL accuracy on dlass-imbalance test data. Here we use CIFAR-IU Din(..3).
Method
IM10
IM100
FEDAYG (McMahan et al., 2017)
61.8
73.0
FEDPROX (Li et al., 2020a)
63.1
72.9
SCAFFOLD (Karimireddy et al., 2020b)
65.4
70.4
FEDDYN (Acar et al., 202I)
68.6
73.3
RED
71.9
76.0
Table 12: Main results in G-FL accuracy and P-FL accuracy (%), following Table 1 of the main paper. FED-ROD is compatible with other generic FL methods.
Dataset
FMNIST
CIFAR-10
Non-IID
Dir(O.1)
Dir(O.3)
Din(O.1)
Dir(0.3)
Test Set
IG-FL
P-FL
[G-FL]
P-FL
|G-FL|
P-FL
[G-FL
P-FL
Method/ Model
GM
|GM
PM
GM
[GM
PM|
GM
|GM
PM
GM
|GM
PM
FEDAVG (McMahan et al., 2017)
81.1
81.0 91.5
83.4
83.2 90.5
57.6
57.1 90.5
68.6
69.4
FEDPROX (Li et al., 2020a)
82.2
82.3
91.4
84.5
84.5
89.7
58.7
58.9
89.7
69.9
69.8
84.7
SCAFFOLD (Karimireddy et al., 2020b)
83.1
83.0 89.0
85.1
85.0 90.4
61.2
60.8 90.1
71.1
71.5 84.8
FEDDYN (Acar et al., 202
83.2
83.2
90.7
86.1
86.1
91.5
63.4
63.9
92.4
72.5
73.2
85.4
FED-ROD (linear)
83.9
83.9
92.7
86.3
86.3
94.5
68.5
68.5
92.7
76.9
76.8
86.4
FED-ROD (hyper)
83.9
83.9 92.9
86.3
86.3 94.8
68.5
68.5
92.5
76.9
76.8
86.8
FEDPROX SCAFFOLD
83.3
83.3 93.8
85.8
85.7
92.2
70.6
70.5 92.5
74.5
74.5
85.7
84.3
84.3 94.8
88.0
88
94.7
72.0
71.8 92.6
77.8
77.7
86.9
FEDDYN
85.9
85.7
95.3
87.5
87.5
94.6
68.2
68.2
92.7
74.6
74.6
85.6
FEDAVG + Meta (Zhao et al., 2018)
83.1
83.1
91.5
84.4
84.3
90.5
58.7
58.9
90.5
69.2
69.2
85.3
FED-ROD BSM)
86.4
86.4
94.8
89.1
89.1
94.8
72.5
72.5
92.8
80.1
80.1
86.6
D.7 COMPARISON TO PERSONALIZED FL ALGORITTMS
From Table in the main paper and Table 10, the personalized FL algorithms are usually outperformed by local models of generic FL algorithms in terms of the P-FL accuracy (i.e., the PM column). The gap is larger when client data are more IID, especially for P-FI. methods whose personalized models do not explicitly rely on weight averaging of other clients' models (e.g.. MTL, LG-FEDAVG, and PFEDME). Some P-FL methods can not even outperform local training alone. A similar observation is also reported in FEDFOMO (Zhang et al., 2021). These observations justify the benefits of FL that similar clients can improve each other by aggregating a global model and updating it locally, while the benefits might decay for very dissimilar clients.
To further demonstrate the effect of building a global model and re-initialing the local/personalized models using it (cf. section 3 in the main paper), investigate DrtTO (Li et al., 2021a), la), state-of- the-art personalized FL. algorithm. We found that DITTO learns two local models. One of them is used to build the global model exactly like FEDAVG. The global model is then used to regularize the other local model (cf. Equation in the main paper), which is used for personalized prediction. To differentiate these two local models, personalized model (PM). We note that, the PM model is kept locally and is never re- re-initialized by the global model. In Table 13, we show the P-FL accuracy using the LM and PM models. The LM model trained in the same way as FEDAVG (with re-initialization) surprisingly outperforms the PM model.
call the former the local model (LM), and the latter the
We further replicate the experiments in (Li et al., 2021a) on robustness against adversary attacks in Table 14. Besides comparing LM and PM, we also evaluate the global model GM for P-FL accuracy. With out adversarial attacks, the LM model outperforms the PM model. However, with
Table 13: The P-FL accuracy by the two local models of DITTO (Li ct al., 2021a).
Method
FMNIST
CIFAR-10
Dir(O.1)
Dir(0.3)
Dir(O.1)
Dir(0.3)
PM LM
89.4 90.8
90.1 90.6
86.8 90.8
81.5
86.2
30

---

Published as a conference paper at ICLR 2022
Table 14: DITTO with adversary attacks. We report the averaged personalized accuracy on benign clients.
Attack
PM
LM
GM
None
94.2
94.7
91.7
Label poisoning
93.6
54.5
84.8
Random updates
93.2
88.7
Model replacement
63.6
49.8
42.2
Table 15: FED-ROD on CIFAR-I0, Dir(O.3).
Test Set
G-FL
P-FL
Network
GM
GM
PM
ConvNet (LeCun al., 1998)
76.9
76.8
86.8
VGGII (Simonyan & Zisserman, 2015)
82.2
82.1
88.2
ResNet8 (He et al., 2016)
80.3
80.0
86.6
ResNet20 (Hc et 2016)
84.0
83.5
88.5
adversarial attacks, the PM model notably outperforms the other models. We surmise that, when there are adversarial clients, the resulting generic model will carry the adversarial information; re-initializing the local models with it thus would lead degraded performance.
D.8 ADDITIONAL STUDIES AND DISCUSSIONS
Different network architectures. FED-R RRD can easily be applied to other modern neural network architectures. In Table 15, we show that FED-ROD can be used with deeper networks.
FED-ROD is not merely fine-tuning. FED-ROD is not merely pre-training the model with BSM and then fine-tuning it with ERM for reasons. First, for FED-ROD (linear), the P-head is learned dynamically with the updating feature extractor across multiple rounds. Second, for FED-ROD (hyper), the hypernetwork has to undergo the local training and global aggregation iterations over multiple rounds. In Table 2 of the main paper, we report the fine-tuning baseline. On CIFAR-I0 Dir(O.3), it has 84.5% for P-FL (PM), lower than 86.4% and 86.8% by FED-ROD (linear) and FED-ROD (hyper). Note that, hypernetorrss allow fast adaptation for new clients.
Comparison to the reported results in other personalized FL papers. Existing works usually report FEDAVG's personalized performance by evaluating its global model (i.e., the GM column in Table 1 of the main paper). In this paper, we evaluate FEDAVG's ' local model wm.. (i.e., the PM column in Table of the main paper), which is locally trained for epochs. We gap between these two models. In (Fallah et al., 2020), the authors investigated a baseline "FEDAvG + update", which fine-tunes FEDAVG's global model w with only few mini-batches for each client. The resulting personalized models thus capture less personalized information than wm in FEDAVG. For fair comparison, we also strengthen PER-FEDAYG (Fallah al., 2020) by updating with more epochs.
huge performance
Effects of local sample size to P-FL performance In subsection 3.2 and Table 1, we show that local models of generic FL algorithms are strong personalized models. Indeed, the local sample size is an important factor in the P-FL performance. If a client has enough training samples, training its own model (the local only baseline) can already be strong without any federated learning. On the other hand, when each client does not have enough samples to train good model on its own. will be crucial have generic model learned from federated learning as the starting point of personalization.
To confirm our observation when clients have insufficient samples, we further conduct the following experiments. First, we enlarge the number of clients for CIFAR-10 10 and FMNIST experiments by five times. That is, each client's ''s data size becomes one-fifth on average. Second, we point out that the experiments CIFAR- 100 in Table 1 are with 20 clients. CIFAR-100 has the same total number of training images as CIFAR- 10 but with 10 times more classes. other words, the number of images per class is one-tenth. Table 16 shows the results: all the experiments are based on Dir(O.3). Even when the local only models perform worse in P-FL, the local models of FEDAVG still perform on a par with personalized FL. algorithms like FEDPER, and FED-ROD can still achieve the best P-FL.
31

---

Published as a conference paper at ICLR 2022
Table 16:
Main results in G-FL accuracy P-FL accuracy (%), following Table of the main paper.
FED-ROD is compatible with other generic FL methods.
Dataset
FMNIST CIFAR-IO CIFAR-I00
Test Set
[G-FL| P-FL [G-FL
P-FL
[G-FL]
P-FL
Method / Model
GM [GM PM
GM
|GM
PM
GM
GM
PM
Local only
72.91
76.91
32.5
FEDAYG (McMahan et al., 2017)
78.1
77.9
85.7
64.2
64.0
77.4
46.4
46.2
61.7
FEDPER (Arivazhagan et al., 20[9)
72.5
72.4
85.5
57.6
55.9
78.0
40.3
40.1
62.5
FED-ROD (hype)
82.6 82.6 90.1
72.7
72.7 82.7
48.5
48.5 62.5
accuracy. We attribute the superior personalized performance by FEDAVG and FED-ROD to the implicit regularization discussed in subsection 3.3.
We also want to point out that, even if each client has insufficient data, the P-FL performance of local only may still have higher accuracy than the GM of FEDAVG on the personalized accuracy, especially when the non-IID condition becomes severe (e.g., Dir (0.1)). When the non-llD condition is severe, harder to train a single GM model to perform well in the personalized setting.
32


Reviews:
[{"review_id": "va8X0sY88Tq", "reviewer": "Reviewer_ZY3s", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The problem is clearly stated and the algorithm is well-organised, also easy to understand and follow;\n2. The problem itself is interesting, non-iid and imbalance is a key challenge in federated learning and real-world setting. Also, increase both local and global performances should be of interest to the community;\n3. The experiments are well designed and comprehensive;\n4. The experimental results provided in the paper do achieve a state-of-the-art on both generic and personalized setting;\n5. Related works are properly cited.\n \nWeaknesses:\n1. Since no code is provided, it is useful to include more training details, for example, training hyperparameters for readers to better utilize the proposed framework;\n2. Also for the experiments, how many repeat trials did you run for a experiment? It is important to also include the error bar in the results;\n3. Lack of theoretical guarantees on the proposed framework.", "summary_of_the_paper": "This paper focuses on a challenging problem in federated learning setting where data across different local workers are imbalanced, and most of the current work in the community are optimizing either generic performance or personalized performance. This paper proposes a new framework, Fed-RoD, which combines both global objective and local objective, to achieve that the trained model is suitable for both generic and personalized task. The main contributions are:\n1. Focuses on an interesting problem which aims to increase both local and global performance in imbalanced federated learning setting;\n2. Designed a two-loss, two-predictor framework to achieve the goal;\n3. Based on the extensive experiments, the performance achieves state-of-the-art on both sides.", "summary_of_the_review": "This paper proposes a new framework aiming to increase generic and personalized performance simultaneously in the imbalanced federated learning setting, the proposed algorithm is simple yet efficient according to the experimental results. Theoretical analysis are somehow insufficient, and some minor fix or explanations are needed to better understand the experiments. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "N0YYC2jA8-R", "reviewer": "Reviewer_KbQt", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper proposes Federated Robust Decoupling (FED-ROD) to excel on both generic FL and personalized FL. With extensive results, it illustrates that strong personalized models emerge from the local training step of generic FL algorithms, and show that class-balanced objectives are effective for improving the generic FL performance when clients have different class distributions. Lastly, the paper demonstrates that FED-ROD enables zero-shot adaptation and much effective fine-tuning for new clients.\n\nweakness: \n1. The paper overall is a bit hard to follow. It would help to possibly re-structure tables, figures with the main body.  \n2. In the experiment part, it would be stronger to expand and provide summary and insights from tables and figures in the body. For example in \"Ablation Analysis\",  \n2. conclusion is weak and would requires some work to summarize the contribution. ", "summary_of_the_paper": "The paper proposes Federated Robust Decoupling (FED-ROD) to excel on both generic FL and personalized FL. With extensive results, it illustrates that strong personalized models emerge from the local training step of generic FL algorithms, and show that class-balanced objectives are effective for improving the generic FL performance when clients have different class distributions. Lastly, the paper demonstrates that FED-ROD enables zero-shot adaptation and much effective fine-tuning for new clients.", "summary_of_the_review": "Technical content of the paper is overall pretty strong and idea seems pretty significant and novel. However, overall the paper is hard to follow. It would benefit and stronger with better organization (as pointed out in the weakness in the above). ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "G2HCSJUGPR1", "reviewer": "Reviewer_RbND", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Summary:\n\nFL trains models both on the server and clients. Such model training becomes problematic when the data distributions on each of the clients diverge from each other. The paper answers the question of prioritizing the generic model at the server or the personalized model on each client. The paper proposes to decouple a model’s dual duties with two prediction tasks. \n\n### Pros:\n\n- The paper is well-written, easy to understand and follow along.\n\n- The paper is well motivated and driven by a real-problem in FL. \n\n- Considering model training on each of the clients as a class-imbalance problem and applying minimizations on the objectives considering the two losses in a multi-task setup is also interesting.\n\n### Cons:\nFollowing are the concerns that can be addressed for change in the final scores.\n\n1. “the literature does not evaluate G-FL algorithms in the P-FL setup by their local models” What does this mean? The evaluations on global test sets or the local test sets? In either case, this sentence does not really make a lot of sense.\n\n2. “taking average over model weights -- indeed acts like a regularizer” is also not a surprising finding, this indeed is general common sense to think that averaging is kind of a regularizer.\n\n3. In fact look at the following paper about bagging (which of course is not an FL algorithm but a simple general ensemble technique of averaging multiple models) that suggests to have regularization effects due to model averaging.\n    - Poggio, T., Rifkin, R., Mukherjee, S. and Rakhlin, A., 2002. Bagging regularizes. MASSACHUSETTS INST OF TECH CAMBRIDGE ARTIFICIAL INTELLIGENCE LAB.\n    - Skurichina, M. and Duin, R.P., 1998. Bagging for linear classifiers. Pattern Recognition, 31(7), pp.909-930.\n\n4. If one performs the P-FL evaluations on local test sets, at the beginning of each round with the newly communicated global model that might perform better on local test sets as opposed to weight for the local model to be fine-tuned and perform testing, not really surprising, even without regularization term because of the implicit reg effect due to averaging.\n\n5. In P-FL, the local test sets used in the experiment are static because of the standard benchmarks used. However, in real-time, these test sets will change and there will be temporal effects that need to be considered. In such cases, building test sets in itself becomes a challenging problem. The whole paper starts with the motivation of closing the gap between G-FL vs P-FL, the gap arises due to the data shifts, for which the corresponding test sets need to be prepared. This issue is not addressed at all.\n\n6. y_P = h^G+h^P, which is uniform weight for the P-FL part, how about you add weights and make them learable, something like this y_P = (1-\\alpha)h^G + \\alpha h^P and start with bigger \\alpha as this function optimizes for the P-FL anyway, why not try something along these lines and see how that improves P-FL and effects the G-FL. Any of the findings in this direction are also useful since you already proposed this and have the setup.\n\n\n### Post Rebuttal\nSatisfied with the authors response, increasing the score to a clear accept.\n", "summary_of_the_paper": "FL trains models both on the server and clients. Such model training becomes problematic when the data distributions on each of the clients diverge from each other. The paper answers the question of prioritizing the generic model at the server or the personalized model on each client. The paper proposes to decouple a model’s dual duties with two prediction tasks. ", "summary_of_the_review": "The idea of treating the non-IIDness on clients in FL as class imbalance is interesting, however, the paper can benefit from addressing the above raised concerns.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This manuscript proposes and analyses can approach to address the centralized and personalized tasks in federated learning jointly. Existing work has tackled this issue by developing separate tasks. Instead, this manuscript proposes a shared architecture that aims to optimize centralized and personalized models. One observation motivating this work is that local models trained during federated learning effectively optimize local task performance. The resulting approach results well when label shifts primarily drive the client variability. Here, the centralized components are trained to optimize a balanced risk, while the local components are trained to optimize the standard empirical risk. 

Reviewers agree that the manuscript is well-written and appropriately addresses the timely issue posed. The main concerns are the clarity of the technical contributions and technical statements during the review. The authors respond to these concerns and have satisfied the reviewers. After discussion, most reviewers are generally strongly positive about the strength of the manuscript contributions.
Decision: Accept (Spotlight)