FedBABU: Toward Enhanced Representation for Federated Image Classification
Authors: Jaehoon Oh
SangMook Kim
Se-Young Yun
Venue: ICLR 2022 Poster (2022)
Keywords: Federated Learning
Representation Learning

Abstract: Federated learning has evolved to improve a single global model under data heterogeneity (as a curse) or to develop multiple personalized models using data heterogeneity (as a blessing). However, little research has considered both directions simultaneously. In this paper, we first investigate the relationship between them by analyzing Federated Averaging at the client level and determine that a better federated global model performance does not constantly improve personalization. To elucidate the cause of this personalization performance degradation problem, we decompose the entire network into the body (extractor), which is related to universality, and the head (classifier), which is related to personalization. We then point out that this problem stems from training the head. Based on this observation, we propose a novel federated learning algorithm, coined FedBABU, which only updates the body of the model during federated training (i.e., the head is randomly initialized and never updated), and the head is fine-tuned for personalization during the evaluation process. Extensive experiments show consistent performance improvements and an efficient personalization of FedBABU. The code is available at https://github.com/jhoon-oh/FedBABU.

Published as a conference paper at ICLR 2022
FEDBABU: TOWARD ENHANCED REPRESENTATION FOR FEDERATED IMAGE CLASSIFICATION
Jaehoon Oh
Sangmook Kim; Se- Ye-Young Yun
Graduate School of KSE, KAIST jhoon. ohe kaist.aasas....... ac.kr
Graduate School of AI, KAIST (sangmook. kim, yunseyoung (aapnnnggkkkkkk @eakik
ABSTRACT
Federated learning has evolved to improve single global model under data het- erogeneity (as a curse) or to develop multiple personalized models using data heterogeneity (as a blessing). However, little research has considered both direc- tions simultaneously. In this paper, we first investigate the relationship between them by analyzing Federated Averaging (MeMahan et al. (017) at the client level and determine that a better federated global model performance does not constantly improve personalization. To elucidate the cause of this personalization performance degradation problem, we decompose the entire network into the body (extractor), which is related to universality, and the head (classifier), which is related to per- sonalization. We then point out that this problem stems from training the head. Based on this observation, we propose a novel federated learning algorithm, coined FedBABU, which only updates the body of the model during federated training (i.e., the head is randomly initialized and never updated), and the head is fine-tuned for personalization during the evaluation process. Extensive experiments show sistent performance improvements and an efficient personalization of FedBABU. The code is available at /httpa://github com/
INTRODUCTION
Federated learning (FL) (McMahan et al. 2017). which is a distributed learning framework with personalized data, has become an attractive field of research. From the early days of this field, improving a single global model across devices has been the main objective (Zhao et al. 2018 Duan et al.) 2019: Li et al.] 2018 Acar et al.] (021). where the global model suffers from data heterogeneity. Many researchers have recently focused multiple personalized models by leveraging data heterogeneity across devices as a blessing in disguise (Chen et al. 2018 Dinh et al. [021 Zhang
al. 2020 Fallah et al., 2020 Shamsian al. 202T: Smith al. 2017). Although many studies have been conducted on each research line, a lack of research remains on how to train good global model for personalization purposes (Ji et al. 202 Jiang et al. 2019). In this study, for personalized training, each local client model starts from g global model that learns information from all clients and leverages the global model fit its own data distribution.
Jiang et al. (209)) analyzed personalization methods that adapt to the global model through fine- tuning on each device. They observed that the effects of fine-tuning are encouraging and that training in central location increases the initial accuracy (of single global model) while decreasing the personalized accuracy (of on-device fine-tuned models). We focus on why opposite changes in the two performances appear. This suggests that the factors for universality and personalization must be dealt with separately, inspiring us to decouple the entire network into the body (i.e., extractor) related to generality and the head (i.e., classifier) related to specialty, as in previous studies (Kang et al. 2019
Yu et al. 22200] Yosinski et al. 2014: Devlin et al. 2019) for advanced analysis. KKng et al. and|Yu al. 2020) demonstrated that the head biased in class-imbalanced environments. Note
(019)
that popular networks such as MobileNet (Howard et al. (017)) and ResNet (He et al.) (016) have
only one linear layer at the end of the model, and the head is defined as this linear layer whereas the
body is defined as all of the layers except the head. The body of the model is related to representation learning and the head of the model is related to linear decision boundary learning. We shed light on the cause of the personalization performance degradation by decoupling parameters, pointing out that such a problem stems from training the head.
"Equal Contribution

---

Published as a conference paper at ICLR 2022
Brosizaaling
Local lipdatc
Canud Serve
Connal Surver
Caraal Sunce
Chos
Clee 2
Cloc-
Clew
Cliceiz
Clew
Cleer
Chai
(a) FedAvg.
Brondcastng
Tool Ujuken
Agreptioo
Sucu deetfer
Cleen
Clas3
Claer
Claer
Class
Clier
Clasr
Clezs
(b) FedBABU,
Figure I: Difference in the local update and aggregation stages between FedAvg and FedBABU. In the figure, the lines represent the decision boundaries defined by the head (i.e., the last linear classifier) of the network; different shapes indicate different classes. It is assumed that each client has two classes. (a) FedAvg updates the entire network during local updates on each client and then the local networks are aggregated entirely. Therefore, the heads of all clients and the head of the server are different. Whereas, (b) FedBABU only updates the body (i.c., all of the layers except the head) during local updates on each client and then the local networks are aggregated mody-partially. Therefore, the heads of all clients and the head of the server are the same.
Inspired by the above observations, we propose an algorithm to learn a single global model that can be efficiently personalized by simply changing the Federated Averaging (FedAvg) (McMahan et al. 2017) algorithm. FedAvg consists of four stages: client sampling. broadcasting, local update, and aggregation. In the client sampling stage, clients are sampled because the number of clients is so large that not all clients can participate in each round. In the broadcasting stage, the server sends a global model (i.e., initial random model at the first broadcast stage an aggregated model afterward) to participating clients. In the local update stage, the broadcast model of each device is trained based on its own data set. In the aggregation stage, locally updated models are sent to the server and aggregated by averaging. Among the four stages, we focus on the local update stage from both the universality and personalization perspectives. Here, we only update the body of the model in the local update stage, i..., the head is never updated during federated training. From this, we propose FedBABU, Federated Averaging with Body Aggregation and Body Update. Figure I describss the difference during the local update and aggregation stages between FedAvg and FedBABU. Intuitively, the fixed head can be interpreted as the criteria or guideline for each class and our approach is representation learning bused on the same fixed criteria across all clients during federated training. This simple change improves the representation power of a single global model and enables the more accurate and rapid personalization of the trained global model than FedAvg.
Our contributions are summarized as follows:
We investigate the connection between single global model and fine-tuned personalized models by analyzing the FedAvg algorithm at the client level and show that training the head using shared data the server negatively impacts personalization.
We demonstrate that the fixed random head can have comparable performance the learned head in the centralized setting. From this observation, we suggest that sharing the fixed random head across all clients can be more potent than averaging each learned head in federated settings.
We propose a novel algorithm, FedBABU, that reduces the update and aggregation parts from the entire model to the body of the model during federated training. We show that FedBABU is efficient, particularly under more significant data heterogeneity. Furthermore, single global model trained with the FedBABU algorithm be personalized rapidly (even with one fine-tuning epoch). We observe that FedAvg outperforms most of preexisting personalization FL algorithms and FedBABU outperforms FedAvg in various FL settings. We adapt the body update and body aggregation idea to the regularization- based global federated learning algorithm (such as FedProx (Li al. 2018)). We show that regularization reduces the personalization capabilities and that this problem is mitigated through BABU.
2

---

Published as a conference paper at ICLR 2022
2 RELATED WORKS
FL for a Single Global Model Canonical federated learning, FedAvg proposed by McMahan et al. (017)), aims to learn a single global model that collects the benefits of affluent data without storing the clients' raw data in a central server, thus reducing the communication costs through local updates. However, it is difficult to develop a globally optimal model for con-independent and identically distributed (non-lID) data derived from various clients. To solve this problem, studies have been conducted that make the data distribution of the clients IID-like or add regularization to the distance from the global model during local updates. Zhhao et al. (2018) suggested that all clients share a subset of public data, and Duan et al.
(2019) augmented data to balance the label distribution of clients. Recently, two studies (Li et al.
2018 Acar et al.) 2021) penalized local models that have large divergence from the global model, adding a regularization term to the local optimization process and allowing the global model to converge more reliably. However, it should be noted that the single global model trained using the above methods is not optimized for each client.
Personalized FL Personalized federated learning aims to learn personalized local models that are stylized to each client. Although local models can be developed without federation, this method suffers from data limitations. Therefore, to maintain the benefits of the federation and personalized models, many other methods have been applied to FL: clustering, multi-task learning, transfer learning, regularized loss function, and meta-learning. Two studies (Briggs et al. 2020 Mansour et al., (020) clustered similar clients to match the data distribution within a cluster and learned separate models for cach cluster without inter-cluster federation. Similar to clustering, multi-task learning aims to Iearn models for related clients simultancously. Note that clustering ties related clients into single cluster, whereas multi-task learning does not. The generalization of each model can be improved by sharing representations between related clients. [Smith et al. (2017) and Dinh eet al. (202T) showed that multi-task learning is an appropriate learning scheme for personalized FL. Transfer learning is also recommended learning scheme because it aims to deliver knowledge among clients. In addition, Yang et al. (2020) and al. (2020b) utilized transfer learning to enhance local models by transferring knowledge between related clients. T Dinh et al. (2020) and Li al. (202T) added a regularizer toward the global or average personalized model to prevent clients' models from simply overfitting their own dataset. Unlike the aforementioned methods that developed local models during training. Chen et al. (2088) and Fallah et al. (2020) attempted to develop good initialized shared global model using bi-level optimization through Model-Agnostic Meta-Learning (MAML) approach (Finn et al.] 2017). A well-initialized model can be personalized through updates on each client (such inner updates in MAML). Jiang et al. (2019) argued that the FedAvg algorithm could be interpreted as a meta- meta-learning algorithm, and personalized local model with high accuracy can be obtained through fine-tuning from global model learndd using FedAvg. In addition, various technologies and algorithms for personalized FL have been presented and will be helpful to read; see Tan et al. (2021) and Kulkarni et al. (2020) for more details.
Decoupling the Body and the Head for Personalized FL The training scheme that involves de- coupling the entire network into the body and the head has been used in various fields, including long-tail recognition (Kang et al. 2019 Yu et al. 2020), noisy label learning (Zhang & Yao, 2020), and meta-learning (Oh et al.] 2021 Raghu et al.) 2019 For personalized FL, there have been attempts to use this decoupling approach. For a consistent explanation, we describe each algorithm from the perspective of local update and aggregation parts. FedPer (Arivahhagan et al. 2019), similar to FedPav (Zhuang et al., 2020), learns the entire network jointly during local updates and only aggregates the bottom layers. When the bottom layers are matched with the body, the body is shared on all clients and the head is personalized to each client. LG-FedAvg (Liang et al. (2020) learns the entire network jointly during local updates and only aggregates the top layers based on the pre-trained global network via FedAvg. When the top layers are matched with the head, the body is personalized to each client and the head is shared on all clients. FedRep (Collins et al., 2021) learns the entire network sequentially during local updates and only aggregates the body. In the local update stage, each client first learns the head only with the aggregated representation and then learns the body only with its own head within single epoch. Unlike the above decoupling methods, we propose a FedBABU algorithm that learns only the body with the randomly initialized head during local updates and aggregates only the body. It is thought that fixing the head during the entire federated training provides the same guidelines on learning representations across all clients. Then, personalized local models can be obtained by fine-tuning the head.
'Athhough there are more studies related to decoupling parameters Rusu et al. 2018 Lee & Choi 2018 Flennerhag et al. 2019. Chen al. 2009b)) we focus on decoupling the entire network into the body and head.
3

---

Published as a conference paper at ICLR 2022
3 PRELIMINARIES
FL training procedure We summarize the training procedure of FL. following the aforementioned four stages with formal notations. Let {1,-.. .N3 be the set of all clients. Then, the participating clients the communication round & with client fraction ratio f is C* = [ By broadcasting, initialized by the global parameter
the local parameters of the participating clients { (0)) [ 06-1, that is, 0f (0) -0 for all i [1, [] and & E [1. Note that 0g is randomly initialized first. On its own device, each local model is updated using locally kept data set. After local epochs T, the locally updated models become (T / [NS] where 7バ. is the number of iterations of one epoch on client C (i.e., " D) nct is the number of data samples for client C₁ and B is the batch size. Therefore, TI. is the total number of updates in one communication interval. Note that our research mainly deals with balanced environment in which all clients have the same size data set (i.e., 1 is constant for all k and i) 2
Finally, the global parameter 0g is aggregated by EIND "ct (T where n LNS. not, at the server. For our algorithm, the parameters A are decoupled into the body (extractor) parameters Oeze and the head (classifier) parameters Octs E RCxdd where d is the dimension of the last representations and C' is the number of classes.
1
Experimental setup We mainly use MobileNet CIFAR100 3 We set the number of clients to 100 and then each client has 500 training data and 100 test data; the classes in the training and test data sets are the same. For the heterogeneous distribution of client data, we refer to the experimental setting in McMahan et al. (017). We sort the data by labels and divide the data into the same-sized IDL shards. Because there is no overlapping data between shards, the size of a shard is defined by Nx.* where | is the data set size, N is the total number of clients, and is the number of shards per user. We control FL. environments with three hyperparameters: client fraction ratio S, local epochs т, and shards per user s. f is the number of participating clients out of the total number of clients in every round and small f is natural in the FL settings because the total number of clients is numerous. The local epochs are equal to the interval between two consecutive communication rounds. To fix the number of total updates to ensure the consistency in all experiments, we fix the product of communication rounds and local epochs to 320 (e.g.. if local epochs are four, then the total number of communication rounds is 80). The learning rate starts with 0.1 and is decayed by a factor of 0. at half and three-quarters of total updates. T is closely related to the trade-off between accuracy and communication costs. A small provides accurate federation but requires considerable communication costs. s is related to the maximum number of classes each user can have; hence, as s decreases, the degree of data heterogeneity increases.
Evaluation We calculate the initial accuracy and personalized accuracy of FedAvg and FedBABU following the federated personalization evaluation procedure proposed in Wang al. (019) to analyze the algorithms at the client level: (1) the learned global model is broadcast to all clients and is then evaluated on the test data set of each client DC (referred to the initial accuracy), (2) the learned global model is personalized using the training data set of each client D by fine-tuning with the fine-tuning epochs of Ty: the personalized models are then evaluated on the test data set of each client D (referred to as the personalized accuracy). In addition, we calculate the personalized accuracy of other personalized FL algorithms (such as FedPer, LG-FedAvg, and FedRep). Algorithm 2 in Appendix A describes the evaluation procedure. The values (X+Y) in all tables indicate the standard deviation of the accuracies across all clients, not across multiple seeds. Here, reducing the variance over the clients could be interesting but goes beyond the scope of this study.
4 PERSONALIZATION OF A SINGLE GLOBAL MODEL
We first investigate personalization of single global model using the FedAvg algorithm, following Wang et al. (2019) and Jiang et al. (2019), to connect a single global model to multiple personalized models. We evaluate both the initial accuracy and the personalized accuracy, assuming that the test data sets are not gathered in the server but scattered on the clients.
Appendix H reports the results under unbalanced and non-lll) derived Dirichlet distribution.
We also use 3conyNet on EMNIST, 4convNet on CIFAR10, 10, and ResNet CIFAR 100. Appendix
F and
report results, respectively. Appendix Alpresents the details of the architectures used.
4

---

Published as a conference paper at ICLR 2022
Table 1: Initial and personalized accuracy of FedAvg on CIFARIO0 under various FL settings with 100 clients. MobileNet is used. The initial and personalized accuracy indicate the evaluated performance without fine-tuning and after five fine-tuning epochs for each client, respectively.
FL settings
s=100 (heterogencity J)
50
==10 (beterogencity f)
7
Initial
Personalized
Initial
Personalized
Initial
Personalized
46 46.93-4-5447
$1.93-15.19
55.68-555.50
57.84-15.08
37.27:1697
77,46:4:578
37.44:+:4.8
52.66:25.0
36.05-t4404
47.17:::.26
24.17::5330
70,41:b683
10
29.58:4:487
34.62
29.57::429
40.59::..25
17.85
63,5117.38
39.07:t-5.22
43.92::5.55
38.20 38:20-::333
49.55-:-3.6
29.12-t:.11
71.24-1-72
0.1
35.39:
39.67445.11
33.49-44.72
43.63-24777
21.14::6.6
67. 67.14-2672
28:18:::483
33.13::522
27.34:::56
38.09
14.40:1:44
62.67:b6522
Table 2: Initial and personalized accuracy of FedAvg on CIFAR 100 under a realistic FL setting (N=100, f=0.1, 1=10) according to p, which is the percentage of all client data that the server also has. Here, the entire (or, full) network or body is updated on the server using the available data.
==100 (heterogencity 1)
5=
s=10 (heterogeneity f)
P
Initial
Personalized
Initial
Personalized
Initial
Personalized
28. 18-48-44
33.13+5m
20.34-406
38.09
14.40
62.67=6652
0.05 (Full)
29. 29
32.59::5:4
27.13=4.44
34.34
18.22:25.4
54.68
0.05 (Body)
28.50+:99
33.03-3336
27.96=436
39.10:33.55
14.78
60.19:6.46
0.10 (Full)
30.59::498
33.34:1:330
29.62=427
35.50:2:484
19.24:5315
49.62==748
0.10 (Body)
32.90
36.82.4-4666
32.81 4.007
18.35
60.94-77.0
Table I describes the accuracy of FedAvg on 100 according to different FL settings (f., т, and 8) with 100 clients. The initial and personalized accuracies indicate the evaluated performance without fine-tuning and with five fine-tuning epochs for each client, respectively. As previous studies have shown, the more realistic the setting (i.e., a smaller f. larger т, and smaller s), the lower the initial accuracy. Moreover, the tendency of the personalized models to converge higher than the global model observed in Diang et al. (2019) is the same. More interestingly, it is shown that the gap between the initial accuracy and the personalized accuracy increases significantly as the data become more heterogeneous, It is thought that small makes local tasks easier because the label distribution owned by each client is limited and the number of samples per class increases.
In addition, we conduct an intriguing experiment in which the initial accuracy increases but the personalized accuracy decreases, maintaining the FL training procedures. In Diang al. (2019), the authors showed that centralized trained models are more difficult to personalize. Similarly, we design an experiment in which the federated trained models are more difficult to personalize. We assume that the server has small portion of p of the non- non-private client data of the clientF] such that the non-private data can be used the server to mitigate the degradation derived from data heterogeneity. We update the global model using this shared non-private data after every aggregation with only one epoch. Table 2 describes the result of this experiment. We update the entire network (F in TTalle 2) on the server. As p increases, the initial accuracy increases, as expected. However, the personalized accuracy decreases under significant data heterogeneity (s=10). This result implies that boosting single global model can hurt personalization, which may be considered more important than the initial performance. Therefore, we agree that the development of an excellent global model should consider the ability to be adequately fine-tuned or personalized.
To investigate the cause of personalization performance degradation, we hypothesize that unnecessary and disturbing information for personalization (i.e., the information of similar classes that a certain client does not have) is injected into the head, when a global model is trained on the server 5l To capture this, we only update the body (B in Table 2} on the server by zeroing the learning rate corresponding to the head. By narrowing the update parts, the personalization degradation problem can be remedied significantly without affecting the initial accuracy. From this observation, we argue that training the head using shared data negatively impacts the personalization
Non- -oin-trivate data are sampled randomly in our experiment, which can violate the FL environments. However. this experiment is conducted simply as a motivation for our study.
'We also investigate personalization of centralized trained models such (Jiang et al. 2019). and the results reported in Appendix C Even this case, training the head also has negative impact on personalization. 'Zhuang et al. (222] and Luo al. (2021) posed similar problem that the head can be easily biased because it is closest to the client's ss label distribution. /hhang et al. (202T) proposed aivergence-aware predictor update
module, and [uo et al. (202T) and Andittue et al. (202T) proposed head calibration method.
5

---

Published as a conference paper at ICLR 2022
5 FEDBABU: FEDERATED AVERAGING WITH BODY AGGREGATION AND BODY UPDATE
We propose novel algorithm that learns a better single global model to be personalized efficiently. Inspired by prior studies on long-tailed recognition (Yu et al. 2020 Kang et al. 2019), fine-tuning (Devlin al. 2019), and self-supervised learning (He et al., 2020), well as our data-sharing experiment, we decouple the entire network into the body and the head. The body is trained for generalization and the head is then trained for specialization. We apply this idea federated learning by never training the head in the federated training phase (i.e., developing single global model) and by fine-tuning the head for personalization (in the evaluation process).
5.1 FROZEN HEAD IN THE CENTRALIZED SETTING
Before proposing our algorithm, we empirically demonstrate that model with the initialized and fixed head (body in Figure 2) has comparable perfor- mance to model that jointly learns the body and the head (full in Figure 2). Figure 2 depiets the test ac- curacy curve of MobileNet on CIFAR100 for various training scenarios in the centralized setting, where total epochs are 160 and learning rate starts with 0.1 and is decayed by a factor of 0. at 80 and 120 epochs. The blue line represents the accuracy when all layers trained, the red line represents the accuracy when only the body of the model is trained, and the green line represents the accuracy when only the head of the model is trained. The fully trained model and the fixed head have almost the same performance, whereas the fixed body performs poorly. Thus, we claim that the randomly initialized fixed head is acceptable, whereas the randomly initialized fixed body is unacceptable; initialized head thought to serve as guidelines. This characteristic is derived from the orthogonality on the head, as is explained in Appendix B These results are the reasons we can update only the body during local training in FL settings.
0.6
0.5
0.4
full
head
0.3
body
0.2
0.1
0.0
0
20
40
60
80
100
120
140
160
Figure 2: Test accuracy curves according to the update part in the centralized setting.
5.2 FEDBABU ALGORITIM
Algorithm I Training procedure of FedBABU.
Based on the insights and results in Section 5.1 we propose a new FL algorithm called
1:
initialize 68 100 Co for each round is I,..., K do m +- mar([Nf).
2:
3:
FedBABU (Federated Averaging with Body Aggregation and Body Update). Only the body is trained, whereas the head is never trained dur- ing federated training; therefore, there is no need to aggregate the head. Formally, the model pa- rameters are decoupled into the body (extrac-
4:
O* <- random subset m. clients
5:
for elieet Cf C in parallel do
6:
of (0) + o% (0 o2 (f + ClientBody (0), end for
7:
8:
not o (Td,), D Pot
9: 02 + 2mm
tor) parameters sext and the head (classifier)
10: end for
parameters Oct.. Note that Oate on any client is fixed with the head parameters of a randomly
11: return 6% c (g
12: function CODDATRODYUDDATTEE....
initialized global parameters 0& until a single
13:
1 -
global model converges. This is implemented by zeroing the learning rate that corresponds to the head. Intuitively, it is believed that the same
14:
For each local epoch 1, do
15:
for each iteration 1, 15 do 0 SGD c 00 C.olo
16:
fixed head on all clients serves as the same crite-
18:
17:
end for
end
ria on learning representations across all clients
19:
return (e
despite the passage of training time.
20:
end function
Algorithm D describes the training procedure of FedBABU. All notations are explained in the FL training procedure paragraph in Section Lines 3-4, Line 6, Line 7, and Line 9 correspond the
to the client sampling, broadcasting, local update, and aggregation stage, respectively. BhentBoodUUdaat function, the local body parameter 0% is always updated based on the same head C (Line 16). Then, only the global body parameter 0l: 0G.. is aggregated (Line 9). Therefore, the final global parameter ØK (Line 11) and the initialized global parameter 66 (Line 1) have the same head parameter C.

---

Published as a conference paper at ICLR 2022
In this section, we first investigate the representation power after federated training (Section 5.2.1 Next, we demonstrate the ability related to personalization (Section 5.2.2 to Section 5.2.44 We further show our algorithm's 's applicability (Section 5.2.5).
5.2. REPRESENTATION POWER OF GLOBAL MODELS TRAINED BY FEDAVG AND FEDBABU
Higher initial accuracy is often required because some clients may not have data for personaliza- tion in practice. In addition, the representation power of the initial models is related to the per- formance of downstream or personal tasks. To capture the representation power, the "without (wo) head" accuracy is calculated by replacing the trained head with the nearest template fol- lowing previous studies (Kang et al., 2019: Snell al., 2017 Raghu et al.) 2019 Oh et al., (021): (I) Using the trained global model and training data set on each client, the representations of each class are averaged into the template of each class. Then, (2) the cosine similarity between the test samples and the templates are measured and the test samples are classified into the near- est template's 's class. Templates are created for each client because raw data cannot be moved.
Table 3:
Initial accuracy of FedAvg and FedBABU on CIFARTO0 under various settings with 100 clients. The trained head is replaced with the near- est template to calculate "woo head" accuracy and MobileNet is used.
FL settings
FedAug
FedBABU
f
w/ head
wlo head
w/ head
wlo head
46.93-55.7
23+--555
48.61 -475
49.97-44409
1.0
4
37 4794
33 5..00
37.32-434
37.20++-435
100
10
29,58-2487
25. 11::4.0
26.69=455
27.70:24511
39. 59.07 =:32
11:07:995
41. 19:2465
0.1
4
38,
4.5
12. 58+-437
36. 56.77-447
36.61--4466
10
28.
18:28:
24.34::455
29.58
29,36:24446
1
45.68-25.0
53. 33:33:5:3
47:
55. 70:
1.0
4
36.05 4:044
42 65
$7.27 25
45 45.25
29.57-422 10
34. 13
28.43m447
36, 19+495
1
38, 38.20
4157.13344
41.33=5116
49, 18:577
0.1
4
33.49:
40.01:::49
34.68=4559
42.43-55.11
27.34-196 10
33. 504
27.91
36.49---37
1
37.27=249
67. 18+
71.23+66711 45
1.0
4
24.17
58.70
64.412244
Table 3 describes the initial accuracy of FedAvg and FedBABU according to the existence of the head. Because all clients share the same criteria and learn representations based on that criteria during federated training in FedBABU, improved representation power is expected When evaluated without the head, FedBABU has improved performance compared to FedAvg in all cases, particularly under large data heterogeneity. Specifically, when s == 100 (i.e., each client has most of the classes of CIFAR100), the performance gap depends on whether the head exists in FedAvg, but not in FedBABU. This means that the features through ABU are well-represented to the extent that there is nothing to be supplemented through the head. When is small, the performance improves when there is no head. Because templates are constructed based on the training data set for each client, it is not possible to classify test samples into classes that the client does not have. In other words, restrictions the output distribution of each client lead to tremendous performance gains, even without training the global model. Therefore, it is expected that fine-tuning the global models with a strong representation can boost personalization.
17.85
51.724766
22.15=171
55.634724
1
29.1 29.12-2711
60.424789
65.98 35.08-763
0.1
4
21.14226:6
54.9116722
25.67=731
59.441643
14.40:2344
50.255:637
18.50
54.939785
5.2.2 PERSONALIZATION OF FEDAVG AND FEDBABU
Table 4: Personalized accuracy of FedAvg and FedBABU on CIFARIO0 according to the fine-tuned part. The fine-tuning epochs is 5 and f is 0.I.
(a) FedBABU,
(b) FedAvg.
FL. settings
Update part for personalization
FL. settings
Update part for personalization
8
Body
Head
Full
8
Body
Head
Full
44.261:3.12
49.7622500
49.
49.67:t:492
41 11.00::::35
43.181:34
43.92=5:5
100
4
44.7415500
44.74::5.10
100
37.43+:496
38.294496
39.67=521
32.45:+542 10
36.48-5504
35.94-45.06
30.62-++499 10
31.92+5004
33.13=552
1
48.24:::523
$6.76:1368
$6.69:1:.16
43.61:1:554
47.51
49.55=536
50
4
41.27:15004
49.45-1541
59.55:::.58
50
37. 37.99::4.68
41.48:441
43.63=4.77
10
35.42::5600
42.55:13.70
42.63:1:.59
10
32.20:1492
36.06:5311
38.09=3.17
1
72.8117.33
75.97=6.29
76.02-b629
56.00=tx.66
69.70177
71.24=1.22
10
4
69. 69.12-t6.00
70.74
71.00:b663
10
34.49-:7.22
65.32
67.14
10
64.7747.14
66.28-26.77
66.32-47702
27.94-4696
60.24-t6.16
62.67=6.52
Appendix O provides the results of FedAvg with different learning rates between the head and the body. which shows the freezing the head completely is critical. Appendix ffurhher provides class-wise analysis during federated training, which shows that freezing the head makes less damage to out-of-class during local updates.
7

---

Published as a conference paper at ICLR 2022
To investigate the dominant factor for personalization of FedBABU, we compare the performance according to the fine-tuned part. TTble da the results of this experiment. Global models are fine-tuned with five epochs based on the training data set of each client. It is shown that fine-tuning including the head (i.e., Head or Full in Table 4a) is better for personalization than mody-partially fine-tuning (i.e., Body in [Table 4a). It noted that the computational costs can be reduced by fine-tuning only the head in the case of FedBABU without performance degradation.
However, in the case of FedAvg, a performance gap appears. Table 4b describes the personalized accuracy of FedAvg according to the fine-tuned parts. It is shown that fine-tuning the entire network (Full in Table 4b) is better for personalization than partial fine-tuning (Body or Head in Table 4b). Therefore, for FedAvg. it is recommended to personalize global models by fine-tuning the entire network for better performance. For consistency of the evaluation, for both FedAvg and FedBABU, we fine-tuned the entire model in this paper.
5.2.3 PERSONALIZATION PERFORMANCE COMPARISON
Table 5: Personalized accuracy comparison on CIFARIO0 under various settings with 100 clients and MobileNet is used.
FL. settings
Personalized accuracy
f
FedBABU (Ours)
FedAve [017
FodPer
2019
LG-FedAvg
2020
FedRep
2021
Per-FodAvg [2020]
2021
Ditto
Local-only
55.79 1:457
51.93_519
$1.95.1..30
53.01
18.29
1..0
39.36.14.78
1.0
44.49:t491
42.66-509
40.87-t:5.05
43.09-+4.34
15.32-+3779
39:07+ 59
31,58-+5.00
100
33.20+4454
34.62=497
32.91
34.64+3.0
13.45#3.26
30.22+6559
21.18:44.54
10.60
49.67
43.92=5555
45.17:1470
40.91:11.50
23.84:1.92
48. 48.10227442
45.46:14.73
0.1
44.74 1:
39.67 521
34.30
37,87:1499
16.01:1348
33.70.1/044
32.46.1:542
35.94+5505 06
23-13-2222
32.08444.7
30.084:5.4 34
13:3333311
25.824:588 83
23.96:244644
61.09::491
$7.84 97.84=3.08
$7.16d:1.2
58.44:1:533
24.75=23:02
43.75277:4
42.70:15.46
1.0
51.56.1:04
47.17-426
48.89:1.5.0
47.78:1472
21.55:14436
37.59
36.57:15.11
50
12.09:5512
50.59
39.90
40.32.4470
19.92:44.50
28
27.27.4..04
28.02++401
56.69+5.15
49. 56-555333
51.63++5.27
52.64411.5
32.88=5.00
43.9627740
43,22:+5.62
0.1
59.55
43.63=4.77
46.31 63
38.54:2711
21.13:13.6
28.67:2.95
31.65:::3.0
42.63
38.09 5.17
39.81
30.79.1..12
15.15.1.01
11.641..16
22.16.14.67
79.17::631
77.46=3.88
74.71:26.35
77.49:1:00
61 61.2811.27
36.5922.95
65.33:17.49
1.0
74.60
70.41 6.8
65.61
66.97.1642
50.59
18.31.11557
64.47.1745
10
66.64:1684
63.51=7.38
59.71177.35
61.5011728
42.13:1..5
11.542887
51.68:17.44
61.52472:
76.02
71.24-782 7.82
69.36-1677
51.75.19.3
60.13-17.72
3:.21
31.91
0.1
71.00-663
67. 67
62. m2
35,80+ 10.55
45.91+7788
14.342951
23.70:::5.84
66.32:7702
62.67=652
59.50:1..3
25.04::12.02
34.30:2.54
9.17
14.24
We compare FedBABU with existing methods from the perspective of personalization. Appendix presents details of the evaluation procedure and implementations of each algorithm. [Table describes the personalized accuracy of various algorithms. Notably, FedAvg is a remarkably strong baseline, overwhelming other recent personalized FL. algorithms on CIFAR 100 in most cases are similar to recent trends in the field of meta-learning, where fine-tuning based on well-trained representations overwhelms advanced few-shot classification algorithms for heterogeneous tasks (Chen et al. 2019a) 2020a; Dhillon et al. 2019 Tian et al. 2022).. FedBABU (ours) further outshines FedAvg; it is believed that enhancing representations by freezing the head improves the performance.
@ These results
In detail, when r=1, the performance gap between FedBABU and FedRep demonstrates the impor- tance of fixing the head across clients. Note that when r=1, if there is no training process the head in FedRep, FedRep is reduced to FedBABU. Moreover, we attribute the performance degradation of FedRep to the low epochs of training on the body. In addition, the efficiency of personalized FL algorithms increases when all clients participate; therefore, FedPer (Arivazhagan el al. 2019) assumes the activation of all clients during every communication round. The Per-FedAvg algo- rithm might suffer from dividing the test set into a support set and query set when there are many classes and from small fine-tuning iterations. In summary, FedAvg are comparable and better than even recent personalized FI. algorithms such as Per Par-FedAvg and Di tito, and F FedBABU (ours) achieves the higher performance than F FedAvg and other decoupling algorithms.
SSimilar results were reported in the recent paper (Collins et al. 2021) ) showing that FedAvg beats recent
personalized algorithms (Smith et al. 2017 Fallah et al. 2020 Liang et al. 2020 Hanzely & Richtárik 2020
Deng et al. 2020 Li al. 2021 Anvazhagan et al. 2019) 9) many cases. Appendix N further discusses the
effectiveness of FedAvg.
8

---

Published as a conference paper at ICLR 2022
5.2.4
PERSONALIZATION SPEED OF FEDAVG AND FEDBABU
Table 6: Performance according to the fine-tune epochs (FL setting: /=0.1, and r=10),
Algorithm
Fine-tune epochs (ry)
0 (Initial)
2
5
10
20
50
FedBABU
FedAvg
27.91
47.34
29.17
35.20
40,6011547
22.39:4777
42.12-2511
34.97=5.33
36 16
42.74:15.00
42.63::5.9
38.09=5.77
40.56:5543
41.941564
41.20#351
41.19135:
40.861511
40.611:23
FedAvg
14.40+1144
27.43
48.63+7.30
58.00 18.08-611
61.27-2615
62. 52.67-6655
63-91-669
64.56-66s
64.89+6.81
10
TedBABU 18.50
63.29
66,05
56. 10:3644
66.40:17.24
66.32:17:2
66.07.12577
66.2412260
66.3242771
We investigate the personalization speed of FedAvg and FedBABU by controlling the fine-tuning epochs Tf in the evaluation process. Table describes the initial (when Tf 0) and personalized (otherwise) accuracy of FedAvg and FedBABU. Here, one epoch is equal to 10 updates in our case because each client has 500 training samples and the batch size 50. has been shown that FedAvg requires sufficient epochs for fine-tuning, achieves better accuracy with a small number of epochs, which means that FedBABU can personalize accurately and rapidly, especially when fine-tuning is either costly restricted. This characteristic is explained further based on cosine similarity in Appendixx K
reported in Jiang et al. (2019). Notably, FedBABU
5.2.5 BODY AGGREGATION AND BODY UPDATE ON THE FEDPROX
Table 7: Initial and personalized accuracy of FedProx and BeAPro+BABBU with w=0.01 on CIFAR 100
with 100 clients and f=0.1.
s=100 (heterogeneity J)
5=50
s=10 (heterogeneity 1)
Algorithm
Initial
Personalized
Initial
Personalized
Initial
Personalized
46.52-14.56
50.95
42.20-14.90
$1.29
28.16:19900
66.39
FedProx
36.544:474
39.83=t4771
33.59:
40.17
18.20=17.2
41.56:19..4
10
28.63-+4.40
31.90
26.88
32.92-+:.000
13.62:4:773
43.48-t-9.2
48.533:5.15
57.44::4.72
46.25:1..11
63.12:::.25
33.13:18.11
78.861:5.70
FedProx
37.17:1441
45.26:14.76
33.8611..
50.18::::14
22.94-19.90
75.71
+BABU
10
27.79
35.68-44.34
27.48
42.37-t6.10
15.66-t8.29
67.15-27.10
FedProx (Li et al. 2018) regularizes the distance between a global model and local models to prevent local models from deviating. The degree of regularization is controlled by Jr. Note that when is 0.0, FedProx is reduced to FedAvg. Table 7 describes the initial and personalized accuracy of FedProx and BBBUU with μ=0.01 when f=0.1, 1, and Appendix M reports the results when f=1.0. The global models trained by FedProx reduce the personalization capabilities compared to FedAvg (refer to Table I] when f=0.1), particularly under realistic FL settings. We adapt the body aggregation and body update idea to the FedProx algorithm, referred to as FedPrOx+BABU, which performs better than the personalization of FedAvg. Furthermore, FedProx+BABB improves personalized performance compared to FedBABU (refer to TTaall when f=0.1). This means that the regularization of the body still meaningful. Our algorithm and various experiments suggest future directions of federated learning: Which parts should be federated and enhanced? Representation!
6 CONCLUSION
In this study, we focused on how to train a good federated global model for the purpose of person- alization. Namely, this problem boils down to how to pre-train better backbone in federated learning manner for downstream personal tasks. We first showed that existing methods to improve a global model (e.g., data sharing or regularization) could reduce ability personalize. To mitigate this personalization performance degradation problem, we decoupled the entire network into the body, related to generality, and the head, related to personalization. Then, we demonstrated that training the head creates this problem. Furthermore, we proposed the FedBABU algorithm, which learns shared global model that can rapidly adapt to heterogeneous data on each client. FedBABU only updates the body in the local update stage and only aggregates the body in the aggregation stage, thus developing a single global model with a strong representation. This global model can be efficiently personalized by fine-tuning each client's model using its own data set. Extensive experimental results showed that FedBABU overwhelmed various personalized FL algorithms. Our improvement emphasizes the importance of federating and enhancing the representation for FI..
9

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
This work was supported by Institute of Information & communications Technology Planning Evaluation (IITP) grant funded by the Korea government (MSIT) INo.2019-0-00075, Artificial Intelligence Graduate School Program (KAIST)) and [No. 2021-0-00907, Development of Adaptive and Lightweight SilgeCoollbborative Analysis Technology for Enabling Proactively Immediate Response and Rapid Learning].
REPRODUCIBILITY STATEMENT
Our code is available https: //g http:etttbocc.ooooooooeooooooe com//Shonn-oh//eeiAAa For convenience re- producibility, shell files of each algorithm are also included.
REFERENCES
Durmus Alp Emre Acar, Yue Zhao, Ramon Matas Navarro, Matthew Mattina, Paul N Whatmough, and Venkatesh Saligrama. Federated learning based on dynamic regularization, In International Conference on Learning Representations, 2021.
Idan Achituve, Aviv Shamsian, Aviv Navon, Gal Chechik, and Ethan Fetaya. Personalized federated learning with gaussian processes. arXiv preprint arXiv:2106. 15482, 2021.
Manoj Ghuhan Arivazhagan, Vinay Aggarwal, Aaditya Kumar Singh, and Sunav Choudhary. Feder- ated learning with personalization layers. arXiv preprint arXiv: 1xiv:1912.00018 2019.
Christopher Briggs, Zhong Fan, and Peter Andras. Federated learning with hierarchical clustering of local updates to improve training on non-iid data. In 2020 International Joint Conference on Neural Networks (IJCNN), pp. 1-9. IEEE, 2020.
David M Chan, Roshan Rao, Forrest Huang, and John F Canny. Gpu accelerated t-distributed stochastic neighbor embedding. Journal of Parallel and Distributed Computing, 131:1-13, 2019.
Fei Chen, Mi Luo, Zhenhua Dong, Zhenguo Li, and Xiuqiang He. Federated meta-learning with fast convergence and efficient communication. arXiv preprint 1802 07876, 2018.
Wei- Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. closer look at few-shot classification. In International Conference on Learning Representations, 2019a.
Yinbo Chen, Xiaolong Wang, Zhuang Liu, Huijuan Xu, and Trevor Darrell. new meta-baseline for few-shot learning. arXiv preprint arXiv:2003.04394 2020a.
Yiqiang Chen, Xin Qin, Jindong Wang, Chaohui Yu, and Wen Gao. Fedhealth: A federated transfer learning framework for wearable healtheare. JEEE Intelligent Systems, 35(4):83 2020b.
Yutian Chen, Abram I. Friesen, Feryal Behbahani, Arnad Doucet, David Budden, Matthew W Hoffman, and Nando de Freitas. Modular meta-learning with shrinkage. arXiv preprint 1rX55:790..05557 2019b.
Gary Cheng, Karan Chadha, and John Duchi, Fine-tuning is fine in federated learning. arXiv preprint arXiv:2/08.07311, 2021.
Gregory Cohen, Saeed Afshar, Jonathan Tapson, and Andre Van Schaik. Emnist: Extending mnist to handwritten letters. In 2017 International Joint Conference on Neural Networks (IJCNN), pp.
2921-2926. IEEE, 2017.
Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai. Exploiting shared represen- tations for personalized federated learning. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICMI. 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 2089-2099, PMLR, 2021. URL
mir.
www.i//oooonnnnnccccccmmmmmmaaat html
Yuyang Deng, Mohammad Mahdi Kamani, and Mehrdad Mahdavi. Adaptive personalized federated learning, arXiv preprint arXiv:2003. 13461, 2020.
10

---

Published as a conference paper at ICLR 2022
Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter the Association for Computational Linguisties: Human Language Technologies, Volume I (Long and Short Papers), pp. 4171-4186, 2019.
Guneet S Dhillon, Pratik Chaudhari, Avinash Ravichandran, and Stefano Soatto. A baseline for few-shot image classification. arXiv preprint arXiv: 02709, 2019.
Canh Dinh, Tung T Vu, Nguyen H Tran, Minh N Dao, and Hongyu Zhang. Fedu: A uni- fied framework for federated multi-task learning with laplacian regularization. arXiv preprint arXiv:2102.07148, 2021.
Moming Duan, Duo Liu, Xianzhang Chen, Yujuan Tan, Jinting Ren, Lei Qiao, and Liang Liang Astraea: Self-balancing federated learning for improving classification accuracy of mobile deep learning applications. In 2019 JEEE 37th International Conference on Computer Design (ICCD), pp. 246-254, IEEE, 2019.
Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Personalized federated learning: meta- learning approach. arXiv preprint arXiv:2002.07945 2020.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. International Conference on Machine Learning, pp. 1126-1135 PMLR, 2017.
Sebastian Flennerhag, Andrei Rusu, Razvan Pascanu, Francesco Visin, Hujun Yin, and Raia Hadsell. Meta-leamiin with warped gradient descent. arXiv preprint 1rXiv:190.000025 2019.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249-256. JMLR Workshop and Conference Proceedings, 2010.
Filip Hanzely and Peter Richtárik. Federated learning of mixture of global and local models. arXiv preprint arXiv:2002.05516 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-1034, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729-9738. 2020.
Andrew Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 1704.000704.4461.,, 2017.
Shaoxiong Ji, Teemu Saravirta, Shirui Pan, Guodong Long, and Anwar Walid. Emerging trends in federated learning: From model fusion to federated learning, 2021.
Yihan Jiang, Jakub Konečný, Keith Rush, and Sreeram Kannan. Improving federated learning personalization via model agnostic meta learning, arXiv preprint arXiv:1909, 10111999991...... 2019.
Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition. In International Conference on Learning Representations, 2019.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Viraj Kulkarni, Milind Kulkarni, and Aniruddha Pant. Survey of personalization techniques for federated learning. In 2020 Fourth World Conference on Smart Trends in Systems, Security and Sustainability (WorldS4), pp. 794-797. IEEF, 2020.

---

Published as a conference paper at ICLR 2022
Yoonho Lee and Seungjin Choi. Gradient- based meta-learning with learned layerwise metric and subspace. In International Conference on Machine Learning, pp. 2927-2936 PMLR, 2018.
José Lezama, Qiang Qiu, Pablo Musé, and Guillermo Sapiro. Ole: Orthogonal low-rank embedding- plug and play geometric loss for deep learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. $109-8118, 2018.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. 2018.
Federated optimization heterogeneous networks. arXiv preprint
Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. Ditto: Fair and robust federated learning through personalization. In International Conference Machine Learning, pp. 6357-6368. PMLR, 2021.
Paul Pu Liang, Terrance Liu, Liu Ziyin, Nicholas B Allen, Randy P Auerbach, David Brent, Ruslan Salakhutdinov, and Louis-Phillnge Moreney. Think locally, act globally: Federated learning with local and global representations. arXiv preprint arXiv:2001.0155 1,1123,, 2020.
Mi Luo, Fei Chen, Dapeng Hu, Yifan Zhang, Jian Liang, and Jiashi Feng. No fear of heterogeneity: Classifier calibration for federated learning with non-iid data, arXiv preprint arXiv:2106.05001. 2021.
Yishay Mansour, Mehryar Mohri, Jae Ro, and Ananda Theertha Suresh. Three approaches for personalization with applications to federated learning. arXiv preprint arXiv:2002. 10619, 2020.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, et al. Communication-effiee learning of deep networks from decentralized data. In Proc. 20th Int't 'I Conf. Artificial Intelligence and Statistics (AISTATS), pp. 1273-1282, 2017.
2017.
Jachoon Oh, Hyungjun Yoo, ChangHwan Kim, and Se-Young Yun. Boil: Towards representation change for few-shot learning. In International Conference on Learning Representations, 2021.
URL
net forum? id=imm rMH
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems, 32: 8026-8037, 2019.
Aniruddh Raghu, Maithra Raghu, Samy Bengio, and Oriol Vinyals. Rapid learning or feature reuse? towards understanding the effectiveness of maml. In International Conference Learning Representations, 2019.
Andrei A Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osin- dero, and Raia Hadsell. Meta-learning with latent embedding optimization. arXiv preprint arXiv: 2018.
Andrew M Saxe, James I. McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv preprint arXiv:1312.6120, 2013.
Aviv Shamsian, Aviv Navon, Ethan Fetaya, and Gal Chechik. Personalized federated learning using hypemetworks. arXiv preprint arXiv:2103.04628. 2021.
Virginia Smith, Chao-Kai Chiang, Maziar Sanjabi, and Ameet Talwalkar. Federated multi-task learning. arXiv preprint arXiv: /705.
1101.710616 10467, 2017.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pp- 4080-4090 2017.
Canh T Dinh, Nguyen Tran, and Tuan Dung Nguyen. Personalized federated learning with moreau envelopes. Advances in Neural Information Processing Systems, 33, 2020.
Alysa Ziying Tan, Han Yu, Lizhen Cui, and Qiang Yang. Towards personalized federated learning. arXiv preprint arXiv:2103. 000000010....0000000000000000000 2021.
12

---

Published as a conference paper at ICLR 2022
Yonglong Tian, Yue Wang. Dilip Krishnan, Joshua B Tenenbaum, and Phillip Isola. Rethinking few-shot image classification: a good embedding is all you need? arXiv preprint arXiv:2003. 11539, 2020.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using I-sne. Journal of machine learning research, 9(11), 2008.
Kangkang Wang, Rajiv Mathews, Chloé Kiddon, Hubert Eichner, Françoise Beaufays, and Daniel Ramage. Federated evaluation of on-device personalization. arXiv preprint arXiv:1910.10252 2019.
Hongwei Yang, Hui He, Weizhe Zhang, and Xiaochun Cao. Fedsteg: A federated transfer learning framework for secure image steganalysis. IEEE Transactions on Network Science and Engineering, 2020.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? arXiv preprint 1.11:1441..792, 2014.
Haiyang Yu, Ningyu Zhang, Shumin Deng. Zonggang Yuan, Yantao Jia, and Huajun Chen. The devil is the classifier: Investigating long tail relation classification with decoupling analysis. arXiv preprint arXiv:2009.07022 2020.
Hui Zhang and Quamming Yao. Decoupling representation and classifier for noisy label learning. arXiv preprint (xXiv:0011008155, 2020.
Michael Zhang, Karan Sapra, Sanja Fidler, Serena Yeung, and Jose M Alvarez. Personalized federated learning with first order model optimization. arXiv preprint arXiv:2012.08565, 2020.
Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Federated learning with non-iid data. arXiv preprint arXiv:/806 180.00880000 2018.
Weiming Zhuang, Yonggang Wen, Xuesen Zhang. Xin Gan, Daiying Yin, Dongzhan Zhou, Shuai Zhang, and Shuai Yi. Performance optimization of federated person re-identification via benchmark analysis. In Proceedings the 28th ACM International Conference on Multimedia, pp. 955-963, 2020.
Weiming Zhuang, Xin Gan, Yonggang Wen, Shuai Zhang, and Shuai Yi. Collaborative unsuper- vised visual representation learning from decentralized data, In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4912-4921, 2021.
13

---

Published as a conference paper at ICLR 2022
A IMPLEMENTATION DETAIL
Our implementations are based on the code presented by (Liang et al. 2020 All the reported results based the last model. For computation costs, used single TITAN RTX and the entire training time of FedBABU on CIFAR100 using MobileNet takes ~2 hours when f=1.0 and r=1. Therefore, as ff decreases and increases, takes less time.
A.1 ARCHITECTURES
In our experiments, employ the 3convNet for EMNIST, 4convNet for CIFARIO, MobileNet for CIFAR100, and ResNet for CIFAR 100. 3convNet and 4convNet are described below. MobileNet (Howard et al. 2017) includes depthwise convolution and pointwise convolution, and ResNet (He
et al.
2016 includes skip connection 10 Please refer to each paper in detail.
3convNet(
(features): Sequential(
(0): Sequential
(0): Conv2d(1, 64, kernel_size=03 3), stride=(1. D. padding=(l, 1))
(1): BatchNorm2d(64, eps=le-05, momentum=0.1, affine=True. wtw.runnngg__tttsaaalll
(2): ReIU((
(3): MuxPool2d(kernel__.
stride=2. padding=0, dilation=l. coil_mode=False)
(1): Sequential
(0): Conv2d(64, 64, seneelssiee((' 3), stride=(1, D), padding=(I, 1))
(1): BatchNorm2d(64, ops=le-05, momentum=..1. affine=True.
(2): ReIU(()
(3): MaxPool2d (aaPool2(Kernee_ssizeee stride=2, padding=0, dilation=l, coil_mode=False)
(2): Sequential(
(0): Conv2d(64,
sixe=1_ 3), stride=(1, D), padding=(1,, 1))
(I): BatchNorm2d(64, eps=le-05, momentum=0.1, affine=True. stco___nnnnnn_ttt=aPPaa
(2): ReLU()
(3): MixPool2d(eerree_iiiaa stride=2, padding=0, dilation=l, ceil_mode=False)
(linear): Linear in features out f6atureee), bias=True)
4conyNet(
(features): Sequential
(0): Sequential
(0): Conv2d(3, =i2 3), stride=(1. 1), padding=(1, 1))
(1): BatchNorm2u(64, eps=le-05, momentum=0.11 affine=True, srack_running_staassttaa
(2): ReLU()
(3): MaaPolla@(eerael_siiez stride=2, padding=0, dilation=l, ceil_mode=False)
(I): Sequential
(0): Conv2d(64, 64, siznel_size==3.. 3), stride=(1. D. padding=(1, 1))
BatchNorm2d(64. eps=le-05, momentum=0.1. atfine=lrue. bock_rruning_statssaalll (2): R)LU()
(3):
stride=2, padding=0, dilation=l, coil_mode=False)
(2): Sequential
(0): C6nv2d(64, 64, kernel_sizo==3. 3), stride=(l, padding=(1,
(1): BatchNorm2d(64. ops=le-05. momeatum=0.1. affine=True. sts__ruunnng_tttttsaaaaae
(2): ReLU()
(3): keePoolld
= padding=0. dilation=l, ceil_mode=false)
(3): Sequential
(0): Conv2d(64, 64, Sernel_size=(? 3), stride=(1, D, padding=(1, 1))
BatchNorm22(64,, eps=1c-05, momentum=0.1. affinc=True. ctt__rroooo_ssttssaaaall (2): R)LU()
(3): MaxPoel2d kernlll size=2, stride=2, padding=0, dilation=l, ceil, eeil_mede=False)
(linear): Linear featuree 2266, out featureeee,,, hias=True)
http:///ihhhu.co////iaan//?//// Ge.eedvv
We use two architectures from http:///gthhb.omkuuuuggugrrrooee
14

---

Published as a conference paper at ICLR 2022
A.2 DATASETS
We used two public datasets, the CIFAR (Krizhevsky et al. 2009 [I and EMNIST (Cohen et al. 2017) 12 for performance evaluation. The composition of each dataset summarized in Table We applied linear transformations such as horizontal flipping and random cropping.
Table 8: Composition of CIFAR and EMNIST.
Dutaset
# Training
Validation
# Classes
CIFAR-10
50,000
10,000
10
CIFAR-100
50,000 731,668
10,000
100
EMNIST
82,587
62
A.3 HYPERPARAMETERS
For LG-FedAvg. FedAvg models corresponding to the FL settings in Table are used as an ini- tialization. It is then trained about a quarter of FedAvg training with a learning rate of 0.001. For FedRep, the number of local epochs for head and body updates were set to r and respectively. For Per-FedAvg, we used the first-order (FO) version, and the 3 value was set as the learning rate value corresponding to the communication round. For Ditto, the used to control the regularization term was set to 0.75.
A.4 EVALUATION OF FL ALGORITHMS
In FL algorithms, because the parts that need to be shared over all clients after federated training may not be shared due to client fraction, we evaluate these algorithms as follows:
Algorithm 2 Evaluation procedure of FedAvg, FedBABU, FedPer, LG-FedAvg, and FedRep.
1: initlist = !]
(If alg is FedAvg or FodBABU)
2: perlist= []
3: for each client E [l, N] in parallel do
4: (,,(0) - 82
5: Actini + (0)) D
(If alg is FodAvg or FedBABU)
6: initlist Aeciss)))
(If alg FodAvg FedBABU)
7:
0,(7)))) + Fine-tune( alg, (y; D(")
8: Accper + ( (Ty): D())
9:
anlist.append Aceppr)
10: end for
11: return initlist, perlist
12:
Junction FINE- TUNE(UUN, 0s, Tf)
13: It - ["C
for zuch fine tune epach 1.. do
15:
each iteration 1, ... I,
16:
0, (G,
(Il aig FedAAg FedBABU FedPer LG-FedAvg) (If alg is FedRep)
17:
0j,cto - (0)
18:
end for end for
19:
20:
for each itcration L. do
(If alg is FodRop)
21: 0reast (O.(0.) end for
22:
23:
return 6;
24:
end function
where 0 indicates the shared parts after federated training (i.e., full in the cases of FedAvg and FedBABU, body in the cases of FedPer and FedRep, and head in the case of LG-FedAvg). All notations are explained in the evaluation paragraph in Section 3 Because communication round is not related to evaluation, is emitted. For FedAvg and FedBABU, initial accuracy is also calculated. Although all clients have different personalized models because some parts are not shared after broadcasting except for FedAvg and FedBABU, it not completely personalized because there is common part 0K Therefore, we fine-tune all algorithms. FedRep trains models sequentially (Line 17 and 21), while others jointly (Line 16). Finally, the mean and standard deviation of initlist and perlist are reported initial accuracy and personalized accuracy in our paper.
hhttpss//www.cs..or// www.://wwwwwwwwwoooooo/oooo kriz/cifar.html
12 http:///wwwwotoooooooooooattttte
15

---

Published as a conference paper at ICLR 2022
B ORTHOGONALITY OF RANDOM INITIALIZATION
We hypothesize that the orthogonal initialization (Saxe al. 2013) on the head, through which the row vectors of the classifier parameters Oets are orthogo- nal, is the best for the body-only update rule. each update, the output of the extractor is pulled toward its label's row of Oat and pushed apart from the other rows of Octs through backpropagation. Therefore, orthogonal weight initialization can separate the out- puts of the extractor well based their class labels. Note that distribution- based random initializations, such Xaiver (Glorot & Bengio (010)) and He (He et al. 2015), have almost orthogonal rows because the orthogonality of two random vectors is observed in high dimensions with high probability (Lezama al. 2018).
0.6
0.3
Xevier-uniform (-0 (..00)2)
0.2
Ho-uniform (.000001) [default]
Orthagonal (-2.55e-10)
0.1
Similar (0.9967)
0
20
40
60
aD
100
120
140
160
Figure 3: Test accuracy curves according to the initialization method when the body is only trained in the centralized setting, The values in parentheses indicate the average of cosine similarities between row vectors.
Figure 3) shows test accuracy curves for many differ- ent initialization methods when only the body is trained in the centralized setting When the row vectors are initialized similarly, a convergence gap appears (green and red lines in Figure 3). as expected. More experiments the centralized setting are reported in |subsection F.I and Appendix G Based on these empirical and theoretical results, we used He (uniform) initialization, the default setting in PyTorch (Paszke et al. 2019), in our paper.
This characteristic can be proved. Xavier (Glorot & Bengio, 2010) and He (He et al. 2015) are representative distribution-base random initialization methods, controlling the variance parameters using network size to avoid the vannosiii gradient problem. Two initialization methods use uniform and normal distribution. In detail, each element follows Ul( (aa, a) or (0 o²) independently, where and Because we used the ReLU non-linear function in our implementations, we calculate a and a by multiplying v2, for scaling of ReLU non-linearity.
defined by non-linearity (such ReLU and Tanh) and layer input-output size.
Because we focus initialization the head, we specify Oct E Cxd, where C is the number of classes (i.e., layer output size) and d is the dimension of the last representation (i.e., layer input size). Namely, each element random variable (wa.)/ee(t,o)) (e(())))EE( follows (((=a,a) or (O, independently.
When Wij ~ U(-u, a), in Xavier initialization, a is defined as v2 x
T+C Similarly, in He
initialization, a is defined as v x or V2 v depending on which to be preserved, forward pass or backward pass. When Wi.j ~ No0, 2) in Xavier initialization, a is defined as v x V a+c.
Similarly, in He initialization, is defined v2 V or v2 V depending on which to be preserved, forward pass or backward pass. However, It is noted that a in the uniform distribution and a in the normal distribution do not affect proof on orthogonality, therefore we keep the form of ud-a,a a) and (00 o²) for simplicity.
We first consider 2d independent random variables (w (e.))) for any p * q [L.C]. Then, we can define d independent random variables [Wp.j W Je(iadd from the above 2d independent random variables. Then, for all EE
d). 2( = Elup.t p B[wg.s [ 0 and V(wp,
V [[rp.)][[[u,, 2 [[up.) [
[w | = V(Wp. V [Wag] because each element in Oats
follows 4d(a, a) or A/ o²) independently. Let S. Σd - "p Then, by the Weak Law of Large Numbers, lim4-~o P(SS, E[Sa]| > e) = lim S[[Sa] > E) = for all e > because [up. bej,,, are independent random variables and there is V such that V[wp, W...] < V for all j.
13 All initialization methods except for 'similar' are provided by PyTorch (Paszke et al. 2019). Here, "similar" initialization is implemented using uniform distribution [0.45, 0.55], and then each row vector is unitized by dividing the norm corresponding the row vector.
16

---

Published as a conference paper at ICLR 2022
C PERSONALIZATION OF THE CENTRALIZED MODEL
In (Jiang et al. 2019), they showed that localizing centralized initial models harder. Similarly, we investigate the personalization of the centralized model under different heterogeneity. First, train two models with 10 epochs, one is entirely updated (F in Table 9) and another is body-partially updated (B in Table 9) using all training data set. Then, the trained model is broadcast each client and then evaluated. Table 9 describes the results of this experiment. It is demonstrated that models updated mody-partially, then personalization ability does not hurt even under centralized training.
Table 9: Personalization of the centralized models. F is trained entirely, and B is trained body-partially in the centralized setting.
Fine-tune epochs (ry)
Model
0 (Initial)
F
40.75
40.77
41 4..80.14.66
42.43.14.78
43.32.14.72
44. 14..5114.61
100
39.64++499
43.37-t5.41
47.66-t:.08
50.16:t:3.24
51.09:+:3.00
52.155+5.04
F
37.92=5544
38.93--5.52
40.68::548
42,40-15533
43.86:25.39
45.27=5543
50
B
37.18
43.66.15.34
51,14
$4.67
$6.34.1488
$7.36.15.21
25.46++625
28.04-16.88
33.43-+7.32
39.56+77.66
44.74+t7448
50.32++1.05
10
23.86115.08
48.63:15.39
69.144:5.7
74.321:5.35
75.90
76.47
D EFFECT OF MOMENTUM DURING LOCAL UPDATES ON PERFORMANCE
We investigate the effect of momentum during local updates on the performance of FedAvg and FedBABU. Table 10) describes the initial and personalized accuracy according to the momentum during local updates. The momentum for personalization fine-tuning is the same as the momentum in federated training. In both cases of FedAvg and FedBABU, appropriate momentum improves the performance, especially personalized accuracy. However, when the extreme momentum (m=0.99) is used, FedAvg completely loses the ability to train models, while FedBABU has robust performance.
Table 10: Initial and personalized accuracy according to momentum (m) during local updates under a realistic FL setting (N=100, f=0.1, and 7=10).
Initial accuracy
m
100
50
10
FedAvg
FedBABU
FedAvg
FedBABU
FedAvg
FedBABU
0.0
26.07
26.38:43.98
24.71::4.15
24.72+t4.67
15.44:7.11
13.6646..15
0.5
26.63114.61
27.01
26.27:14.87
27.58:1475
16.60.16.14
17.98
0,9
18.18
29.38114.74
27.34:14.96
27.91
14.40
18.50
0.99
19.84
30.62:14.60
1.00=1.10
30.01
1.00::3.32
15.68
Personalized accuracy
m
100
50
10
FedAvg
FedBABU
FedAvg
FedBABU
FedAvg
FedBABU
0.0
28.46.14.05
30.24:1:.39
30.02.14.65
33.43-L5.26
40.80-19.00
59.85-1744
0.5
30.44
32.70:::465
33.64.14.78
38.65:1..14
54.32-18.33
67.3411.32
0.9
33.13:1:.2
35.94
38. 38.09::3.17
42.63:13.59
62.67:16.52
66.32-17.02
0.99
23.88:14.44
34.17:14.75
1.12-1448
42.17::3.2
1.00:::.32
61.02-17844
E EFFECT OF MASSIVENESS ON PERFORMANCE
We increase the number of clients from 100 to 500, and then each client has 100 training data and 20 test data. Table II] describbs the initial and personalized accuracy of FedAvg and FedBABU on CIFAR 100 under various FL settings with 500 clients. The performance of FedAvg is slightly better than that of FedBABU in many cases; however, FedBABU overwhelms FedBABU in the case of extreme FL setting (s=10, 1=0.1, and 1=10). More interestingly, if each client has a few data samples, then local epochs 7 should more than 1. It is thought that massiveness makes data size insufficient on each client (when the total number of data is fixed) and brings out other problems.
17

---

Published as a conference paper at ICLR 2022
Table 11: Initial and personalized accuracy of FedAvg and FedBABU on CIFARI00 under various settings with 500 clients. The used network is MobileNet.
FL settings
Initial accuracy
Personalized accuracy
T
FedAvg
FedBABU
FedAvg
Bc
13.71
11.85
16.12.17.99
15.00
1.0
4
18,98
16.79-8.38
21.1719.005
19.9411864
15.70.18.18 10
15.66.18.23
17.54.1891
18.47
100
1
14.59.17.72
10.53-17.25
16.46.18.27
13.90.17.96
0.1
4
18.90.18.33
17.36.18.85
20.88
20.30.19.27
16.79.17.90 10
14.38
18.22.18.37
16.76-18.33
1
13.67:t8.05
11,04=7.01
19.29:19.31
17.39-8.11
1.0
18.36-
16.60-8843
22.76-t:9.5
23.15-19.51
15.95-18.57 10
14.81
19.57
21.78
50
14.58
10.72:16.81
19.65:19.14
16.05-18.11
18.28+t8.79
16.79+9904
22.39-+-9778
23.57++9.79
10 15.79+
15.07+
18.52++8.69
20.83++-8.00
1
9.91
7.41
37.62.111.76
35.02:110.71
1.0
15.34.8.49 4
12.67-7.558
19.91
49.07
12.9917.98 10
12.93
35.47-11129
48.07
10
10.96.17.22
8.42.L6.58
38.951111.6
35.13
0.1
15.03.1..27
13.99
37.92.111.64
19.99.111.94
10
12.39
13.29
32.41
19.83
F RESULTS OF 4CONVNET ON CIFARIO
F.1 ACCURACY CURVES IN THE CENTRALIZED SETTING
When 4convNet is used on CIFARIO, a similar tendency appears. SraparingFiguree 2 and Figure the importance of learning representation is emphasized more when tasks are difficult. In addition. Figure 3] shoows the necessity of orthogonality when the body is only trained and approximation of random initialization to the orthogonal initialization using 4convNet on CIFAR10.
0.8
0.7
0.6
0.70
Xavier (norma))
0.5
0.65
Xavier (uniform) (normal)
Wh-my Mm/ M/ /
0.60
(unifarm)
Orthogonal
body
0.55
Similar
40
60
100
120
140
0
20
40
60
80
100
120
140
160
Figure 4: Test accuracy curves of 4convNet CIFARIO according to the update part in the centralized setting.
Figure 5: Test accuracy curves of 4convNet on CIFAR10 according to the initialization method when the body is only trained.
F.2 EXPERIMENTAL RESULTS
We also evaluate algorithms on CIFARIO using 4convNet. Table 12 and Table 13 are the experiment results corresponding to the Table 3 and Table in the main paper, respectively. The tendency of performance comparison and the superiority of our algorithm appears similarly when using 4convNet CIFAR10.
18

---

Published as a conference paper at ICLR 2022
Table 12:
Initial accuracy of FedAvg and FedBABU according the existence of classifier on CIFAR1O under various settings with 100 clients. The used network is 4convNet.
FL settings
FedAvg
FedBABU
w/ classifier
w/o classifier
w/ classifier
w/o classifier
68.89 6.339
74.411.622
71.50.17.35
75.0011.46
1.0
6..56.16.84
66.17447.44
68, 6.667
71.77:17.29
10
60.06.1:5.68
65.21
64.66.16.00
69.04-16.68
63.79911.55
69.48.16.72
69.36.1..61
71.21
0.1
59.65.1..03
66. 11:3.77
66.80
69.27-18.05
10
67.32:16.76
63.19.16.81
6..68.1
67.58.16.42
66.56:19.39
80.73:18.27
68.54::8.45
80.53:77.99
1.0
52.60
70.53-19915
63.79:19.12
78.3441:8.30
8..55 10
71,03-+-8.77
61,10::99.80
77.13+17.10
5
8..1
71.27=10.03
63.44119.82
77.64-18.26
0.1
8.57
68.64 11,36
89.93+t8.32
74.61-t8.81
10.52 10
67.07++8.77
55.55
73.29-+-9.25
19.47-115.16
68.60.16.96
14.61
88. 38.34-17..05
1.0
4
41.08
83.87=10.92
55.43
88.80118.57
39.47 10
82.41
48.52
86.33.18.93
2
40.48 114.9
11.43
57.22
118.31
0.1
26.72
78.56
83.12.19.68 36.57=11.23
14.39
12.39
10.94
84.37
Table 13: Personalized accuracy comparison CIFARIO under various settings with 100 clients.
The used network is 4convNet.
FL scttings
Personalized accuracy
FedBABU (Oun)
FedAvg
FedPer
LG LG-FedAvg
FedRep
Per-TeedAgg
Dits
Local-Only
80.52-t:351
79.72-t5.4
78. 78.51::500
80.37+:312
71 11.96M
79.89-531
79.89
79.30-1567
13.35-t6.15
71.04+1.50
74.75
72.10-b.37
77.99-t6.4
10
10
76.86:1500
71.03:1..66
69.25
71.91
60.27
66.52.2744
12.88.16.11
55.40=11.78
78.80:1:5:1
73.05.15.04
77.22
67.951:000
7:.21
77.23
75.27.16.44
0.1
77.45:26:23
71.06:6..1
71.13:6778
64.17::229
68.7826691
67.324759
71.49:b6.4
76.25:b6.16
69.11::5m
66.43:669
61.1527332
60. 60.70:1576
61.22:24114
64.97:t707
85.17:16.43
83.82:16775
84.50:2::00
85.041::35
78.98
74.9611735
82,6411685
83.65::5599
77.47:28.22
76.96:22666
17.54:2:.14
17:1771011
83,77:b6.93
5
83.48.16.11
77.01
76.17.2.11
77. 77.28.17.53
70.62
53.31 12:12
79.01
68.79-1334
82,766:6..m
75.08
80.69::763
63. 30:t
83.71
72. 15-15:2
72.08-t993
0.1
81.31 2::11
94
74. 14:
19.17:t:0.18
77.64-19.51
32:32-111
66, 66.36:1-..5
10
80.99:::764
74,29 2.
74.92=22322
19:b11133
68.71:2:047
47. 48 13.70
18.66112251
91 91.41:bb.59
91 91.40:hh.12
91. 18
92.38-:5:4
91. 11:61:
72. 12.02
91 91-41t
1.0
91.24::7.33
88.91 28.91
89,14:2844
S8 9766779900
87, $7.742200
43.41
92,17:b642
2
90.53-4-220 20
88.40 48.48
15:99.9
$7.38
86. 19:
36. 36.42-1839
90.15
90.85.9.100
90.98:16
86.36:110.57
90.33:::.52
68 68.63:11434
91-88
63. 63.28=11377
64.16:21812
87.85-1:.97
10.72
47.00-E:24.5
1966
54.69::17.67
10
88..9311.33
10.7
86.69.28.04
33. 23.30.1:22.9
83.85
26.61=1953
43.641:21.00
F.3 ABLATION STUDY ACCORDING TO THE LOCAL UPDATE PARTS
We decouple the entire network in more detail during local updates to investigate whether the body should be totally trained. For this ablation study, we used 4convNet on CIFARIO. Table 14 and Table 15 show the initial and personalized accuracy according to the local update parts, respectively. For consistency, all cases are personalized by fine-tuning the entire network in this experiment. In all cases, FedBABU has the best performance, which implies that the body must be totally trained at least when using NeonyNet. It is believed that the same (body-totally) or similar (body except for a few top layers) trends appear when large networks are used.
19

---

Published as a conference paper at ICLR 2022
Table 14: Initial accuracy comparison on CIFARIO under various settings with 100 clients. The used network is Neo.
settings
Local update parts
Conv 1234 (FedBABU)
Conv 2ww1123++innur
f
Conv1
Conv12
Conv123
(FedAvg)
25.22:18.34
47.23:16.91
68.44:26.55
71.50=77.35
68.8 28.89
1.0
23.02:1:957
45.73:17.55
60.25:17.69
68.40:b6.67
61.56:26.84
17.80
42.51:1.00
60.48:16696
64.66:26.00
60.06:15.66
10
19.33
42.84++815
62,71 1:77.
69.36+:561
63.79-+7.59
0.1
4
19.85 8:99
41,58-+x.3
55,26-t6668
66.80
59.65
10
18.99:9..10
44.38:17.9
54.61=6.66
62.68-7706
57.32:16.76
23.42
44.35-t9.59
10.47
68.54
66.56:t9.39
1.0
22.34
40.07:18.3
54.5019..1
63.79-19.12
22.60.17.779
18.94
39.59:19:03
51.25
61.10=99.00
51.55
5
20.78:17.90
38. 10.00
52. 899
63 1.
55. 13.13:1
0.1
19.33:1 10.59
19.71
17.16
59,9 19.93
48.77
23.71++8.86
36.74-t9.72
46.97+-9.11
55.55
16:32-+10:5
18.72-t
45.01-t13.57
18.86-11598
63.51 1:.661
59.47-115.16
1.0
4
20.08= 10.09
34.58+11.44
10.56
15.43
41.08=13.42
14.51 10
29.74+11084
17.05
48.52++1.47
39.47=13.44
16.17
37.08112245
13.57
57.22:
40.48-114.99
18.82
28.75::14.1
44.39-16.49 13.16::1927
25.12415..3
26.72-14554 36.57:11.23
17.4001110.53
23.73117887
22.29
Table 15: Personalized accuracy comparison on CIFAR10 under various settings with 100 clients. The used network is 4convNet.
FL settings
Local update parts
Conv 1234
Conv 1334-LLar
f
Conv
Conv 12
Conv 123
(FedBABU)
(FedAvg)
55.94
64.88118.34
78. 58.74
80.52-15.51
79.72=15.24
1.0
4
53.06-17.17
62.84:1..57
71.85:6.111
79.30-1567
73.35
52.8917.144 10
62.02:16448
71.16:16.24
5..86:1..08
71.03:15.86
10
53.85.18.01
64.40:16.17
72.79-1646
78.80
73.05:15.94
0.1
50.75-tx.21 4
61.79-t6.84
66.45+
77.45
71.06
51.95:17445 10
62.72:17.64
67.41:6631
76.25:16.16
69.1
67.49 19974
74.39:17.74
82.55
85.17:1643
83.82-16.75
1.0
66. 10.02
71.77++8.48
79.68+6.77
83.65-55.99
77.47-8832
64.87-8..1
69.73-1.110
76.25
83.48-66.13
77.01
5
65.50=19.00
74.50
74.50+t..7
82.76=6.44
75.08
0.1
4
65.05
70.80:19.29
8..44:18.06
$1.31:77.17
74.87
65.61 1:8.73 10
69. 19.66.18.688
74.79
80.99
74.29
85.44.19.20
88.20-18.86
90.61-26448
91.41
91.40:16.12
1.0
84.46.19.30
83.94:110.18
88, 88.8618862
91.24417.33
88.91119.0
81,62.19966
86.20:110.72
88.58119.1
90.53
88.48.1844
2
85.46-1-9.68
85.78+
88. 8.41
90.98
16.36-110557
0.1
4
82.74-10..4
85.13
85,9111000 10.87
87, $.58
85, 10.94
10
82.52+
84.98-1-9.25
85,46-19.74
88.93
15.14
G RESULTS OF RESNET18 AND RESNET50 ON CIFAR100
G.1 ACCURACY CURVES IN THE CENTRALIZED SETTING
We experiment with ResNet18 and ResNet50. In the centralized setting, test accuracy curves according to the update part (Figure 6 and Figure 8) and according to the initialization method (Figure 7 Figure 9) have the same trend we have seen before.
20

---

Published as a conference paper at ICLR 2022
0.8
0.8
0.6
0.6
full
0.4
head
Xavier (normal)
body
Xavier (uniform)
0.3
He (normal)
0.2
He (unifor) [default] Orthogonal Similar
0.1
40
60
80
100
120
140
160
20
80
100
120
140
160
Figure 6: Test accuracy curves of ResNet18 8 on CIFAR 100 according to the update part in the centralized setting.
Figure 7: Test accuracy curves of ResNet on CIFAR 100 according the initialization method when the body is only trained.
0.8
0.8
0.7
body
0.6
0.4
0.4
Xavier (normal)
0.3
Xavier (uniform)
0.3
He (normal)
0.2
He (uniform) [default] Orthagonal
0.2
0.1
Similar
120
140
160
Figure 8: Test accuracy curves of ResNet50 on CIFAR100 100 according to the update part in the centralized setting.
Figure Test accuracy curves of ResNet50 on CIFAR 100 according to the initialization method when the body only trained.
G.2 EXPERIMENTAL RESULTS
Table 16 and Table 17 are the results of ResNet18 and ResNet50, respectively. It is shown that the performance gap between FedAvg and FedBABU increases when ResNet is used rather than when MobileNet (in the main paper) is used. Furthermore, it is observed that as the complexity of the model increases (ResNet18 + ResNet50), the performance of FedAvg decreases, while that of FedBABU does not.
Table 16: Initial and personalized accuracy FedAvg and FedBABU on CIFAR100 under various settings with 100 clients. The used network is ResNer18. is 0.1.
FL settings
Initial accuracy
Personalized accuracy
FedAvg
FedBABU
FedAvg
FedBABU
57.26:4471
59.87-14.27
60.39
66.48
100
4
44.79
49.02-15.01
49.32-44.0
55.65-1481
10
34.52+4473
40.60-1..62
39.04-14.77
47.42-4:.66
53.731+4.97
59.23
63.214:5.33
71.16::4.75
50
4
42.47:15.11
49.03=14.56
52.764:5.99
62.544
37.09+4420
40.71
47.13-44.47
54.99+55.77
42.88-18.08
53.65
76.84:17.08
83,8315522
10
29.15:18.18
39.76.18.23
67.29-L6.42
77.40
10
26.54.1766 30.96
66.18
72.49:16.19
21

---

Published as a conference paper at ICLR 2022
Table 17: Initial and personalized accuracy FedAvg and FedBABU on CIFARI00 under various settings with 100 clients. The used network is ResNer50. f is 0.1.
FL settings
Initial accuracy
Personalized accuracy
FodAvg
FedBABU
FodAvg
FedBABU
42.89115.34
60.78:14.74
47.591.5.1
65.74.14.72
100
4
33.59.4435
50.71
39.56.14443
56,35.14.43
10
32.88 40.59
37.32.14.47
45.52.1:.29
42. 5:7
59,51
53.04
70.58+:5.06
50
4
30.97 48.65=#4.44
42.76-t4995
59.644$577
30.90:14.76 10
4..14
43.28-1..49
52.13
28.18 18.999
51.17=6697
65.50+1..2
81,43::6.20
10
19.31
38.45
58.92-7740
73. 63.86-66.96
10
18.41:7.22
30.22-18.26
59.5011..0
68.42:48.04
H RESULTS OF RESNET10 ON DIRICHLET DISTRIBUTION-BASEI CIFAR100
We evaluate ours and existing algorithms under unbalanced and non-IID settings derived from the Dirichlet distribution. Figure I0 and 11 describe the distributions of train and test data points and non-IID depending on 3, which is the hyperparameter of the Dirichlet distribution. (Table 18 describes the results on Dirichlet iistribution-based non-IID CIFAR 100 in the realistic FL setting (/=0.1 and 1=10). The lower 3 implies the larger heterogeneity.
Figure 10: Data distribution (3 = 1.0).
Figure 11: Data distribution (3 = 0.5).
Table 18: Personalized accuracy comparison on Dirichlet distribution based non-IID CIFAR100 with 100 clients (FI setting: 1=0.1, and 1=10). The used network is MobileNet.
Personalized accuracy
FedBABU (Ours)
FedAvg
FedPer
LG-FedAvg
FedRep
Per-FedAvg
Ditto
Locul-only
1.0
34.92-1622
32. 10
34,21 4:568
25.00 25.00-66122
12.68-527
18.8544497
7.11
23.73::5711
0.5
44.6715.80
40.211566
35.62:16.17
27.24
12.5214432
18.3615.001
8.13:4469
31.2411:.19
I RESULTS OF 3CONVNET ON EMNIST
Table 19 describes the results on EMNIST using network that consists of three convolution layers and linear classifier. We set the number of users to 1488, and the number of data points per user was approximately 450. We fixed the fraction ratio 0. and local epochs at 10 (which is the realistic setting used in our paper). The results demonstrate that FedBABU also has the best performance on EMNIST.
Table 19: Personalized accuracy comparison on EMNIST with 1488 clients (FL setting: f=0.1, and 1=10). The used network is 3convNet.
Personalized accuracy
FedBABU (Ours)
FedAvg
FedPer
LG L.GFeeAAg
FedRep
Per FedAvg
Ditto
Local- Only
60
85.27-1133
84.524t442
54.68-26.77
82.75:b478
2
78.45-1222
83. 93.38:is5.4
92
89.30-1500
$5.46:155116
58.72-28.0
81.6441580
82.03-5.34
80.61-602
83.23:1:606
81.63:1577
95.96-t5.0
89.83-12.3
15.86-
73.50++1300
94.61+
14.94-11486
12.27
9:.066
22

---

Published as a conference paper at ICLR 2022
J RESULTS ON IN-DISTRIBUTION (ID) AND CLASSES
(OOD)
To investigate the accuracy of in-distribution (ID) and out-of-distribution (OOD) classes, scatter IID-like test set (s=100) to all clients. Namely, some classes are in test data set (s=100), but not in training data set (s=50 s=10). Table descriibs the results of FedAvg and FedBABU on this experiment. Here, in-distribution accuracy is the test accuracy of the classes present in the training data set; otherwise, out-of-distribution accuracy. Before personalization (Before in tables), FedBABU beats FedAvg from both ID and OOD. After personalization (After in tables), FedBABU becomes fit strongly for the ID classes than OOD classes. The OOD accuracy of FedBABU decreases more than that of FedAvg, but the accuracy of FedAvg simiarly drops after personalization. These results indicate that no personalization is the best strategy for both FedBABU and FedAvg. Therefore, because the OOD accuracy of FedBABU is higher than that of FedAvg before personalization. it is concluded that FedBABU has better performance than FedAvg when the optimal strategy for OOD is used.
Table 20: In-distribution (ID) and dutto--disttibution (OOD) accuracy of FedAvg and FedBABU before/after personalization under various settings with 100 clients. The used network is MobileNet.
FodAvg
FedBABU
FL settings
ID accuracy
OOD accuracy
ID accuracy
OOD accuracy
$
Before
After
Before
After
Before
After
Before
After
47.30 47.30+7705
55. 15:1577757
46.46:2674
32.85+5.677
48. 48.30+2755
57.92=7798
46.81
24.06+:5.2 27
1.0
37.30=8836
45,30=881
37.24:1.29
27.05
38.992173
49.70=28.00
31.76:13711
13.88:1478
10
30.98.177:57
38.46:19.04
30.54:13.8
10.14:::.18
27.63:1741
37.40
29.69116.24
5.72.13.52
50
39.90:2::55
47.0627.96
39.06226.17
28.32:4:.76
42.98::8200
53.35
42.14:27.99
16.04:1:.08
0.1
33.23:17788
41.25:18.19
35.70:16.1
27.38.1563
37.07:18002
47.04:19.18
36.34.16.100
11.67:1432
10
28. 77
36.32-47411
27.15:t6.00
18.05-45.44
28.26-18.44
38.74-t9088
29.06-
6.70-42.94
42.16:1:8.8
78. 66
41.51:1:.12
14.25
45.76:11641
19.60:11135
44.93:1...1
3,93:1
1.0
30 15.77
68, 18.38-11614
31.43-+-5.00
7.13++9.10
38.43++1227
71 71.59-1520 29
37.05-55.1 19
0.13-20.36
10
26.30.116.40
61.21:117:16
23.70:14.50
2.27:11.48
27.20:11:12
64.21:11224
25.49:1444
0.00:10:00
10
14.47=#1800
69. 14:87727887
34. 14.10:44.8)
9.744:2.5
42.57-219.11
75. 18:
39.64-15.04
3.324
0.1
26. 16.866216000
14.24=16.45
2..09
6.03:t2.73
31.31-17227
11.38=15.55
30.75-:5.23
0.06::0.24
10
18.42::1228
16.21
19.67:3..9
2.95:1192
22.30214465
64.73:11.116
22.466:2.71
0.00:10:00
K SIMILARITY BETWEEN CLIENTS DURING FINE-TUNING
We investigate the cosine similarity
Bicitaa condaa SEAT
between different clients during 20
fine-tuning epochs, described in Fig
ure 12 During fine-tuning for per-
sonalization evaluation, the entire net- work is updated in the cases of both FedAvg and FedBABU. From the re- sults, it is found that classifier (black line in Figure 12p closely related to personalization. In other words, all clients have similar extractors during fine-tuning (i.e., large cosine similar- ity), while make different classifiers based on their own data. Furthermore, it is observed that the cosine similarity on the head between different clients decreases more rapidly in the case of FedBABU than FedAvg. It is believed that this is because the classifier of FedAvg starts personalization from the federated convergent classifier, while that of FedBABU starts personalization from the initialized orthogonal classifier. This characteristic explains rapid personalization, introduced in Section 5.2.4
Figure 12: Layer-wise cosine similarity of FedAvg and Fed- BABU trained under the realistic FL setting (f=0.1 and 1=10) during fine-tuning. The blue-, green-, and red-like lines represent low-, middle-, and high-level convolution lay- ers. respectively. The black line represents the last linear layer (i.e., head).
23

---

Published as a conference paper at ICLR 2022
L FEDBABU WITH BODY UPDATE ON THE SERVER
Table 21: Initial and personalized accuracy of FedBABU on CIFAR100 under realistic FL settings (N=100, f=0.1, 1, and ==10) according to the which is the percentage of all client data that the server also has. Here, the body updated only on the server using available data.
8=100 (heterogeneity
50
10 (heterogeneity f)
Initial
Personalized
Initial
Personalized
Initial
Personalized
0.00
29.38=4.74
35.94:1:.06
27.91 +:5277
42.63-1...59
18.50
66.32:1:.002
0.05
28.68
36.25
26.94
41.01
21.32::5.85
66.56:17.24
0.10
32.49
39.93:14.76
31.39::4.84
47.02
24.34::5.32
68.95=16.92
Let us repeat the experiment in Section again, where the server has small portion p of the non- privacy data of the clients. Table 2I describes the initial and personalized accuracy of FedBABU when the global model is body cody-partially updated on the server using available data (such as experiments (B) in Table 2) FedBABU with body updates on the server improves personalization compared to FedAvg with body updates on the server, maintaining the initial accuracy (refer to (B) in TTall 2).
This result demonstrates that training the head negatively affects personalization even when trained locally. In addition, the performance of FedBABU improves as p increases. This implies that there is room for enhancing the representation beyond that of FedBABU,
M BODY AGGREGATION AND BODY UPDATE ON THE FEDPROX
Even when all clients are active every communication round (i.e., f=1.0), the personalized perfor- mance of BedProx+BABU beats not only that of FedAvg but also that of FedBABU.
Table 22: Initial and personalized accuracy of FedProx and FedProx+BABU with μ of 0.01 on CIFARI00 with 100 clients and f of 1.0.
s=100 (heterogeneity. .)
8=50
8=10 (heterogeneity f)
Algorithm
Initial
Personalized
Initial
Personalized
Initial
Personalized
42.25::4.58
56.22:14.54
52.08::499
59.95114.99
44.39-1755
78.9616.16
FedProx
40.32+t4.00
43.17
45.99-t6.05
29. 29.73-16.74
67.66+
10
30.75+t4.3
33.43-t4.48
29.44-t4.24
35.655+4.60
19.63:t..73
57.83--7.13
45.02+t:.42
61.48
54.13-t4.86
66.54:1:.66
50.42-19.63
83.75-15.97
FedProx
39.01
46.31
38.79-55.15
52.96=15446
34.09-17.28
77.28+:5.6
+BABU
10
28.69-14.64
36.29:14.45
30.39
44.46:15.49
25.16.15.84
69.77-16.6
N DISCUSSION ON THE EFFECTIVENESS OF FEDAVG
MOCHA (Smith al. 2017) a representative personalized FL paper using regularization based on relationships between tasks, pFedMe Dinh et al. (020) and Ditto (Li et al. 2021) train local models with regularization based the divergence between global model and local models. If the weight on regularization is set to zero, these regularized personalized FL algorithms reduce to the local-only algorithm; on the other extreme, these reduce to the FedAvg algorithm (i.e., 81 personalized FL algorithms such as FedPer (Collins et al., 2021]. in which each client has parts that are not aggregated (i.e., personalized parts), can be explained similarly. If there is no personalized part, these personalized FL algorithms reduce to the FedAvg; the other extreme, i.c., if the entire network is not aggregated, these personalized FL algorithms reduce to the local-only algorithm. In summary, personalized FL algorithms are in in-between local-only and FedAvg algorithms and have better personalized performance than both local-only and FedAvg in general.
0x). The
In the same vein, FedAAvgFFine-tniig is also in-between local-only and FedAvg because this algorithm uses two extreme algorithms sequentially (i.c., FedAvg followed by local-only). However, intuitively, FedAvg=Fine- a bit more FedAvg-based, whereas personalized FL. algorithms
24

---

Published as a conference paper at ICLR 2022
bit more local-only-basee This is because FedAvg+Fine-tung tungenuinnnnnn is the same as FedAvg before personalization, whereas personalized FL algorithms develop different clients' models from scratch like local-only. Therefore, we believe that FedAAy+Finee-iin for representation) from federation than personalized FI. algorithms from this difference (FedAvg- based v.s. local-only-basede An additional difference between FedAvg+Fine-tunin and regularized personalized FL algorithms is that FedAvg+Fine-tuning optimizes clients separately using their own data set based on the well-federated extractor, whereas regularized personalized FL algorithms optimize clients jointly from scratch. In other words, FedAvg+Fin -engg++---nuniinnn can easily optimize all clients' models based on their own data sets. However, it is too difficult to optimize all clients' models simultaneously. Therefore, regularized personalized FL. algorithms.
eams more benefits (particularly
can achieve better personalized performance than
Furthermore, very recent work (Cheng et al. 2021) |) has addressed the effectiveness of Fedevg-FFnne tuning theoretically. In Cheng et al. (2021), ) they compared local-only (zero collaboration), FedAvg (zero personalization) (McMahan al. 2017), and pFedMe (T Dinh et 2020)) in the high-dimensional asymptotic limit by analyzing Mars-naiance. They demonstrated that the asymptotic test loss of FedAv-+Fineetuning (FTFA in Cheng al. (2021) matches that of Per-FedAvg and that the asymptotic test loss of FedAvg+Fine mene-ttning with a ridge regularizer (RTFA in Cheng et al. (021)) matches that of pFedMe their stylized linear regression model.
Per-FedAvg (Fallah et al. 2020),
Although Cheng et al. (2021) thoroughly addressed this problem their stylized linear regression model, there are few results on real data sets except for test accuracies. It is believed that "why TedAvv--iine-tuinng effective" itself needs be studied more using real data sets, as "why MAML Finn et al. 2017) is effective" has been addressed in (Raghu et al. 2019).
O FEDAVG WITH DIFFERENT LEARNING RATES
Table 23: FedAvg with different learning rates under realistic FL. setting (/=0.1 and r=10). We set the body's initial learning rate (at) 0.1.
8
FedAvg (0x=00))
FodAvg (on=0.1 x oz)
FodAvg (a,,0001 x as)
FedBABU (ax=0)
100
24.34
27.10.14.43
28.37
29.36
50
33. 1..00.088
35.97....1
36.2 1.5008
16..771.,.37
10
50.25
52.96.17.52
54.04417.60
54.93
What we want to emphasize is the importance of making all clients have the same class boundary by freezing the head. We argue that this shared classifier enhances the representation power of the federated model, which is an important factor for personalization. From this perspective, if the head moves even a little bit client by client, the different class boundaries to train feature extractor are used during local updates. Therefore, the federated model's feature extractor might hurt.
To verify this hypothesis, we experiment using different learning rates depending on the parts. The above table describes the federated model's 's accuracy without classifier (i.e., w/o classifier accuracy in Table 3) to investigate representation power of models. We set the body's initial learning rate as 0.1, the fraction ratio as 0, and local epochs 10. ag and as are the head's learning rate and the body's learning rate, respectively.
From this result, we believe that the federated model's representation power increases as the head's learning rate decreases, i.c., as a learning signal for the body is larger. In this trend, we highlight that the best performance is due to the same classifier on all clients.
P CLASS-WISE ANALYSIS DURING FEDERATED TRAINING
To explain why FedBABU has better performance than FedAvg after federated training (i.e., before fine-tuning), we analyze the federated training procedure of FedAvg and FedBABU wisely under the realistic federated settings 1 and 7=10). For this analysis, assume that each client has 500 training data and 10,000 test data (i.c., the whole test data) of CIFAR-100, Test data set is divided into groups for each client; in-class test data and out-of-class test data. In-class test.
25

---

Published as a conference paper at ICLR 2022
30
F
Fedlep Bacas
TedDAOU
20-
55
100
200
253
53
300
239
380
106
200
(a) 8= 8=100 (in-class).
(b) 8=50 (in-class).
(c) 10 (in-class).
FedSAOL
55
(d) s= 100 (out-of-class).
(c) 8=50 (out-of-class
(1) s=10 (out-of-class).
Figure 13: In-class and out-of-class test accuracy curves according to the heterogeneity. The models trained under the realistic federated settings (f=0.1 and 7=10).
data indicates test data whose class is in the classes in training data, whereas out-of-class test data indicates test data whose class is not in the classes in training data. For instance, client A has classes 0-9 training data, then in-class test data indicates test data whose class one of 0-9 and out-of-class test data indicates test data whose class is one of 10-99.
Figure 13 describes the in-class ((a)-(c) and out-of-class ((d)-(f) test accuracy curves according to the heterogeneity. Client sampling, broadcasting, local updates, and aggregation stages are repeated during federated training. The accuracies per epoch are averaged right after broadcasting the aggregated model to the selected clients and during local updates on them. It is observed that in-class accuracy of the aggregated model is significantly low and increases dramatically during local updates, whereas out-of-class accuracy of the aggregated model is significantly high and decreases dramatically during local updates, as heterogeneity larger. In all cases, FedBABU has higher out-of-class accuracy than FedAvg after local updates. It implies that freezing the head makes less damage to out-of-class during local updates and can lead better aggregation. Furthermore, when heterogeneity is large (s=10), FedBABU has higher in-class accuracy than FedAvg after local updates.
Q FEDBABU WITH THE NON-ORTHOGONAL CLASSIFIERS
We demonstrate that orthogonal initialization on the head is required for desirable performance in the centralized setting (Appendxx B). To investigate whether this characteristic still required under FL settings, we compare FedBABU with the orthogonal head (which is proposed) to FedBABU without the orthogonal head. FBU without the orthogonal head is designed the same way as 'similar' in Apppendix B) Table 24 describes the results. As we expected, if the head does not consist of the orthogonal row vectors, FedBABU cannot achieve desirable performance.
Table 24: Initial and personalized accuracy FedBABU on CIFARI00 according the head's orthogonality under various FL settings with 100 clients (f=0.1). MobileNet is used.
100 (heterogencity І)
50
s=10 (hetcrogeneity
Orthogonul
Initial
Personalized
Initial
Personalized
Initial
Personalized
41.02-4499
49.67-t492
41.33:15.10
56.69-13.1
35.05-7763
76.02-66.9
(proposed)
36.77=4447
44.74=5.10
34.68+4.58
49.55=t3.58
25.67-731
71.00+66.6
29.38
35.94+3306
27.91
42.63
18.50#772
66.32+t7.z
11.05-3.31
17.30:t341
11.01
23.22:14.16
9,37444.73
47.09447.93
X
18.68
24.13-13:04
18.32:t:.69
29.87:4451
14.02=47:00
58.63-58.45
10
10.46-+4110
25.23-+-4n5
20.81-1:.11
32.82-+4775
11.71
88.54-18.67
26

---

Published as a conference paper at ICLR 2022
R DESCRIPTION OF DATA DISTRIBUTION ACCORDING TO THE SHARDS PER USER
To help understanding data distribution of clients according to the shard per user 8, we provide examples of them. Figure describbs examples of data distribution of clients when is 10 (left) 2 (right). Consider that there are 10 clients and CIFARIA 10 dataset is scattered to each client. When 10, total shards 100 because 10 users and 10 shards per user. Then, the CIFARIO dataset is divided into 100 shards, hence each shard consits of 500 samples with the same class, Here, each client has this situation, each client can have up 10 classes. On the contrary, when is 2, total shards are 20 because 10 users and 2 shards per user. Then, the CIFARIO 10 dataset is divided into 20 shards, hence each shard consits of 2500 samples with the same class. Here, each client has shards randomly. In this situation, each client can have 2 classes at most. Namely, as gets smaller, the number of classes that each clients has is limited (i.e., label distribution owned by each client limited) and the number of samples per class increases.
10 shards randomly.
5000
5000
6003
4:000
2003
2000
(a) s=10.
(b) s=2.
Figure 14: Examples of data distribution of users according to the shards per user.
S PERFORMANCE WITH LARGER TOTAL EPOCHS
We fixed the total number of epochs to 320, i.e., K (total communication rounds) x (local epochs) 320. This is because the performance gap between algorithms appeared and Collins et al. (202I) used total number of epochs to 500, where 1=100 and 7=5 on CIFARI00 However, because algorithms may not converge, when s is 20 and 5. Table 25 describes the results with the total epochs of 640 when is 0.1. The results demonstrate that BBU is the best under large heterogeneity (i.e., s=20, 10, and 5), which is the situation federated learning researchers attempt to solve.
add the results evaluate them with larger total epochs of 640. Furthermore,
Table 25: Personalized accuracy comparison CIFAR100 under various settings with 100 clients and MobileNet is used with the total epochs of 640 (f=0.1).
FL settings
Personalized accuracy
FedBABU (Ours)
FedAvg [2017]
FedPer [2019]
LG-FedAvg (2020)
FedRep 2021
Per-FedAvg (2020)
Ditto 2021
1
59.90-14.52
5..34
17.34
53.24
24. 24.19:33.8
48.41
51.86-24.98
100
4
4-4.67
43.70
47.00
43.87=b4.5
18.1943386
41.79-1.08
36.46:b4.18
10
36.88 1..66
36.25:14.17
39.66 1:496
35.65
14.60-1:441
32.27-1744
29.11:14.50
6:.15-1:5.20
57.10:14.69
61.67
53.89.14.90
32.94:1:.10
46.11
50.78.15.84
50
79.66-t:3.79
49.42-t:.11
52.70
48.18:+5.00
25.83-+5.06
36.51
36.09-t5.49
47.00 10
30.33-t:5.2
47.00
38.311:5.2
14
30.31
30.08
1
75.12-45.33
66.87-15.16
71.51
59.82-+5.63
10.95:66.17
41.77=18.9
47.76:28.87
20
4
67.71 4:5.26
5,550
65.87
54.76
42.71
28.71+19.00
36.05=t
10
57.55-16.95
52.44-14.73
56.47-16.03
42.11
37.09
23.39-17.1
27. 27.49
I
82.26:15.49
66.32-16.22
79.44 44
69, 69.23
66.1948.16
38.64:110.37
42.82++14.08
78.67
63.38
74.00
54.67=10.20
17.42++8.4
19.81
15.06
10
72.72-16.77
68.04416.81
68.41
42,44
48.72-18.30
15.39-19.
23.08
1
88.16-155.70
84.42-16.43
85.04_16.30
81.03
77.80+19.1
12.10:115.62
19.34
5
4
84.91
-17005
80.26:17.33
78.65 1:661
12.84-17.17
8.36
22.6 21:21
10
81.05
1:6.99
74.70:17.50
75.19
19.75
62.77.18.05
5.92.19.88
14.80-121.38
27

---

Published as a conference paper at ICLR 2022
T QUALITATIVE COMPARISON BETWEEN FEDAVG AND FEDBABU
Figure 15 descrriss t-SNE visualization (Van der Maaten & Hinton 2008) of representations learned by FedAvg and FedBABU on CIFAR 10 and CIFAR 100 for a qualitative comparison. For speed accelcration, we used t-SNE-CUDA (Chan et al. 2019). For CIFARIO0 the sparse classes are converted into coarse classes (e.g., bicycle + vehicle). However, the difference cannot captured by the naked eye. Therefore. we further provide t-SNE visualization using only sub-classes of 'vehicle' super-class, described as Figure I6 For CIFARIO, airplane, automobile, ship, and truck classes are used, and for CIFAR 100, bicycle, bus, motorcycle, pickup truck, train, lawn-mower, rocket, streetcar, tank, and tractor are used.
(a) FedAvg (CIFARIO, train).
(b) FedAvg (CIFARIO, test).
(c) FedBABU (CIFARIO, train).
(d) FedBABU (CIFARIO, test).
(e) FedAvg (CIFARI00, train).
(I) FedAvg (CIFARI00. 100, test).
(g) FedBABU (CIFARI00, train).
(h) FedBABU (CIFARI00, test).
Figure 15: t-SNE visualizations of representations learned by FedAvg and FedBABU, For CIFAR10 and CIFAR100, s set to and 10, respectively. The models are trained under the realistic federated settings (f=0.1 and 1=10).
28

---

Published as a conference paper at ICLR 2022
(a) FedAvg (CIFARIO, train).
(b) FedAvg (CIFARIO, test).
4..
(c) FedBABU (CIFARIO, train).
(d) FedBABU (CIFARIO, test).
(e) FedAvg (CIFARI00, train).
(f) FedAvg (CIFARIO0, test).
(g) FedBABU (CIFAR100, train).
(h) FedBABU (CIFARION 100, test).
Figure 16: t-SNE visualizations of representations learned by FedAvg and FedBABU using only sub-classes of 'vehicle" super-class. For CIFAR10 10 and CIFAR100, & is set to 2 and 10, respectively. The models are trained under the realistic federated settings (f=0.1 and 1=10).
29


Reviews:
[{"review_id": "hbna7XSycHx", "reviewer": "Reviewer_yAVA", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- A simple method that retains the FedAvg learning scheme and achieves good results.\n- The analyses in the paper are exhaustive and indeed support the author's claim for the most part. \n- The paper is written clearly and easy to understand.\n- The results seem to be reproducible and code was provided.\n\nWeaknesses:\n- Although this is the first paper, that I am aware of, that stresses the importance of learning the feature extractor layers only, I do not consider this idea as an entirely novel contribution of this paper. First, as the authors state, similar ideas are presented in the few-shot learning literature, which is somewhat (yet not entirely) similar in spirit to federated learning setup (hence the success of some FSL methods in FL, such as Per-FedAvg and Reptile). Furthermore, in [1] the authors suggest exactly that learning procedure only with Gaussian processes. I think that this paper should have discussed [1] and compared to it as it also showed large improvements over PFL methods.\n- Unless I missed something, I believe that the comparison to baseline methods is not entirely fair. First, it seems that the total number of updates was fixed to 320. Yet, it may be that some baselines require more updates to converge to the optimal solution. \nI believe that a better approach to compare with baseline methods would be to set a maximal number of communication rounds and use validation-based early stopping. Then this method could be evaluated against baselines in terms of performance and convergence speed. Second, from the Appendix, it seems that hyper-parameters of baseline methods were not adjusted. This can severely hurt their performance.\n- Following the last bullet, did you use a validation split? If not, how did you set the hyper-parameters for your method and baseline methods? Some hyper-parameters, such as the learning rate, may have a detrimental effect on the performance [2], and a good configuration may vary between baseline methods.\n\nGeneral comments:\n- Did you train the feature extractor with the nearest template method or in the standard way?\n- The current order of the rows in Table 2 makes it a bit hard to compare between F and B rows. I would switch the ordering to the following: p=0,0.05(F),0.05(B),0.1(F),0.1(B).\n\n\n[1] Achituve, I., Shamsian, A., Navon, A., Chechik, G., & Fetaya, E. (2021). Personalized Federated Learning with Gaussian Processes. arXiv preprint arXiv:2106.15482.  \n[2] Hsu, T. M. H., Qi, H., & Brown, M. (2019). Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint arXiv:1909.06335.", "summary_of_the_paper": "The paper highlights the importance of the shared parameters in personalized federated learning (PFL). The general idea advocated in the paper is that during training, only the feature extractor should be learned and aggregated on the server, while the classifier head should remain fixed. Personalization is achieved after training by fine-tuning the entire local model for each client. Through extensive experiments, the authors support their claim and show improved performance compared to several baseline methods.", "summary_of_the_review": "Overall it's a nice paper, the results and the analyses are good. Nevertheless, there are some issues. First, it ignores [1] which uses a similar idea. Second, I am not entirely convinced that the empirical evolution of the baseline methods is fair. I am willing to reevaluate the paper based on the author's response and adjustments to the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Z2yxKru0AZv", "reviewer": "Reviewer_TnVE", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1. The paper takes practical issues of federated learning: what creates degradation problems in training heterogeneous data in federated learning. For me, the problem itself is real and practical.  \n2. The proposed training scheme is novel and very easy to implement. The authors provide the reason why the model head and the body should be trained separately by comparing the performance (Table 2). \n3. This paper provides comprehensive experiments, including both qualitative analysis and quantitative results, to show the effectiveness of the FEDBABU training algorithm in federated learning. \n\nCons: \n1. Although the proposed method provides a study(Table 2), I still suggest the authors provide more investigation of the problem of training the head in federated learning. \nHow does the randomness of the shared head affect the performance?\nQualitative comparison between training the body only and training the whole network. Maybe plotting the distribution before the head.\nHow about sharing the fixed random two(or three?) last layers instead of one (head)\n\npg 5. ‘It is thought that a small s makes local tasks easier because the label distribution owned by each client is limited and the number of samples per class increases.’ <- Could you elaborate this portion more with evidence?\n\n", "summary_of_the_paper": "The paper provides an interesting suggestion in the field of federated learning. In particular, it proposes to enhance federated learning performance by breaking down the network into the body and head and proposes to only train the body during the federated learning phase. Specifically, the authors show that a fixed random classifier can have comparable performance to the learned classifier, and thus propose to share a fixed random classifier across all clients. The authors show the proposed method is efficient especially when the level heterogeneity of the data is more intense.", "summary_of_the_review": "Overall, I vote for accepting. I like the logical flow that 1) capturing the root of degradation lies in the head, 2) prove the feasibility of using a shared random head, 3) compare the performance between the proposed scheme and the baseline. I think the authors find/tackle the problem in a clear manner. My remaining concern is if it were really the last layer(head) only that creates a degradation problem in a federated learning setting. If so, why? Hopefully, the authors can address my concern in the rebuttal period.\n\n=====POST-REBUTTAL COMMENTS======== \nThanks for the authors' response. The author`s responses address well my concern, and I recommend this paper be accepted.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "XofkUwcnRR_", "reviewer": "Reviewer_4NJT", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper describes enough details and evaluations to understand their proposed methods and its superior performances compared with other methods. The idea itself is simple but there are enough supporting materials in the paper. Supplementary materials also help and  support their proposals. Focusing on the final personalization part and how the whole learning sequence should be is well described.\nThis paper reveals many advantages against other competitive methods. Among them its robustness with higher performance against data hetegeneity is impressive for practical federated learning. This paper also reveals that  head-part only training is enough for personalization and this leads to faster learning speed. This is also important beneficial performance point in federated learning.\nAs for weakness the random head part is important in FedBABU and supplemental materials argues this issue but more performance evaluations dependent on head part initialization would support their proposal.", "summary_of_the_paper": "Federated learning has two development targets of a single global model and multiple personal models under data hetegeneity. This paper focuses on multiple personal model development and proposes FedBABU, Federated Averaging with Body Aggregation and Body Update. In FedBABU they split the whole network into the body part with representation function and the head part with classification function. To make better representation part  is important for personalization and they train the local models and a global mode with the fixed head until personalization. In personalization they train each local model which consists of shared global body part and local specific head part using local data. This paper shows FedBABU outperformers other FedAvg-variant personalization algorithms in many viewpoints. Such as the final performance, robustness against hetegeneity, and faster learning speed.\n", "summary_of_the_review": "The problem definition is clear and there are enough materials and descriptions to support their proposals. This paper has supplementary materials and those also support their proposals.\nIn federated learning data  hetegeneity is a most important issue. As for their performance advantages in comparison with other competitive FedAvg-variant methods, FedBABU is robust against data etegeneity and outperforms others in personalization as well.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "6p2HcqRYIT0", "reviewer": "Reviewer_UZKM", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I feel that the paper is promising, but empirically underpowered to support the claims it is making. The overall concept for FedBABU aligns with broader empirical trends in the ML community related to transfer learning, but this does not eliminate the authors' burden to provide either strong empirical results, or theoretical motivation (or both) for their work. As-is, the claims made in the paper are too strong given that all experiments are only on one dataset which is synthetically partitioned into clients. I think that it could also use some revisions for clarity, both in writing and presentation of results. I provide other detailed feedback below.", "summary_of_the_paper": "This paper presents a new personalization algorithm for FL wherein only the \"body\" of a model is updated during federated training (not the terminal linear layer), and then a model is fine-tuned (or \"personalized\") on client data. The authors suggest that this resolves a tension in the existing literature/methods, where better global performance typically comes at the expense of worse local performance. Results are supported with experiments on CIFAR100 with comparisons to several baseline personalized FL methods, and FedAvg.", "summary_of_the_review": "\n## Major Comments\n\n* The paper makes some very strong claims which I don't find to be fully supported by the experiments. For example, \"we argue that training the head using shared data negatively impacts the personalization\". This is a strong, broad claim, but it is made on the basis of experiments on only a single, artifically-federated dataset (CIFAR100). In the absence of (i) stronger empirical evidence, or (ii) some theoretical motivation for this claim (either novel theory or via connections to existing theory), it is difficult to fully get on board with these claims, which form the foundation of the proposed algorithm. If there is empirical evidence from other works supporting the authors' conclusions here, please also cite it. (Another claim needed more support is that training the \"head\" during federated rounds degrades performance.)\n\n* The claim that \"randomly fixed classifiers are acceptable...\" is not clearly connected to the authors' statements about orthogonality, which are relegated to the Appendix. Even on reading Appendix B, it's not clear to me how this discussion of initialization proves the authors' claim, or whether this also might imply that personalization of the head might also be unnecessary under certain conditions. Perhaps you could draw a more explicit connection, or link this to work on randomly-featured models, since I sense a connection.\n\n* It's not clear to me why the authors opt to fine-tune the entire model, in order to accommodate FedAVG (mentioned at end of Section 5.2.2), when the results on FedBABU suggest that fine-tuning only the head is sufficient. This seems to require extra compute, and increase the risk of overfitting, while providing no benefit to the algorithm of interest, FedBABU. Please clarify. (I will note that, in T5 (Raffel et al 2020), the authors show that fine-tuning the entire model is preferable to fine-tuning only the later layers, if I remember correctly.)\n\n* I feel that a key analysis missing from the paper is related to the degree of overfitting (or not) that happens during fine-tuning. It is unsurprising that fine-tuning a small amount improves performance, on the clients' data; it is important to know, however, if that results in a much poorer performance on out-of-distribution data (i.e. another client, or a more globally-representative dataset); I was surprised not to see such an analysis in the paper, particularly because this seems of both theoretical importance (the authors imply a kind of tradeoff between global and local model accuracy) and practical importance (it is likely that real-world client data will drift). I would suggest some empirical evidence to this end, or at the very least some discussion of it.\n\n* The finding, in Sec 5.2.3, that FedAVG outperforms *all* preexisting personalization algorithms, except FedBABU, is to me very significant. If true, I think that this should be a \"main contribution\" of the paper and should be mentioned much earlier, since it suggests that this is the only current personalization algorithm that actually achieves its intended aim.\n\n## Minor Comments\n\n* The tables are very hard to read; I would recommend replacing them with bar graphs (with error bars) and optionally including the tables in the supplement.\n\n* The switching nomenclature between body/extractor and head/classifier was quite confusing to me as a reader, particularly since \"classifier\" often refers to the *entire* model and I am inclined to read it that way. Please choose one terminology and use it consistently.\n\n* Fig. 1 could be improved a great deal. It needs an illustration of \"head\" vs. \"body\", as this is a (the?) major difference from FedAvg that is not currently aptured. Along those lines, it would also be useful to note what is transmitted to the server in the rightmost column, since again, this differs between FedBABU and FedAvg. Also, note that the figure has room to be wider; perhaps you could add a fourth column characterizing the personalization step of FedBABU.\n\n* It seems there is no explicit connection made to the \"pretrain -> fine-tune\" paradigm that is increasingly prevalent in the broader machine learning community (now also referred to as the use of \"foundation models\"), although some works in this area are cited (e.g. BERT). An explicit connection would be useful here, and would help motivate the use of a similar approach in FL via FedBABU. Please clarify how, if at all, you believe this method to be different from this transfer learning paradigm.\n\n* On p.5, the claim that global training pays attention to \"unnecessary and confusing information for personalization\" seems like an incorrect characterization. It may be \"irrelevant\" to a specific client at a given point in time, but we might reasonably believe that such data actually improves the generalization capabilities of the model (including if that clients' distribution changes in the future) -- does that make it \"unnecessary\"?\n\n\n* The authors reference multiple times using a learning rate of zero to not update the head layer (which implies that a no-op is performed, for every paramter, with learning rate zero)...why are these updated being performed at all, instead of only updating \\theta_{ext}?\n\n* The conclusion references \"regularization\" as reducing models' ability to personalize; I am not sure which part of the analysis/experiments this refers to. Please clarify.\n\n## Typos etc.\n\n* All of the \"i.e.\"s in the abstract should be removed.\n\n* P.1: \"popular networks have one linear layer...and ResNet\" -- this sentence is confusing; please revise.\n\n* P.2 The phrase \"representation learning based on the same fixed criteria\" is not clear; a similar phrase is used in 5.2. Please clarify what a \"criteria on learning representations\" (5.2) is and what the \"intuition\" you are suggesting here may be.\n\nP.3 \"When the bottom layers are matched with the body\" and \"When the top layers are matched with the head\" are unclear.\n\n* P4 \"decayed 0.1 times\" --> decayed by a factor of 0.1\n\n* Why abbreviate (B), (F) in Table 1? I would suggest \"Body Updates\" vs. \"Head + Body Updates\" or similar.\n\n## Update after revisions\n\nI thank the authors for their response. Having read the author response, reviewed the updated manuscript, and considered the other reviews, I am keeping my score the same. While the authors addressed several more minor points regarding the paper, I don't believe they addressed my main issue with the paper, which is that the empirical evidence is fairly weak and limited to only a single dataset; I think that either more empirical evidence, or some theoretical support for the proposed method, is needed before being able to argue for acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper makes some novel and interesting observation pertaining the relationship between data heterogeneity and personalization. Reviewers like the paper and ideas in general but raised several concerns. The rebuttal rectified several confusions and provided more clarification which convinced the reviewers that the paper is above bar for publication.
Decision: Accept (Poster)