Personalized Federated Learning with Feature Alignment and Classifier Collaboration
Authors: Jian Xu
Xinyi Tong
Shao-Lun Huang
Venue: ICLR 2023 notable top 5% (2023)
Keywords: Federated Learning
Personalization
Collaboration

Abstract: Data heterogeneity is one of the most challenging issues in federated learning, which motivates a variety of approaches to learn personalized models for participating clients. One such approach in deep neural networks based tasks is employing a shared feature representation and learning a customized classifier head for each client. However, previous works do not utilize the global knowledge during local representation learning and also neglect the fine-grained collaboration between local classifier heads, which limits the model generalization ability. In this work, we conduct explicit local-global feature alignment by leveraging global semantic knowledge for learning a better representation. Moreover, we quantify the benefit of classifier combination for each client as a function of the combining weights and derive an optimization problem for estimating optimal weights. Finally, extensive evaluation results on benchmark datasets with various heterogeneous data scenarios demonstrate the effectiveness of our proposed method.

Published as a conference paper at ICLR 2023
PERSONALIZED FEDERATED LEARNING WITH FEA- TURE ALIGNMENT AND CLASSIFIER COLLABORATION
Jian Xu, Xinyi Tong, Shao-Lun Huang"
Tsinghua Shenzhen International Graduate School, Tsinghua University
ABSTRACT
Data heterogeneity is one of the most challenging issues in federated learning, which motivates a variety of approaches to learn personalized models for participat- ing clients. One such approach in deep neural networks based tasks is employing a shared feature representation and learning a customized classifier head for each client. However, previous works do not utilize the global knowledge during local representation learning and also neglect the fine-grained collaboration between local classifier heads, which limit the model generalization ability. In this work, we conduct explicit local-global feature alignment by leveraging global semantic knowledge for learning better representation. Moreover, we quantify the benefit of classifier combination for each client as function of the combining weights and derive an optimization problem for estimating optimal weights. Finally, extensive evaluation results on benchmark datasets with various heterogeneous data scenarios demonstrate the effectiveness of our proposed method.
INTRODUCTION
Modern learning tasks are usually enabled by deep neural networks (DNNs), which require huge quantities of training data to achieve satisfied model performance (Lecun et al., 2015 Krizhevsky cct al. 2012 Hinton et al., 2012). However, collecting data is too costly due to the increasingly large volume of data even prohibited due privacy protection. Hence, developing communication- learning algorithms is of significant importance for fully taking advantage of the data in clients, e.g., data silos and mobile devices (Yang et al. 2019 Li et al. 2020a). To this end, federated learning (FL) emerged as an innovative technique for collaborative model training over decentralized clients without gathering the raw data (McMahan et al. 2017). A typical FL setup employs a central server to maintain a global model and allows partial client participation with infrequent model aggregation, e.g., the popular FedAvg, which has shown good performance when local data across clients are independent and identically distributed (IID). However, in the context of FL, data distributions across clients usually not identical (non-IID heterogeneity) since different devices generate or collect data separately and may have specific preferences, including feature distribution drift, label distribution skew and concept shift, which make it hard to learn single global model that applics to all clients (Zhao et al. 2018 Zhu et al. 2021a Li et al. 2022).
efficient and privacy-preservine
To remedy this, personalized federated learning (PFL) has been developed, where the goal is to learn a customized model for each client that has better performance on local data while still benefiting from collaborative training (Kulkarni et al. 2020 Tan et al. 202 1a Kairouz et al. (021)) Such settings can be motivated by cross-silo FL., where autonomous clients (e.g., hospitals and corporations) may wish to satisfy client-specific target tasks. A practical FL framework should be aware of the data heterogeneity and flexibly accommodate local objectives during joint training. On the other hand, the DNNs based models are usually comprised of feature extractor for extracting now-dimensional feature embeddings from data and classified for making classification decision. The success of deep learning in centralized systems and multi-task learning demonstrates that the feature extractor plays the role of common structure while the classifier tends to be highly fask-correlated (Bengio et al.) 2013 Collins et al.) 2021 Caruana 1993). Moreover, clients in practical FL problems often deal with similar learning tasks and clustered structure among clients is assumed in many prior works (Ghosh) et al. 2020 Sattler et al. 2021). Hence, learning a better global feature representation and exploiting correlations between local tasks are of significant importance for improving personalized models.
"Corresponding author: shaolun. www.huaangggoo @ www.tssnnguu.c... enn.uua.edu.cn
We denote classifier head the final output layer for decision-making and abbreviate classifier.

---

Published as a conference paper at ICLR 2023
In this work, we mainly consider the label distribution shift scenario, where the number of classes is the same across clients while the number of data samples in each class has obvious drift, i.e., heterogeneous label distributions for the local tasks. We study federated learning from multi-task learning perspective by leveraging both shared representation and inter-client classifier collaboration. Specifically, we make use of the global feature centroid of cach class to regularize the local training, which can be regarded as explicit feature alignment and is able to reduce the representation diversity across locally trained feature extractors, thus facilitating the global aggregation. We also conduct flexible classifier collaboration through client-specific linear combination, which encourages similar clients to collaborate more and avoids negative transfer from unrelated clients. To estimate the proper combining weights, we utilize local feature statistics and data distribution information to achieve the best haasvvarancce trade-off by solving quadratic programming problem that minimizes the expected testing loss for each client. Moreover, with a slight modification, our framework could still work well under the concept shift scenario, where the same label might have varied meanings across clients.
Our contributions. We focus on deep learning- basedd classification tasks and propose a novel FI. framework equipped by personalized aggregation of classifiers (FedPAC) and feature alignment to improve the overall performance of client-specific tasks. The proposed framework is evaluated on benchmark datasets with various levels of data heterogeneity to verify the effectiveness in achieving higher model performance. Our evaluation results demonstrate the proposed method can improve the average model accuracy by 2~5%% To summarize, this paper makes the following key contributions:
We quantify the testing loss for each client under the classifier combination by characterizing the discrepancy between the learned model and the target data distribution, which illuminates new bias- variance trade-off.
novel personalized federated learning framework with feature representation alignment and opti- mal classifier combination is proposed for achieving fast convergence and high model performance.
Through extensive evaluation on real datasets with different levels of data heterogeneity, we demonstrate the high adaptability and robustness of FedPAC.
Benefits of FedPAC. The benefits of our method over current personalized FL. approaches include:
(i) More local updates for representation learning. By leveraging feature aligmment to control the drift of local representation learning, each client can make many local updates with less local local-global parameter diversity at each communication round, which beneficial in learning better representation in communication-effiiee
manner.
(ii) Gains by classifier heads collaboration. We employ a theoretically guaranteed optimal weighted averaging for combing heads from similar clients, which is capable of improving generalization ability for data-scarce clients while preventing negative knowledge transfer from unrelated clients.
2 RELATED WORK
2.1 FEDERATED LEARNING WITH NON-IID DATA
Many efforts have been devoted improving the global model learning of FL with non-IID data. A variety of works focus on optimizing local learning algorithms by leveraging well-designed objective
regularization (Li et al. 2020b Acar et al.) 2021a; Li et al. 2021b) and local bias correction (Karim- ireddy et al. 2020b a Murata & Suzuki 2021 Dandi et al.
2021). For example, FedProx (Li et al.
2020b) adds proximal term the local training objective keep updated parameter close the
original downloaded model, SCAFFOLD (Karimireddy et al. 2020h) introduces control variates to correct the drift in local updates, and MOON (Li al. 202Tb) adopts the contrastive loss to improve the representation learning. Class-balanced data re- re-sampling and loss re-weighting methods can improve the training performance when clients have imbalanced local data (Hsu et al.) [2000] Wang eet al. 2021 CCen & Chao 2021). Besides, data sharing mechanisms and data augmentation methods also investigated to mitigate the non-llI data challenges (Zhao et al.) 2018 Yoon et al.) 2021).
From the model aggregation perspective, selecting clients with more contribution to global model performance can also speed up the convergence and mitigate the influence of noD-III data (Wang) et al. 2020 Tang et al. 2021: Wu & Wang 2021] Fraboni et al. (021). With the availability of public data, is possible to employ knowledge distillation techniques obtain a global model despite of
the data heterogeneity (Lin et al., 2020 /huu et al., (021b). The prototype-based methods are also
utilized in some FL. works, such (Michieli & Ozay 2021 proposes prototype- based weight
2

---

Published as a conference paper at ICLR 2023
attention mechanism during global aggregation and (Mu et al. 2021 Tan et al. 2021b Ye et al.) 2022, Zhou et al. 2022) utilize the prototypes to enhance the local model training. Different from above methods, this paper aims at learning a customized model for each client.
2.2 MODEL PERSONALIZATION IN FL
In the literature, popular personalized FL methods include additive model mixture that performs linear
combination of local and global modes, such as L2CD (Hanzely & Richtárik 2020) and APFL (Deng
et al. 2020); multi-tat task learning with model dissimilarity penalization, including FedMTL (Smith
ctt al. 2017). y pFedMe (Dinh et al. 2020) and Ditto (Li et al., 2021c); meta-lcarning based local
adaption (Jiang et al. 2019 Fallah et al. 2220 Acar et al.) 2021bb; parameter decoupling of feature
extractor and classifier, such as FedPer, LG-FedAvg and FedRep (Arivazhagan et al., [2019] Liang
cet al. 2020; Collins et al. 2021). A special type of personalized FL approach is clustered FL to
group together similar clients and learn multiple intra- intra-group global models (Duan et al. 2021 Ghosh
ct al. 2020 Mansour et al., 2020; Sattler et al. 2021). Client-specific model aggregations are also investigated for fine-grained federation, such as FedFomo and FedAMP (Zhang et al. 2021b
Huang et al. 2021] Beaussart et al. 202T), which have similar spirit to our approach. Nevertheless,
existing client-specific FL methods are usually developed by evaluating model similarity or validation accuracy in a heuristic way, and these techniques need to strike a good balance between communi- cation/computation overhead and the effectiveness of personalization. FedGP based on Gaussian processes (Achituve et al. 2021) and selective knowledge transfer based solutions ZZaagg et al. 202Ia Cho et al.) 2021) are also developed. however those methods inevitably rely on public shared data set inducing points set. Besides. pFedHN enabled by server-side hypemetwork (Shamsian cct al. 202I) and FedEM that learns a mixture of multiple global models (Marfoq et al. (022)) are also investigated for generating customized model for each client. However, pFedHN requires each client to communicate multiple times for learning a representative embedding and FedEM significantly increases both communication and computation/storage overhead. Recently, Fed-RoD (Chen & Chao 2022) proposes to use the balanced softmax for learning generie model and vanilla soltmax for per- sonalized heads. FedBABU (Oh et al. 2022) proposes to keep the global classifier unchanged during the feature representation learning and perform local adoption by fine-tuning. kNN-Per (Marfoq et al. 2022) applies the ensemble of global model and local kNN classifiers for better personalized performance. Our work shares the most similar learning procedure with FedRep (Collins et al., 202T], but differs in the sense that we employ global knowledge for guiding local representation learning classifier heads combination for each client.
and also perform pnoorticalllygguaantteo
3 OVERVIEW OF PROPOSED FRAMEWORK
In this section, we provide overview of our approach FedPAC for training personalized models by exploiting a better feature extractor and user-specific classifier collaboration. Theoretical justification and algorithm design will be presented in the next two sections.
3.1 PROBLEM SETUP
We consider a setup with m clients and a central server, where all clients communicate to the server to collaboratively train personalized models without sharing raw private data. In personalized FL,, each client i is equipped with its own data distribution Pxy on X' x y. where X is the input space and y is the label space with K categories in total. We assume that p(i) and p() are different for
any pair of client and j. which is usually the case in FL.. Let é X x ) + R+ denote the loss function given local model w, and data point sampled from Xx, e.g., enoss-y loss, then the underlying optimization goal of PFL can be formalized as follows:
min W
F(W)
= 7
E
(2.))~ [6( x,
(I)
where W = (W1,, W2 .... ) denotes the collection of all local models. However, the true under-
lying distribution is inaccessible and the goal is usually achieved by empirical risk minimization (ERM). Assume each client has access to n, IIID data points sampled from p(o) denoted by
D. {(f{) vv ( whose corresponding empirical distribution is p(0 F and we assume the
XY
"The empirical distribution is defined as p(0 (I, v) := Ela 1 1( (== x
(9)
3

---

Published as a conference paper at ICLR 2023
empirical marginal distribution p is identical to the true pl. Then the training objective is
w arg min Σ [C((w:) +
(2)
where (i(w;) - n. - (w;; Mi is the local average loss over personal training data, e.g.. empirical risk: is some kind of global information introduced to relate clients, and the (-)-) is predefined regularization term for preventing w, from over-fitting.
3.2 SHARING FEATURE REPRESENTATION
Without loss of generality, we decouple the deep neural network into the representation layers and the final decision layer, where the former is also called feature extractor, and the latter refers to the classifier head in classification tasks. The feature embedding function f X -> TRd is a learnable network parameterized by yy and d is the dimension of feature embedding. Given data point I, a prediction for extracted feature vector 2 = f(z) can be generated by linear function g()) parameterized by Gg. In the rest of paper, we omit the subscripts of 0, and Φ, for simplicity, i.e., w = {0, Φ). Since each client only has insufficient data, the locally learned feature representation is prone to over-fitting and thus cannot generalize well. A reasonable and feasible idea is to take advantage of data available at other clients by sharing the same feature representation layers. However, multiple local updates on private data will result in local over-fitting and high parameter diversity across clients, which make the aggregated model deviate from the best representation. To tackle this, we propose new regularization term for local feature representation learning.
Feature A Alignment by Regularization. The clients need to update the local model by considering both supervised learning loss and the generalization error. To this end, we leverage the global feature centroids and introduce new regularization term to the local training objective to enable the local representation learning benefit from global data. The local regularization term is given by
R,(0;;) Σ [a.) - cy. 112
(3)
where fe, (zz) is the local feature embedding of given data point zj, and Cy, is the corresponding global feature centroid of class V/, À is haper- to balance supervised loss and regularization loss. Such a regularization term is of significant benefit to each client by leveraging global semantic feature information. Intuitively, it enables each client to learn task-invariant representations through explicit feature distribution alignment. As result, the diversity of local feature extractors (0, == could also regularized while minimizing local classification error. We notice that the global feature centroid is similar to the concept of prototype that is widely used in few-shot learning, contrastive learning and domain adaption (Snell et al.) 2017]] [i et al., [001a]] Pan et al., 2009). Our theoretical insight in Section 4.2 will further reveal that this regularization term could be inspired by explicitly reducing generalization error and thus helps to improve the test accuracy.
3.3 CLASSIFIER COLLABORATION
In addition to improving feature extractor by sharing representation layers, we argue that merging classifiers from other clients with similar data distribution can also offer performance gain. Unlike previous work in (Arivazhagan et al., 2019 Collins et al.) (021) that only maintains the locally trained classifier, we also conduct client-specift weighted average of classifiers to obtain a better local classifier. Intuitively, the locally learned classifier may have a high variance when the local data is insufficient, therefore those clients that share similar data distribution can actually collaboratively train the personalized classifiers by inter-client knowledge transfer. The challenge is how to evaluate the similarity and transferability across clients. To this end, we conduct a linear combination of those received classifiers for each client to reduce the local testing loss:
(1+1)
((+1)
s.1.
04g
with each coefficient Ouj 2 0 determined by minimizing local expected testing loss, which can be formulated as the following optimization problem:
4

---

Published as a conference paper at ICLR 2023
a arg min E. a. Px [e Σ Ozj ,,
For better collaboration, need to update the coefficients a, adaptively during the training process.
Remark 1. Consider a simple case where data are homogeneous across clients. In such a setting, learning a single global model is optimal for generalization. However, some existing methods, such as FedPer and FedRep, do not take into account the combination of classifier heads and thus lead to limited performance in less heterogeneous cases. In contrast, our method conducts adaptive classifier combination and effective in both homogeneous and heterogeneous scenarios.
4 THEORETICAL ANALYSIS AND INSIGHTS
In this section, we introduce the theoretical analyses of our proposed approach for improving personalized federated learning from both the feature extractor and classifier perspectives. Specifically, we consider the framework introduced in Section 3.2] where the local model of each client contains the shared feature extractor and personalized linear classifier 9i parameterized by w, (0, &x). respectively. For the convenience of theoretical analysis and evaluation, we choose the following linear discriminative mode
Q K (1 [1+5(1 (xx) g(v))
which has the similar formation the linear model used in (Xu & Huang 2020 Tong et al. 2021
Note that usually model parameters are updated in an end-to-end manner according to the general training objective (2). However, when the classifier collaboration is discussed, we will concentrate on the scenario where the feature extractor is fixed. Then, we simply use Qy (g) to denote the model and learn the individual classifiers {gi {g of different clients and linearly combine them for each client i with the weights ax, Further, we introduce the x3 disttane in (7). which is different from the conventional chi-square divergence and is defined as follows. For any distributions and Q supported on X x V. the x² -dittance between P' and Q is given by
(P(x, y) Q(x,y))²
(P) = Σ NEX,EEY
P (x)
where p() is the marginal distribution over X. Here we select this distance measure as the loss function mainly in consideration of the interpretability and deriving the analytical solution.
For this discriminative model, ideally, the local model could be derived by minimizing the x²- distance between (())
x X and p(i) Q Yyxx. which refer to the true local data distribution and the product of marginal distribution and the learned model, respectively. However, each client will only access limited training samples and the locally learned classifier might not work well enough, which directly motivates the inter-client collaboration. Since the and g are highly correlated during the model learning, jointly analyzing them is difficult. Instead, we fix either f or by turns and apply the analysis-inspine method to optimize the learning of the unfixed one in an iterative manner.
4.1 OPTIMAL CLASSIFIER COMBINATION
Under the fixed shared feature extractor f. we can empirically learn a classifier 9. based on the f and training samples at client in the following way:
g = arg min x3 ((1) (() XY: 1 (g)), Vi E [m]]
(8)
g
Then, we can use linear combination by 9 Eド= وواة substitute for the original 9; as illustrated in Section 3.3 Similar to the training loss in (8). analytically we use the x²(, :) over true data distribution as the testing loss to evaluate the performance of resulted personalized classifier. Let C; : (oa, ... ".,,""",,,, ,am) and the testing loss for client is given by
(a,)s) == E [x3 pky p 1 (G)))
'This linear model has the same prediction criteria the softmax classifier and could be interpreted as the approximation of softmax model with large temperature the logits.
5

---

Published as a conference paper at ICLR 2023
where the expectation is taken over the sampling process, i.e., empirical distributions (P )m that follow the multinomial distributions (Csiszár 1998).
Before performing theoretical analyses, we assume that underlying marginals over X' are identical across clients and the extracted feature embedding is zero-centered as follows.
Assumption 1. For all E [m]] we have p (x) Px(x).
Assumption 2. For the given feature extractor f. we have Epx [f(X)] =0
Assumption 1-2 mainly technical to derive our analytical solution and are not strictly required for our method to work in practice. In the general cases, determining the combining weights of classifiers trained by other clients requires to learn the generative models. However, under the identical marginal assumption we can focus on the discriminative models. When the whole input space accessible, we could apply rero-centering satisfy the zero-mean feature assumption. Now, the testing loss for each client i under the linear combination of classifiers by the following theorem.
characterize
Theorem 1. The testing loss for client defined in 9 is quadratic function w.r.f. ar given by
R:(ox)=x² ( p 1
(9.)
m
p Oisy
Y (g))
+
Σ
m
U V2,
2
+x²
p Q (gi)
n
(10)
where for all j E [m], 9j == Elğs] and Vj is a variance term that related to the local feature statistics. Details and proofs are provided in Appendix A.2
We argue that the established testing loss is related the bias-variance trade-off in the personalized classifiers, where the first term in (TO) refers to the bias term of the model utilizing other clients and the second term refers to the variances from sampling, which are shown in Σド= V, V//nj- We utilize the classifiers in other clients to reduce the variance, and meanwhile it increases the bias the true distribution p()
XY. To achieve the optimal viai-variance trade-off, we define the best combination
coefficients as
m
o := arg min (o:), S.L. Σ oij =1 j=1
and aa Vj
(11)
Note that the terms appearing in the testing loss are related to the true distributions, which can be estimated by the corresponding empirical feature statistics, with details provided in Appendix A.3 When the optimal at is estimated, we realize our proposed personalized classifier
4.2 REDUCING TESTING Loss BY FEATURE ALIGNMENT
Given a fixed classifier head gi and dataset D, at client i, the testing loss can be rewritten as follows:
((0) Q R4 x2 xY, Q + >> 2 p (2, (z,w) P (z. w) f(x)* (v) C. (12)
where C, is a constant and it is easy to see that the first term is the empirical loss and the remainder can be viewed generalization gap. Due to the discrepancy between true underlying distribution Pxy and empirical distribution p()) minimizing training loss could result large generalization
gap when training data is insufficient. On the other hand, the second term in [12 can be rewritten by
AR, = 2E 2Ep
p(0
Bpm Xiy
[f()]
E pco Xix
[f())]
(Y)/K
(13)
Benefiting from FL, the true conditional distribution p(o) could be better estimated by taking advantage of data available at other clients, and therefore the above term can be further estimated by local empirical data D, and global feature centroids follows,
2 Σ Th
AR,
(c f(x))1 g: (
(14)
where Cy is the feature centroid of class estimated by collecting local feature statistics. Intuitively, this error term (14) is directly induced by the inconsistency of local-global feature representations,

---

Published as a conference paper at ICLR 2023
which means the learned feature distribution is not compact. However, there exists trade-off between training loss and generalization gap, and simultaneously minimizing them is infeasible. Therefore. similar to (Tan et al. 2021b), design a regularization term as Eq. (3) during the feature learning phase to explicitly align the local-global feature representations.
ALGORITHM DESIGN
In this section, we design alternating optimization approach to iteratively learn the local classifier and global feature extractor. Locally updated model parameters along with feature statistics will be transmitted to the central server for aggregation (more details provided in Appendix A.5
5.1 LOCAL TRAINING PROCEDURE
For local model training at round we first replace the local representation layers 0 by the received global aggregate ((*) and update private classifier analogously. Then, we perform stochastic gradient decent steps to train the two parts of model parameters as follows:
Step 1: Fix 0,, Update Φ; Train Φ; on private data by gradient descent for one epoch:
d( 7g ( :.)
(15)
where &; denotes the mini-batch of data, "g is the learning rate for updating classifier.
Step 2: Fix New Φ,, Update 0, After getting new local classifier, we turn to train the local feature extractor based on both private data and global centroids for multiple epochs:
( -
7s Ve ((
(++ D :.)) + R :(0)
(16)
where My is the learning rate for updating representation layers, (( E R is the collection of global feature centriod vector for each class, and K | is the total number of classes.
Feature Statistics Extraction: Before local feature extractor updating, each client should extract the local feature statistics f1 and Vi(1) through single pass on the local dataset, which will be utilized to estimate the optimal classifier combination weights for each client. Moreover, after updating the local feature extractor, we compute the local feature centroid for each class as follows,
El (v) (5) = =k)fo =f) (x) (0)
(1+1) i
E 1( =k) .4kk E [K].
(17)
5.2 GLOBAL AGGREGATION
Global Feature Representation. Like the common algorithms, the server performs weighted averaging of local representation layers with each coefficient determined by the local data size.
( ) 3,00 B,
(18)
Classifier Combination. The server uses received feature statistics to updates the combination weights vector a; by solving (1]) and conducts classifier combination for each client i.
Update Global Feature Centroids, After receiving the local feature centroids, the following centroid aggregating operation is conducted to generate an estimated global centroid e₁ for each class k.
k
(++1) k
Vk E [K].
(19)
-1=
EXPERIMENTS
6.1 EXPERIMENTAL SETUP
Datasets and Models. We consider image classification tasks and evaluate method four popular datasets: EMNIST with 62 categories of handwritten characters, Fashion-MNIST with 10 categories of clothes, CIFAR-10 and CINIC-10 with 10 categories of color images. We construct two

---

Published as a conference paper at ICLR 2023
different CNN models for EMNIST/Fashion-Mi and CIFAR-10/CINIC-10, respectielly Details of datasets and model architectures are provided in Appendix B
Data Partitioning. Similar to (Karimireddy et al. 2020b; Zhang et al. 202 Huang et al. 202 we make all clients have the same data size, in which 8% of data (20% by default) are uniformly sampled from all classes, and the remaining (100 5)% from set of dominant classes for each client. We explicitly divide clients into multiple groups while clients in each group share the same dominant classes, and we also intentionally keep the size of local training data small to pose the need for FL. The testing data on each client has the same distribution as the training data.
Compared Methods. We compare the following baselines: Local-only, where each client trains its model locally; FedAvg that learns a single global model and its locally fine-tuned version (FedAvg- FT); multi-task learning methods, including APFL., pFedMe and Ditto: parameter decoupling methods, including LGG-eeAAvg FedPer, FedRep and FedBA BU; Fed-RoD and kNN-Per that learn an extra local classifier based on the global feature extractor and use model ensemble for prediction; FedFomo that conducts inter-client linear combination and pFedHN enabled by a server-side hypernetwork.
Training Settings. We employ the mini-batch SGD as a local optimizer for all approaches, and the number of local training epochs is set to E = 5 unless explicitly specified. The number of global communication rounds is set to 200 for all datasets, where all FL approaches have little or no accuracy gain with more communications. We report the average test accuracy across clients.
Table 1: The comparison of final test accuracy (%) on different datasets. We apply full participation for FL system with 20 clients, and apply client sampling with rate 0.3 for FL. system with 100 clients.
EMNIST
Fashion MNIST
CIFAR-I 10
CINIC 10
Method
20 clients
100 clients
20 clients
100 clients
20 clients
100 clients
20 clients
100 clients
Local-only
73.85
74.11
85.68
86.37
65.43
64.68
63.19
63.33
FedAvg
71,85
75.97
85.28
86.89
70.05
73,82
58.38
62,52
Fed
84.56
79.34
69.38
74.44
FedPer
75.91
77.06
87.43
87.88
68.37
72.26
63.47
66.26
LG FedAvg
74.76
75.81
85.66
86,28
65.19
65.38
63.75
63,93
FedRep
75.31
75.60
88.25
88.54
71.36
72.58
66.67
66.96
FedBABU
83.52
85.59
89.68
91,38
76.46
78.39
70.58
74.67
Fed-RoD
81.94
84.57
90.11
91.61
77.16
81.97
71.82
75.20
ANN-Per
80.21
69.98
74.23
APEL
83.22
84.95
89.45
90.88
74.39
77.32
68.78
73.45
pFedMe
83.28
84,89
89.76
89.85
68.81
69.85
63.37
65,31
Ditto
84.21
86.61
90.43
91.08
77.02
80.09
70.
74.25
FedFomo
83.95
85.63
88.95
90.19
73.33
76.21
66.68
70.22
PRodHN
80.35
78.16
88.36
87.93
76.95
79.39
71.59
70.85
FedPAC
86.46
89.88
91.83
92.72
81.13
83.36
74.96
77.36
6.2 NUMERICAL RESULTS
Performance Comparison. We conduct experiments on two setups, where the number of clients is 20 and 100, respectively. For the latter, we apply random client selection with sampling rate C'=0.3 along with full participation in the last round. The training data size in each client is set to 600 for all datasets except EMNIST, where the size is 1000. The main results are presented in Table it obvious that our proposed method performs well on both small-scale and large-scale FL systems. For all datasets, FedPAC dominates the other methods on average test accuracy, which demonstrates the effectiveness and benefit of global feature alignment and inter-client classifier collaboration. The classifier combination weights are shown in Appendix [C.I] We also notice that FedAvg with local fine-tuning is already a strong baseline for PFL, resulting in competitive performance as the start-of tfet-off-hh-arr art methods.
We have two key design components in FedPAC, i.e., feature alignment (FA) and classifier combination (CC). Here we conduct ablation studies to verify the individual efficacy of those two components. We make use of FA and CC separately on four datasets and report the average accuracy on 20 clients. As demonstrated in Table both of them can help improve the average test accuracy and the combination of them is able achieving the most satisfactory model performance, which means a better global feature extractor and more suitable personalized classifiers could be built under our proposed method.
Ablation Studies.
8

---

Published as a conference paper at ICLR 2023
Table 2: Ablation study. FA denotes Feature Alignment and CC denotes Classifier Collaboration; None means neither FA nor CC used, while Both means FA and CC are applied simultancously.
Design Choices in FedPAC
Dataset
None
w/ FA
w/ CC
Both
EMNIST (%)
75.63
86.45
82.78
86,46
Aishion-MNIST (%)
87.93
89.74
89.92
91.83
CIFAR: (%)
71.68
79.43
78.19
81.13
CINIC-10 (%)
66.20
72.89
70.84
74.96
100
CIFAR-10
CIFAR-1D
FedAug FT
FodRop
85
Ditto
Toxt Accurayy (55)
FFddHN FodPAC
80
80
Test Accuracy
75
70
65
FedAvg
DFedMe
60
FedRop
300
600
900
1200
15500
2000
Data Size
(a) Effect of data heterogeneity
(b) Effect of data size
Figure 1: Performance comparison CIFAR- 10 dataset with varying data heterogeneity and local data size.
Effects of Data Heterogeneity and Data Size. We vary the values of to simulate different levels of data heterogeneity, while s=0% indicates a highly heterogeneous case (pathological non-IID) and s=80% means data across clients are more homogeneous. We evaluate the CIFAR-10 10 dataset and the results of different methods are reported in Figure (a). It can be found that our method consistently outperforms other baselines, which demonstrates its adaptability and robustness in variety of heterogeneous data scenarios. We also test our FedPAC and FedAvg with varying local data sizes, recording the resulted model accuracy as in Figure (b). The results indicate that clients with different data sizes can consistently benefit from participating in FL and our method achieves higher performance gain.
Robustness to Concept Shift. Though the concept shift is not our focus, we show that our method also applicable and robust in such scenarios. simulate the concept shift scenario, each group of clients will perform specific permutation of labels. The results in Table 3 show that the label permutation hurts the performance of FedAvg-like algorithms that learn global classifier as different learning tasks are conflicting. On the other hand, the general improvement over Local-one colll-only demonstrates that leveraging the global feature extractor can still benefit from data among clients with concept shift.
Table Test accuracy (%) under concept shift. We apply full client participation for system with clients.
Dataset
Local anly
FedAvg FT
FedRep
Ditto
pFeedMee
FedBABU
Fed RoD
FedPAC
Fashion MNIST
85.68
87.38
87.12
87.78
86.84
87.28
87.60
91.09
CIFAR-10
70.55
68.12
62.71
78.45
7 CONCLUSION AND FUTURE WORK
In this paper, we introduce the global feature alignment for enhancing representation learning and a novel classifier combination algorithm for building personalized classifiers in FL, providing both theoretical and empirical justification for their utilities in heterogeneous settings. Future work includes analyzing the optimal model personalization in more complex settings, e.g., decentralized systems or clients with dynamic data distributions and investigating the optimal aggregation of local feature extractors.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENTS
The research of Shao-Lun Huang is supported in part by the Shenzhen Science and Technology Program under Grant KQTD20017080015082111 National Key R&D Program of China under Grant 1FA07
REFERENCES
Durmus Alp Emre Acar, Yue Zhao, Ramon Matas Navarro, Matthew Mattina, Paul N. Whatmough, and Venkatesh Saligrama. Federated learning based on dynamic regularization. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021a.
Durmus Alp Emre Acar, Yue Zhao, Ruizhao Zhu, Ramon Matas Navarro, Matthew Mattina, Paul N. Whatmough, and Venkatesh Saligrama. Debiasing model updates for improving personalized federated training. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 21-31. PMLR, 2021b.
Idan Achituve, Aviv Shamsian, Aviv Navon, Gal Chechik, and Ethan Fetaya. Personalized federated learning with gaussian processes, arXiv preprint arXiv:2106. 15482, 2021.
Manoj Ghuhan Arivazhagan, Vinay Aggarwal, Aaditya Kumar Singh, and Sunav Choudhary. Feder- ated learning with personalization layers. arXiv preprint arXiv::912.000
2019.
Martin Beaussart, Felix Grimberg, Mary-Anne Hartley, and Martin Jaggi. Waflle: Weighted averaging for personalized federated learning. arXiv preprint arXiv:2110. 100.07770,,978, 2021.
Yoshua Bengio, Aaron C. Courville, and Pascal Vincent. Representation learning: A review and new 2013.
perspectives. IEEE Trans. Paltern Anal. Mach. Intell.,
Rich Caruana. Multitask learning: A knowledge-basse
source of inductive bias. In Paul E. Utgoff (ed.), Machine Learning, Proceedings of the Tenth International Conference, University of Massachusetts, Amherst, MA, USA, June 27-29, 1993, pp. 41-48. Morgan Kaufmann, 1993.
Hong-You You Chen and Wei-Lun Chao. On bridging generic and personalized federated learning. arXiv preprint arXiv:2107.0077 2021.
Hong- You Chen and Wei-Lun Chao. On bridging generic and personalized federated learning for image classification. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022, 2022.
Yae Jee Cho, Jianyu Wang, Tarun Chiruvolu, and Gauri Joshi. Personalized federated learning for heterogeneous clients with clustered knowledge transfer. arXiv preprint arXiv:2 arXiv:2109.08) 109.0811111111 19, 2021.
Gregory Cohen, Saeed Afshar, Jonathan Tapson, and André van Schaik. EMNIST: extending MNIST to handwritten letters. In 20/7 International Joint Conference on Neural Networks, IJCNN 2017, Anchorage, AK, USA, May 14-19, 2017, pp. 2921-2926. IEEE, 2017.
Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai. Exploiting shared represen- tations for personalized federated learning. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 2089 2099. PMLR, 2021.
Imre Csiszár. The method of types [information theory]. IEEE Transactions on Information Theory, 2466)22505-2533, 1998.
Yatin Dandi, Luis Barba, and Martin Jaggi. Implicit gradient alignment in distributed and federated learning. CoRR, fas/21066.18977 2021.
Yuyang Deng, Mohammad Mahdi Kamani, and Mehrdad Mahdavi. Adaptive personalized federated learning. arXiv preprint arXiv:2003. 13461, 2020.
Canh T. Dinh, Nguyen H. Tran, and Tuan Dung Nguyen. Personalized federated learning with moreau envelopes. In Conference on Neural Information Processing Systems, NeurIPS, 2020.
10

---

Published as a conference paper at ICLR 2023
Moming Duan, Duo Liu, Xinyuan Ji, Yu Wu, Liang Liang, Xianzhang Chen, and Yujuan Tan. Flexible clustered federated learning for client-level data distribution shift, arXiv preprint arXiv:2108.097493 2021.
Alireza Fallah, Aryan Mokhtari, and Asuman E. Ozdaglar. Personalized federated learning with theoretical guarantees: A model-agnostic meta-leamiing approach. In Advances in Neural Informa- fion Processing Systems 33: Annual Conference Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
Yann Fraboni, Richard Vidal, Laetitia Kameni, and Marco Lorenzi. Clustered sampling: Low- variance and improved representativity for clients selection in federated learning. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, 2021.
Avishek Ghosh, Jichan Chung, Dong Yin, and Kannan Ramchandran. An efficient framework for clustered federated learning. arXiv preprint arXiv:2006.04088 2020.
Filip Hanzely and Peter Richtárik. Federated learning of a mixture of global and local models. arXiv preprint 200.::000002.555/6 2020.
Chaoyang He, Songze Li, Jinhyun So. Mi Zhang, Hongyi Wang. Xiaoyang Wang, Praneeth Vepakomma, Abhishek Singh, Hang Qiu, Li Shen, Peilin Zhao, Yan Kang, Yang Liu, Ramesh Raskar, Qiang Yang, Murali Annavaram, and Salman Avestimehr: Fedml: A research library and benchmark for federated machine learning. CoRR, abs/2007. 135188 2020.
G. Hinton, L. Deng, D. Yu, G. E. Dahl, A. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. N. Sainath, and B. Kingsbury. Deep neural networks for acoustic modeling speech recognition. IEEE Signal Processing Magazine, 29(6):82-97 2012.
Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Federated visual classification with real-world data distribution. In Computer Vision ECCV 2020 16th European Conference, Glasgow, UK, August 23-28, 2020, 2020.
Yutao Huang. Lingyang Chu, Zirui Zhou, Lanjun Wang, Jiangchuan Liu, Jian Pei, and Yong Zhang. Personalized cross-silo federated learning on non-iid data. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Virtual Event, February 2-9, 2021, pp. 7865-7873. AAAI Press, 2021.
Yihan Jiang, Jakub Konečný, Keith Rush, and Sreeram Kannan. Improving federated learning personalization via model agnostic meta learning. arXiv preprint arXiv: 1900://999.88, 2019.
Peter Kairouz, H. Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, and et al. Advances and open problems federated learning. Found. Trends Mach. Learn., 14(1-2):1-210, 2021.
Sai Praneeth Karimireddy, Martin Jaggi. Satyen Kale, Mehryar Mohri, Sashank J Reddi, Sebastian U Stich, and Ananda Theertha Suresh. Mime: Mimicking centralized stochastic algorithms in federated learning. arXiv preprint arXiv:2008.03600 2020a.
Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank J. Reddi, Sebastian U. Stich, and Ananda Theertha Suresh. SCAFFOLD: stochastic controlled averaging for federated learning. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 5132-5143. PMLR, 2020b.
Alex Krizhevsky and G. Hinton. Learning multiple layers of Toronto, 2009.
features from tiny images. University of
Alex Krizhevsky, I. Sutskever, and G. Hinton. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems (NIPS), 2012.
Viraj Kulkarni, Milind Kulkarni, and Aniruddha Pant. Survey of personalization techniques for federated learning. In 2020 Fourth World Conference on Smart Trends in Systems, Security and Sustainability (WorldS4), pp. 794-797. IEEF, 2020.

---

Published as a conference paper at ICLR 2023
Y Lecun, Y Bengio, and G Hinton. Deep learning. Nature, 521((553))436 444, 2015.
Junnan Li, Pan Zhou, Caiming Xiong, and Steven C. H. Hoi. Prototypical contrastive learning of unsupervised representations. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021a.
Qinbin Li, Bingsheng He, and Dawn Song. Model-contrastive federated learning. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pp. 10713- 10722. Computer Vision Foundation / IEEE, 2021b.
Qinbin Li, Yiqun Diao, Quan Chen, and Bingsheng He. Federated learning on non-iid data silos: An experimental study. In 38th IEEE International Conference on Data Engineering, ICDE 2022, Kuala Lumpur, Malaysia, May 9-12, 2022, pp. 965-978. IEEF, 2022.
Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. Federated learning: Challenges, methods, and future directions. IEEE Signal Process. Mag. 3703):00-60, 2020a.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization heterogeneous networks. In Proceedings of Machine Learning and Systems 2020, MLSys 2020, Austin, TX. USA, March 2-4, 2020. misys.org, 2020b.
Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. Ditto: Fair and robust federated learning through personalization. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 6357 6368. PMLR, 2021c.
Paul Pu Liang, Terrance Liu, Liu Ziyin, Nicholas B Allen, Randy P Auerbach, David Brent, Ruslan Salakhutdinov, and Louis- SoiisPPhllippe Morency. Think locally, act globally: Federated learning with local and global representations. arXiv preprint arXiv:2001.0152. 2020.
Tao Lin, Lingjing Kong, Sebastian U. Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, 2020.
Yishay Mansour, Mehryar Mohri, Jae Ro, and Ananda Theertha Suresh. Three approaches for personalization with applications federated learning. arXiv preprint UrXiv:2002./061/ 2020.
Othmane Marfoq, Giovanni Neglia, Aurélien Bellet, Laetitia Kameni, and Richard Vidal. Federated multi-task learning under a mixture distributions. arXiv preprint arXiv:2108. 10252, 2021.
Othmane Marfoq, Giovanni Neglia, Richard Vidal, and Laetitia Kameni. Personalized federated learning through local memorization. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 15070-15092, 2022.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agtiera Arcas. communication-effiitaa learning of deep networks from decentralized data. International Conference Artificial Intelligence and Statistics (AISTATS), volume 54, pp. 1273-1282, 2017.
Umberto Michieli and Mete Ozay. Prototype guided federated learning of visual feature representa- tions. arXiv preprint arXiv:2105.08992 2021.
Xutong Mu, Yulong Shen, Ke Cheng. Xueli Geng, Jiaxuan Fu, Tao Zhang, and Zhiwei Zhang, Fedproc: Prototypical contrastive federated learning non-iid data. CoRR, abs/2109.12273. 2021.
Tomoya Murata and Taiji Suzuki. Barianeri reduced local SGD for less heterogeneous federated learning. In Proceedings of the 38th International Conference Machine Learning. ICML 2021, 18-24 July 2021. Virtual Event, 2021.
Jachoon Oh, Sangmook Kim, and Ye-Young Yun. Fedbabu: Toward enhanced representation for federated image classification. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022, 2022.
12

---

Published as a conference paper at ICLR 2023
Yingwei Pan, Ting Yao, Yehao Li, Yu Wang, Chong-War Wah Ngo, and Tao Mei. Transferrable proto- typical networks for unsupervised domain adaptation. In JEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019 pp. 2239-2247 Computer Vision Foundation/ IEEE, 2019.
Felix Sattler, Klaus- Müller, and Wojeiech Samek. Clustered federated learning: Model- agnostic distributed multitask optimization under privacy constraints. IEEE Trans. Neural Networks Learn. Syst. 32(0):3710-3722, 2021.
Aviv Shamsian, Aviv Navon, Ethan Fetaya, and Gal Chechik. Personalized federated learning using hypernetworks. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 9482-9502. PMLR, 2021.
Virginia Smith, Chao-Kai Chiang, Maziar Sanjabi, and Ameet S. Talwalkar. Federated multi-task learning. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 4424 4434, 2017.
Jake Snell, Kevin Swersky, and Richard S. Zemel. Prototypical networks for few-shot learning. In Advances Neural Information Processing Systems 30: Anmual Conference on Neural Information Processing Systems 2017. December 4-9, 2017, Long Beach. CA, USA. pp. 4077-4087, 2017.
Alysa Ziying Tan, Han Yu, Lizhen Cui, and Qiang Yang. Towards personalized federated learning. arXiv preprint (rXiv:2103.00711 2021a.
Yue Tan, Guodong Long, Lu Liu, Tianyi Zhou, and Jing Jiang. Fedproto: Federated prototype learning over heterogeneous devices. CoRR, abs/2105.0024; 2021b.
Minxue Tang, Xuefei Ning, Yitu Wang, Yu Wang, and Yiran Chen. Fedgp: Correlation-based active client selection for heterogeneous federated learning. CoRR, abs/2103.13822 2021.
Xinyi Tong, Xiangxiang Xu, Shao-Lun Huang, and Lizhong Zheng. A mathematical framework for quantifying transferability in multi-source transfer learning. In Advances in Neural Information Processing Systems 34: Anmual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pp. 26103-26116, 2021.
Hao Wang, Zakhary Kaplan, Di Niu, and Baochun Li. Optimizing federated learning on non-iid data with reinforcement learning. In 39th IEEE Conference on Computer Communications, INFOCOM 2020, Toronto, ON, Canada, July 6-9, 2020, pp. 1698-1707. IEEE, 2020.
Lixu Wang, Shichao Xu, Xiao Wang, and Qi Zhu. Addressing class imbalance in federated learning. In Fiitt AAAI Conference on Artificial Intelligence, AAAI 2021, Virtual Event, February 2-9, 2021, pp. 10165-10173. AAAI Press, 2021.
Hongda Wu and Ping Wang. Node selection toward faster convergence for federated learning non-iid data. CoRR, abss21155.0004. 2021.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnis a novel image dataset for benchmarking machine learning algorithms. CORR, 708.07747 2017.
Bing Xu, Naiyan Wang, Tianqi Chen, and Mu Li. Empirical evaluation of rectified activations in convolutional network, 2015.
Xiangxiang Xu and Shao-Lun Huang. Maximal correlation regression. IEEE Access, 8:2659 2020. doi: 10.
Qiang Yang, Yang Liu, Tianjian Chen, and Yongxin Tong. Federated machine learning: Concept and applications. ACM Trans. Intell. Syst. Technol. 10(2), January 2019. ISSN 2157-6904.
Rui Ye, Zhenyang Ni, Chenxin Xu, Jianyu Wang, Siheng Chen, and Yonina C. Eldar. Fedfm: Anchor-based feature matching for data heterogeneity in federated learning, 2022. URL https
w/axxiv..rgaabs////vova 07615
13

---

Published as a conference paper at ICLR 2023
Tehrim Yoon, Sumin Shin, Sung Ju Hwang, and Eunho Yang. Fedmix: Approximation of mixup under mean augmented federated learning. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
Jie Zhang, Song Guo, Xiaosong Ma, Haozhao Wang, Wencao Xu, and Feijie Wu. Parameterized knowledge transfer for personalized federated learning. arXiv preprins arXiv:2111.02862, 2021a. la
Michael Zhang, Karan Sapra, Sanja Fidler, Serena Yeung, and Jose M. Alvarez. Personalized federated learning with first order model optimization. In 9th International Conference Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021b.
Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Federated learning with non-iid data. arXiv:1806. 00580.05822.0 2018.
Tailin Zhou, Jun Zhang, and Danny Tsang. Fedfa: Federated learning with feature anchors to align feature and classifier for heterogeneous data, 2022. URL
arg/abs/2211 2211
09299
Hangyu Zhu, Jinjin Xu, Shiqing Liu, and Yaochu Jin. Federated learning on non-iid data: A survey. Neurocomputing, 465:371-390 2021a.
Zhuangdi Zhu, Junyuan Hong, and Jiayu Zhou. Data-free knowledge distillation for heterogeneous federated learning. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 12878-12889, PMLR, 2021b.
14

---

Published as a conference paper at ICLR 2023
APPENDIX
A PROOFS OF THEORETICAL RESULTS
A.1 RELATION BETWEEN X -DISSANCE AND K-L DIVERGENCE
p(t) Yix ())x)
(1 E
(z)
Σ
p( Prix (1lx) log
(20)
(y
DEX
BEY
With log(1 1 + 2) ~ x
p() (1)x))) ~ Py)
2
(Prxx(l>>)) Q (v)))
Eg(g) Σ
X
(21)
ZE
yey
(PD XIX (u/x) - Orix(1))))
(22)
(v)
ZEN
VEY
pl02 (x) (x - Qvix(v()))
(23)
(x)
ZEX BEY
X vIx (ulx) PS (1)
(1)
(24)
(x)
ZEX yEy
(P XY. pk0 1 (9)
(25)
A.2 PROOF OF THEOREM 1
For better readability, we denote (y) t the uniform distribution over the label space. We first express the 9i. Since we have
2
(00) ( (1) XY: ())x(9))
Σ
(26)
NEY (x)
then, for all y' EV,
0x3 xx, p() 0 (g)
(p0) p(()
Og(v')
= Σ 8' +2 Σ ( F(()((()) f(x) ZEX
xEX
-2 (1 xEX
(27)
where to obtain the last equality, we have used the assumption that E Ep [f(X) (in im- plementation, we can apply zero-centering to satisfy this assumption) and the notation that
A ( A E p() ((())((X)
f
Set the gradient
27 to zero, and we obtain
g.(y) = Py Pv)
[A
1-
Σ REX
500
(28)
15

---

Published as a conference paper at ICLR 2023
We precisely express the testing error (9)
R((o::) 18 xx²
(P (p
p Q (0:))
m
m
Xi
p(t XY
Bigy p Q (9:)
(
Σ
Xi
(xX) Q
(g:), px 2 1rix (ix(()))
(29)
where gj = Eīğ is the expectation over the data sampling process.
Next, for the first term, we have
xi
p(0) XY
xi (Pxr. PSu Q (g:)
1=1
P vix (g:). Σ + x3
ai
Q x 091x(11))
p
(30)
which comes from the fact that for any g. have
( (2.
x Q (g. (z.) w)
[PX Q 2 ())((,)))))))))
Σ REA,
pk (x)
Σ
PPy - - ( Py (w)
Σ
(X)(
=EX,EEy
E
(1)
(x) (y
1111
Σ Exxr (2(o))
NEX.EE
LEx
E aa
(X))
(x)))((()))
0.
(31)
Then, for the bias term in the RHS of 30 we have
X
pl
YIX
(9t)
ag
(g,)
Kijaki ΣP (z) (f) (F 8T (())gr(())
2.
ΣΣX
Oig Diy'
(32)
j=ly==1
where by Assumption we further have
Djj'= = Σ p() (I))
(F - 8T (x)gg(())
F.
tr
(1) (A
(h(i,))
(33)
DEy
and
htl, y) A P(0 (y) E XiY=y
[f(X), 1= 1....,, m.
(34)
16

---

Published as a conference paper at ICLR 2023
Finally, for the variance term in (29 we have
V3- ny EE
x2
p 0
x
(g)),
P Q xx))
Σ (F (g) 9;))² -9;))²
nj EE
Σ ( f(x) s(oo)) )
ny E
Σ (x)
f
LDEK
D
tr [A | Var
Σ (p (2))
- pk) (2.( {(x))) )(0))
nj
y=1
| Σ
plD Y (y) tr [A.)] E
Ep
[((X))(((XI - -Ie (e))
1 B Bppp [F(X)]
x|Y=
xiv==
==1
(35)
A.3 EXPRESSIONS OF THE QUANTITIES CONTAINED IN
In computing (.) after obtaining f. we can estimate the quantities by the empirical expectations for each i E [m]. including
(1) - (P)(()))
(36)
A
Ag & ppe (((X)F((x)))
(37)
E p(o
((X))T(X)) +- Aiv - E
pco X
[( (X)):
(38)
X
E pro
[f(x) +
P:,
E ALO
[f(X). xiV=
(39)
Now, the quantities contained in could be estimated accordingly. Then, a" can be computed by solving non-negative quadratic programming problem (e.g., using evxpy toolbox).
A.4 EXPRESSIONS OF THE GENERALIZATION GAP IN 14
For fixed classifier head gi and dataset D, at client i, we have
R; =x=x² (2/(pp)))) ((1) (i)
yix (1x((.))) (9:)
=x² (P/i) pby + XY, Q (9))
p(0) p(0) Q
(P0) pl y) pl) v) + p(0) (x,
- - -
pf)
NEXYEY
= x (Ply) ((0) 2 (xxx()))))))) + Σ 2 (2.0) - p()) ( f(.)) g(()K (PLY) pLP)
2.9
AR
(40)
where the first term is the empirical loss and the last term is a constant C₁ that measures the discrepancy between the empirical data distribution and the true underlying data distribution client
17

---

Published as a conference paper at ICLR 2023
For the second term AR, by assuming the empirical marginal distribution py identical the true marginal distribution py", we can further rewrite it by
A = 2 Σ P (1)
Σ
p()) ( - xix(xll)) F(1))
gi(u)/K
x
2 p(o
I6
p(o) XIY
[f()) - E po [f(x)] 9.(Y)/K,
(41)
where the empirical conditional distribution can be estimated by local dataset D;, while the true conditional distribution is unknown. However, profiting from the data available at other clients in FL, the true conditional distribution could be better estimated by calculating the global feature centroid of each class. Denote cy the feature centroid of class yt, we can further get the following estimate
A R
Cyn f(x))) gi(l)/K
(42)
T4
1=1
A.5 PRACTICAL CONSIDERATION FOR ALGORITHM DESIGN
As shown in A.2 and A.3 to estimate the combination weights we need the covariance matrix of feature embeddings, which has the size of d² and could be large the feature dimension d increases. For the algorithm design, we make some slight modifications that do not affect the final performance. In consideration of the request of communication efficiency and privacy protection, directly use the identity matrix to replace the feature covariance matrix of each client
la.
(43)
This operation compromise and can be inspired by the following fact. The cardinality is extremely large number and the feature f(x) is low-dimensional vector. When the random noise (variance of all possible samples) added in the sample space (not Gaussian) projected onto the low-dimensional space, the corresponding variance would be isotropic, which can be reflected as an identity matrix with constant scaling and the scaling can be WLOG omitted in our calculation as does not affect the results.
With this approximation, we can simplify the quantities and 33) into
R (v) (O) tr
|FXDI|??
Vn +
( Y
Epm
E P
[F(X)
( (0) E P
(44)
y=1
and
Dip
Σ (h(i.v) ((.-))((())))((.)) - =( NEY
(45)
By now, we can easily calculate the values V, locally and get the values of Day'' by collecting μi (P (1) from each client ( [mz]. Then, can estimate the combination weights for each client by solving a quadratic programming problem (46) efficiently.
R.(o;)) =
Oaj Djy'
n2j
m2
Σ
Σ
p
(y) tr (A)) - -1 (1))5)))))))
n2y
j=1
+
-Σ
Oig' tr
Σ
p(0 (u)paso 1100 (1) [200) (u)u - ( T
(2)
yEy
(46)
18

---

Published as a conference paper at ICLR 2023
Algorithm FedPAC
Input:
Learning rate {my. Mg),, number of communication rounds T, number of clients m,
number of local epochs E, hyper-parameter X
2: Server Executes:
3: Initialize:w (0) and (
4: for / - 1, .....
do
5: Select client set C+
6: Broadcast ) to selected clients for local update
7: Collect models and statistics from clients
8: Get global feature extractor ((++1) as (18) and global centriods e(t+1) as 19
9: Get personalized classifier Φ (1++1) by solving
Send (0(++1), . (((+1)) back to client E C.
end for ClientUpdate(i, (t),, )
12:
13: Update local feature extractor 0 <- ((t)
14: Extract feature statistics u4; and (2(1)
Train (((+1) for epoch and a for E epoch by turns
16: Extract local feature centroids a as (17
Return (0((+1) o (++1) (1+1)
(( to server
It is also worth noting that the feature statistics used for estimating combination weights are calculated before local feature extractor training, while the class-wise feature centroids used for estimating global feature centroids obtain a good estimate of updated global feature centroids without extra server-client communications. The main training procedure is summarized in Algorithm
extracted after updating local feature extractors. By such compromise, can
A.6 COMMUNICATION AND PRIVACY CONCERNS
We acknowledge that FedPAC requires the transmission of local feature statistics between clients and server. However, this does not raise high privacy and communication issues since local feature statistics represent only an averaged statistic over all local data of www-iiensioona feature represen- tations. On the other hand, without privacy protection circumstances, is hard to strictly guarantee the privacy. Even the vanilla FedAvg that only shares and aggregates local models would need more crivacy-preservinn techniques to enhance the system reliability practice, e.g., homomorphic encryption (HE). With homomorphic encryption, the clients receive private keys to homomorphically encrypt their models before sending them the server. The server doesn't 't own key and only sees the encrypted models. But the server can aggregate these encrypted weights and then send the updated model back the client. The clients can decrypt the model parameters because they have the keys and can then continue with the next round of local training and feature statistics extraction.
Now come back to the framework of FedPAC, where not only local models but also category- level feature statistics are transmitted, the aggregation of models can still be protected by applying HE techniques while the category-level feature statisties could be directly used compute the personalized combination coefficients. We acknowledge that to further quantify the potential leakage from prototypes would involve in designing specific inversion attacks, which are out of the scope of our work. But we consider that the additional privacy risk may not be an issue when only category- level feature statistics are provided server while the model parameters are encrypted. Without accessing the feature extractor layers, the server is hard to infer meaningful semantic data information from only the prototypes. And the privacy-preeerving techniques are already equipped in many open-source FL libraries and widely applied in real-world applications.
Finally, would like point out that the trade-offs between utility, security and efficiency still exist in the context of personalized FL. Our framework mainly focuses on improving the model utility, for which additional communication and computation are required. We consider that the performance improvement is the main incentive for a client to participate the federated learning. In such cases, the clients may hope exposure more information for better collaboration.
19

---

Published as a conference paper at ICLR 2023
B DETAILS OF EXPERIMENTAL SETUP
All experiments are implemented in Py and simulated in NVIDIA GeForce RTX 3090 GPUs. Codes for the results in this paper provided in the supplementary material.
B. DATASETS AND MODELS
We consider image classification tasks and evaluate our method on four popular datasets: (1) EMNIST
(Extended MNIST) 62-class image classification dataset, extending the classic MNIST dataset
(Cohen et al., 2017). ) It contains 62 categories of handwritten characters, including 10 digits, 26
uppercase letters and 26 lowercase letters; (2) Moshion-MNIST with 10 categories of clothes (Xiao)
et al. 2017): (3) CIFAR-10 10 with 10 categories of color images (Krizhevsky & Hinton (009);
and (4) CINIC-10 (He et al., 2020), which more diverse than CIFAR-IO as it is constructed from two different sources: ImageNet and CIFAR-10. We construct two different CNN models and CIFAR-1O/CINIC-10, respectively. The first CNN model constructed by two convolution layers with 16 and 32 channels respectively, each followed by max pooling layer, and two fully-connected layers with 128 and 10 units before softmax output. And we use the LeakyRel.U (Xu et al., 2015) activation function. The second CNN model is similar to the first one but has one more convolution layer with 64 channels.
for EMMNIS/Fashhon-MNNT
B.2 DATA PARTITIONING
Similar to (Karimireddy et al. 2020b Zhang al., 2021b Huang al. 2021). make all clients have the same data size, in which 5% of data (20% by default) are uniformly sampled from all classes and the remaining (100- 8)% from set of dominant classes for each client. We evenly divide all clients into multiple groups while in each group the clients have the same dominant classes. Specifically, for Rashion-MNiST, CIFAR-10 and CINIC-10 datasets which have 10 categories of images, we divide the clients into 5 groups and use three consecutive classes as the dominant class set for each group, where the starting class is O, 2, 4, 6 and 8 for the groups, respectively. For EMNIST dataset, we divide the clients into groups and use the digiss/upperrass/lowewaas as the dominant set for each group, respectively.
B.3 IMPLEMENTATION DETAILS
Training Settings. We employ the min-batch SGD as the local optimizer for all approaches. The step size 1) of local training set to 0.01 for EMNIST/Fashin-MNNNN Notice that our method alternatively optimizes the feature extractor and the classifier. To reduce the local computational overhead, we only train the classifier for one epoch with a larger step size Mg = 0.1 for all experiments, and train the feature extractor for multiple epochs with the same step size 7y 17 as other baselines. The weight decay is set to 5e-4 and the momentum is set to 0.5. The batch size is fixed to B = 50 for all datasets except EMNIST, where we set B = 100. The number of local training epochs set E for all federated learning approaches unless explicitly specified. And the number of global communication rounds is set to 200 for all datasets, where all FL approaches have little or no accuracy gain with more communications, For all methods, we report the average test accuracy of all local models for performance evaluation.
and 0.02 for CIFAR-IO/CINIC-
Compared Methods. We compare our proposed FedPAC with the following approaches: Local-only, where each client trains model locally without collaboration between clients; model interpolation based method APFL (Deng et al. 2020); multi-task learning based methods, including PFedMe (Dinh et al. 2020) and Ditto (Li et al.) 2021c);; parameter decoupling based methods, including LG- FedAvg (Liang
2020) with local feature extractor and global output layers. FedPer (Arivazhagan ct al. 2019) and FedRep (Collins et al. 202I) that learn personal classifier on top of a shared feature extractor, FedBABU (Oh et al. 2022) that keeps the global classifier unchanged during the feature representation learning; ensemble methods that utilize both global and local models, including Fed- RoD (Chen & Chao 2022) and kNN-Per (Marfoq al. 2002);; FedFooo (Zhang et al.) (222lb) that performs weighted combination of local models, and pFedHN (Shamsian et al. 2021) that employs server-side hypernetwork to generate personalized models.
Hyper-parameters. For FedPAC, we tune the penalty coefficient À over (0.1, 0.5, 1,5, 10) and fix it to for all settings. For APFL., tune the parameter over (0.25, 0.5, 0.75) and set 0.25 for
20

---

Published as a conference paper at ICLR 2023
all settings; For pFedMe, search À over (0.1, 100} and set to 10, and also choose K for all settings; For, Ditto, we tune the parameter À among {0.1, 1, 10), and use for all settings. tune the interpolation parameter over {0.5, 0.6, 0.7, 0.8). For FedFomo,
For KNN-Per, the number of models downloaded learning rate over {0.1, 0.01, 0.001) and then fix both the server and client learning rate to 0.01.
set as recommended in the paper. For pFedHN, we tune the
C ADDITIONAL EXPERIMENTAL RESULTS
C.I PERSONALIZED CLASSIFIER COMBINATION WEIGITT
In this part, investigate the classifier combination FedPAC by estimating optimal client-to-client weights overtime, visualizing the calculated coefficient matrix during training process in Figure 2 and Figure 3 for EMNIST and CIFAR-10 datasets, respectively. We depict clients in the same group next to each other (e.g. clients 0, 1, 2, 3 belong to group in CIFAR-10 setup). The weights are calculated according to A.
A.3 and recorded in round 1, 20 and 50 arranged from left to right in each subfigure. The results demonstrate that our method can reliably encourage clients with similar data distributions collaborate more and prevent collaboration between unrelated clients. And it can be found that when data distributions across clients are highly heterogeneous, each client tends collaborate only with those clients belonging to the same group, such as Figure (b) and Figure (b)) When the data distributions are more homogeneous, each client will perform the combination of classifiers from all the other clients obtain better personalized one, as in FigureZid and Figure
(a) 0%
(b) 20%
(c) 3c) 50%
(d) 80%
Figure 2: Classifier combination weights on EMNIST datasct under various levels of data heterogencity.
(a) 0%
(b) S 20%
Cosa 12
Cit
Citb Lihorr
(c)
50%
(d) 80%
Figure 3: Classifier combination weights on CIFAR-10 datasct under various levels of data heterogeneity.
21

---

Published as a conference paper at ICLR 2023
EMNIST
Fashior-MNIST
80-
90-
Text Accursiyy 1%6
85
70
Accurscy
FodAvg
Fedfug-FT
80
FedAvg-FT
60-
FedRep
PedRep
APFL
0 775
APPL
50
pResHiN
70-
pFosHNN
FedPAC
FedPAC
80
Communicatiionn Rounds
Communication Rounds
(a) EMNIST
(b) Fashion Fashion-MNIST
CIFAR-10
CINIC-10
70
60
Accuracy
Fecevg
FedAvg
60
FodAvg
50
FT
FodRcp
FedRop
APFL
Test
40
APFL
Ditto
Dido
FodFomo
FedFomo
40
30-
pFediNN
FodPAC
FedFllx
30
80
160
20
160
Communication
Rounds
(c) CIFAR-10
(d) CINIC-10
Figure 4: Test accuracy over communication rounds under different training methods for FL with 20 clients.
C.2 CURVES OF TEST ACCURACY DURING TRAINING PROCESS
Figure Paeueett the evolution of average test accuracy over global communication rounds for partial experiments shown in Table From which, it can be seen that our method almost outperforms other baselines during the whole training process, except the FedFomo has better model accuracy in the initial training phase. Besides, the accuracy curves of FedAvg with local fine-tuning almost overlap with Ditto, and result in competitive performance as other start-of-the- methods, which demonstrate that FedAvg with local fine-tuning is simple yet effective, and tough-to-beat method.
C.3 EFFECT OF LOCAL EPOCHS.
To reduce communication overhead, the clients tend to have more local training steps, which can lead to less global communication and thus faster convergence. Therefore, we monitor the behavior of FedPAC using number of values of E, whose results are given in Figure 5 The results show that larger values of E benefit the convergence speed and never hurt the model accuracy under our framework. Nevertheless, there is trade-off between the computations and communications. ie.. while larger requires more computations at local clients, smaller E needs more global communication rounds to converge.
C.4 COMPARISON WITH SINGLE/MULTIPLE GLOBAL MODEL(S) BASED METHODS
In this part, we compare our method with improved FL algorithms that only learn single global model and the multiple-mode models based approaches. We choose the two popular baselines in conventional FL, i.e., FedProx [Li et al.) 2020b) and SCAFFOLD (Karimireddy et al. 2020b), to learn single global model. For methods learning multiple global models, we evaluate the HypCluster in (Mansour et al.) 2020) and the recently proposed FedEM (Marfoq et al. 2021). We construct two experimental settings, where the number of global models is set 3 and 5, respectively. is worth noting that we may not know the amount of latent distributions in advance, therefore it could be non-trivial to select the number of model components clusters for striking balance between communcction//comppaaai
22

---

Published as a conference paper at ICLR 2023
CIFAR-10
Test Accuracy
60
50
FedAvg, E=2
FedPAC, E=2
FedAug, E-5
FedPAC, F=5
FedAvg,
FadPAC,
FedPAC, E-20
FedPAC, E-20
40
80
120
160
200
Communication Rounds
Figure 5: Performance comparison in average test accuracy CIFAR-10 10 datasct with varying local epochs.
overhead and the performance of personalized models. We report the results in the following Table 4 where we also evaluate the local fine-tuned version of FedEM method (FedEM-FT), which is not applied in the original paper. From the results we can find that FedEM with local fine-tuning can lead satisfactory performance most cases at the expense increased communication and computation costs, and the high performance could be attributed to the model ensemble, which however would also increase the computational burden during local inference.
Table 4: The comparison of final test accuracy (%) on different datasets. We apply full participation for FL system with 20 clients, and apply client sampling with rate 0.3 for FL system with 100 clients.
EMNIST
Pishion-MNIST
CIFAR-10
CINIC-10
Method
20 clients
100 clients
20 clients
100 clients
20 clients
100 clients
20 clients
100 clients
FedProx
72.38
75.92
85.20
86.79
69.52
73.03
$8.19
61.91
SCAFFOLD
72.66
76.68
86.29
89.31
72.56
77.22
62.58
70.98
1lypCluster (k=3)
74.52
87.68
88.63
90.08
70.16
78.91
64.95
70.18
HypCluster (k=5)
83.30
88.04
89.08
91.50
73.80
79.57
71.38
FodEM (M=3)
73.48
75.77
85.97
87.43
72.44
77.01
RodEM-FT (M=3)
83.28
86.65
90.36
91.54
79.69
82.46
74.21
77.02
RodEM (M=5)
74.83
77.85
86.09
87.96
73.41
76.63
63.32
66.21
FodEM-FT (M=5)
83.42
86.72
90.56
91.59
80.17
83.15
74.47
78.35
FedPAC (ours)
86,46
89.88
91.83
92.72
81.13
83.36
74.96
77.36
C.5 COMPARISON WITH FEDPROTO
We notice that recent work FedProto Tan al.) 2021b) also applied a similar regularization for model training. However, we find that only sharing global prototypes without the shared feature extractor would exhibit little performance improvement as shown in the following table. We evaluate the FedProto with locally trained classifiers in our experimental settings. where FedProto-G means using global prototype regularize the local training without model exchanging and FedProto-L means only using local prototype without any communication. From the results we could find that both of them lead to similar results and cannot even outperform FedAvg-FT.
In this paper, the feature alignment in our method is inspired by our analyses, which focus on the principle of minimizing the testing loss and illuminates that the local-global feature distribution discrepancy would increase the generalization error. The theoretical result is consistent with the intuition that the similar local regularization term to force the model to learn more compact intra-class features. And it is also easy understand why local prototype to substitute global prototype can also
understand why feature alignment would make sense. Consequently, we apply
23

---

Published as a conference paper at ICLR 2023
Table 5: Comparison of accuracy (%) with FedProto. We 20 clients with full participation.
Dataset
Local-only
FodAvg-FT
FedProto-G
FedProto-L
FedPAC
Fashion-MNIST
85.68
90:47
87.47
87.45
91.83
CIFAR-10
65.43
76.56
67.48
67.45
81.13
achieve similar performance. From this point, the contribution in the part of feature alignment may not lie in the algorithm implementation but in the interpretation and understanding.
C.6 RESULTS IN DATA IMBALANCED SETTING
For the main experiments, the training sample sizes are balanced for different clients and it is of significant interest to investigate the effect of imbalanced data sizes for more heterogencous settings. Instead of the fixed size of 600 samples per client, here we randomly choose the size from {300, 600, 1200} for each client regardless the local label distributions. It is worth noting that focus on the relatively mild imbalanced setting consideration of the potential straggler issues in FL when data sizes are highly imbalanced, which could be problem of interest to investigate separately. We evaluate various methods on the Pashion-MNISSTCCEE which we can find that our method is robust to the quantity-skew and still outperforms other baselines.
10 and show the results in table from
Table 6: Comparison of test accuracy (%) under imbalanced setting with 20 clients and sampling rate 1.0.
Dataset
Local-oaly
FodAvg-FT
FedRep
Ditto
Fed-RoD
pFedHN
HypCluster
FedEM
FedPAC
Fashion-MNIST
85.65
90.75
90.13
90.71
89.95
87.67
88.36
90.72
92.33
CIFAR-10
64.68
77.32
71.28
78.32
77.48
78.12
73.43
79.26
81.58
C.7 RESULTS PATHOLOGICAL NON-IID SETTING
As a supplement, we also add some results on the pathological non-lII setting with 20 clients, where each group has different number of classes. More precisely, there are 3/3/5/5/8 different classes for clients in each specific group and training samples are balanced across classes at each client. We conduct two sets of experiments with different class allocations, where the local sample size is still fixed to 600 for each client. In the first set of experiments, the class set is randomly selected for each client and could be varied even for clients in the same group. while in the second set of experiments, the class sets differ across groups but are shared for clients belonging to the same group. We take the example and compare the following table. From the numerical results we could conclude that even though the hardness of local task varies across clients when the number of classes is different, our method still consistently achieves the best averaged accuracy attributed to the feature alignment and classifier collaboration mechanisms, which demonstrates the wide utility of our proposed framework in label distribution skew scenarios.
CIFAR- 10
method with several baselines, reporting the results in
Table 7: Test accuracy (%) under pathological setting of CIFAR-10 with 20 clients and sampling 1.0.
Class Allocation
Local-only
FedAvg-FT
FedRop
Dirto
Fed-RoD
DFodHN
HypCluster
FodEM
FedPAC
Random
67.57
77.42
72.71
77.59
74,85
77.23
66.52
80.26
80.53
Clustered
64.62
75.85
69.76
76.25
73.58
77.51
72.98
78.71
79.65
C.8 RESULTS DIRICHLET ALLOCATION SETTING
We also apply another widely used non-IID data partition strategy, i.e., distributing samples with the same label across the clients according to a Dirichlet distribution with a concentration parameter 3. Specifically, sample pk Dirm (9) and allocate Pk,i proportion of the instances of class
24

---

Published as a conference paper at ICLR 2023
k to client In this work, set 8 = 1.0 and consider FL setup with 100 clients, evaluating proposed method as well as other baseline methods on the Fashion-MNIST and CIFAR-10 datasets. The test accuracy results recorded in the following Table g from which method again outperforms the other baselines, exhibiting the highest performance in both cases.
see that our
Table 8: Comparison of test accuracy (%) under Dirichlet allocation with 100 clients and sampling rate 0.3.
Datasel
Local only
FedArg-FT
FedRRp
Ditlo
pFed
HyppCluster
FedEM
FedPAC
Fashion MNIST
86.78
89.57
91.43
CIFAR-10
34.68
61.88
44.54
73.53
56.39
75.74
73.69
76.79
78.53
C.9 RESULTS IN HIGHLY DATA-SCARCE. CASE
Notice that previous work usually partitions all the samples from the whole dataset to clients, while only assign fixed number of samples to cach client. Thus, the amount of training samples for each client in our work is not such large, which also the main motivation for classifier collaboration. Consequently, the Local-only method cannot achieve satisfactory performance. If local data size sufficiently large, stand-alone training or FedAvg-FT should achieve similar accuracy with our method. Here we also provide additional experimental results for relatively higher data-scarce case. We evaluate several competitive baselines on CIFAR-IO with 1000 clients, where we allocate only 50 samples (mini-batch size) for each client with default partition strategy and randomly select 30 clients each round. The results listed in the following table clearly indicate that our method outperforms others with more significant improvement.
Table 9: Test accuracy (%) on CIFAR-10 with 1000 clients and sampling rate 0.03.
Method
Local-only
FedAvg-FT
FedRop
Ditto
FedPAC
Acc(%)
44.91
61.12
47.45
61.32
68.56
C.10 RESULTS IN FEATURE SKEW SETTING
is worth noting that our framework for classifier collaboration actually does not need the assumption of shared class-conditional distributions, which means it can automatically distinct those classifiers that can promote the performance for the target client and exclude the irrelevant ones. One example is the concept shift setting provided in the main text. Now, we provide another example of the setting with feature distribution skew, where we rotate the images in each group with a certain degree. We compare some methods Fashion-MNIS: and from the results we can find that our method still achieves the best performance.
Table 10: Test accuracy (%) on Fashion MNIST under feature skew setting with clients.
Method
Local-only
FodAvg-FT
FedRop
Ditto
FedBABU
Fod-RoD
FedPAC
Aco(%)
78.53
81.76
79.86
82.16
81.29
80.27
85.56
25


Reviews:
[{"review_id": "plJzRlcfK-", "reviewer": "Reviewer_zWd3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new learning framework to tackle the label distribution skew challenge in federated learning. Specifically, the paper introduces a novel feature representation alignment technique and a classifier combination technique to exploit both shared representation and inter-client classifier collaboration. By using a local regularization term related to the global feature centroids of each class, the framework mitigates the drift of local features during the learning process. By introducing a linear combination of classifiers with optimized coefficients during training, the framework allows similar clients to share more information while avoiding negative transfer from more distinct clients.", "summary_of_the_review": "This paper introduces global feature centroids in feature regularization and classifier combination weights selection through quadratic programming to improve personalized federated learning. The proposed method is backed by good intuitive interpretation, theoretical analyses, and convincing evaluation results. The ideas introduced in this paper could be inspiring to the relevant community.", "strength_and_weaknesses": "Strengths: The proposed method is well-motivated. The use of global feature centroids in local training is straightforward and practical, and the idea of forming classifiers combination coefficient selection as a quadratic programming problem is elegant. Evaluation and ablation studies seem convincing.\n\nWeaknesses: The proposed framework tackles the specific label distribution skew challenge in federated learning. It would be nice if the authors could discuss whether the method generalizes to other challenges caused by heterogeneous data distributions in federated learning, for example, when the number of classes is different (if so, whether the method needs any adaptations; if not, why the method has certain limited applications).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "YwTySLVt2a", "reviewer": "Reviewer_Ddph", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work solves personalized federated learning (PFL) setting by developing feature alignment mechanism and classifier collaboration. Concretely, this feature alignment minimizes the distance between local feature representations and global centroids. And classifier collaboration utilizes linear combination of multiple local classifiers to customize new classifier. The current work conducts experiments on four datasets to verify the effectiveness of proposed method.", "summary_of_the_review": "This work solves personalized federated learning (PFL) setting by developing feature alignment mechanism and classifier collaboration. Concretely, this feature alignment minimizes the distance between local feature representations and global centroids. And classifier collaboration utilizes linear combination of multiple local classifiers to customize new classifier. The current work conducts experiments on four datasets to verify the effectiveness of proposed method.\n\nHowever, I have several concerns on this work.\n\n1. Eq. (1) is the optimization goal of traditional federated learning (FL) instead of that of PFL task.\n\n2. The feature alignment regularization needs to access the global centroids which are obtained by aggregating local centroids. Under this condition, the data distribution information of each client has been leaked out, which violates the requirement of privacy protection in PFL task.\n\n3. In classifier collaboration, why the simple linear combination of local classifiers can find these clients with the similar distributions? It would be better to analyze the combination coefficients to support this point.\n", "strength_and_weaknesses": "Strength:\n\n1. The improvement of experimental results is significant.\n\nWeaknesses:\n\n1. The description of some details in the proposed method is not clear.\n\n2. Need to make comparisons with some recent PFL works to verify the effectiveness.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "XvIoG7SNOg", "reviewer": "Reviewer_8wN3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes to deal with the data heterogeneity in federated learning by local-global feature alignment and weighted combination of local classifiers. The whole model is decomposed into two parts, a feature extractor $f$ and a linear classifier $g$ built on $f$. The feature alignment is achieved by extracting feature centroid for each label class. The parameters of each local classifier $g_i$ is combined with other classifiers by optimized weights so as to bundle similar local clients. ", "summary_of_the_review": "The paper is overall well-written but there are some critical issues that need to be addressed. I'm happy to increase my evaluation if my concerns (see weakness) are addressed.\n\n----------- Post rebuttal ----------\n\nAll of my concerns are well-addressed. I increased my evaluation from 5 to 8.", "strength_and_weaknesses": "**Strength**\n1. The proposed framework is novel. The decomposition of feature learning and linear classification helps customize local clients.\n2. The performance is theoretically analyzed based on a simplified model and numerically validated by experiments on several datasets.\n\n**Weakness**\n1. The heterogeneity of data considered in this paper is a specialized one. Generally, the heterogeneous local data distributions will be different in both total number of samples and number of feasible label classes. Current setting assumes each client has the same size of data and observes the full set of label classes. It is interesting to see the performance of the proposed method when the local data distributions are heterogeneous enough.\n2. In Eq. (3), the authors assume there only exists one centroid for each label class. This assumption needs to be better explained. It is likely that samples in one label class may come from two different centroids. Simply averaging all the features as one centroid may not always be true.\n3. In Eq. (4), the authors mention $\\alpha$ is optimized to reduce the local testing loss. It is not clear whether the local test data is different from the test data used to show Table 1.\n\n\n\n  ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "CinfydAUCB", "reviewer": "Reviewer_ivCQ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose FedPAC for personalized federated learning, which aims to learn a better representation by leveraging global semantic knowledge. Comprehensive experimental results multiple benchmarks demonstrate the effectiveness of the proposed framework.", "summary_of_the_review": "I am generally optimistic about this submission. Considering the well-motivated problem, novel framework and impressive experimental results, I have no negative comments on the quality of the paper.", "strength_and_weaknesses": "Strength\n\nThe structure of the proposed FedPAC is reasonable and easy to follow.\nThe studied problem (personalized federated learning) is interesting and different applications may benefit from it.\nthe experiment results are extensive and impressive\nThe authors also give a theoretical analysis.\n\nWeaknesses\n\nThe description of motivation could be improved.\nThe paper could benefit from describing personalized federated learning in more detail.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-5%