Fed-CO$_{2}$: Cooperation of Online and Offline Models for Severe Data Heterogeneity in Federated Learning
Authors: Zhongyi Cai
Ye Shi
Wei Huang
Jingya Wang
Venue: NeurIPS 2023 poster (2023)
Keywords: Federated Learning
Data Heterogeneity
Model Cooperation
Mutual Learning
Knowledge Transfer
TL;DR: We propose Fed-CO2, a universal FL framework that handles both label distribution skew and feature skew within a Cooperation mechanism between the Online and Offline models.

Abstract: Federated Learning (FL) has emerged as a promising distributed learning paradigm that enables multiple clients to learn a global model collaboratively without sharing their private data. However, the effectiveness of FL is highly dependent on the quality of the data that is being used for training. In particular, data heterogeneity issues, such as label distribution skew and feature skew, can significantly impact the performance of FL. Previous studies in FL have primarily focused on addressing label distribution skew data heterogeneity, while only a few recent works have made initial progress in tackling feature skew issues. Notably, these two forms of data heterogeneity have been studied separately and have not been well explored within a unified FL framework. To address this gap, we propose Fed-CO$_2$, a universal FL framework that handles both label distribution skew and feature skew within a Cooperation mechanism between the Online and Offline models. Specifically, the online model learns general knowledge that is shared among all clients, while the offline model is trained locally to learn the specialized knowledge of each individual client. To further enhance model cooperation in the presence of feature shifts, we design an intra-client knowledge transfer mechanism that reinforces mutual learning between the online and offline models, and an inter-client knowledge transfer mechanism to increase the models’ domain generalization ability. Extensive experiments show that our Fed-CO$_2$ outperforms a wide range of existing personalized federated learning algorithms in terms of handling label distribution skew and feature skew, both individually and collectively. The empirical results are supported by our convergence analyses in a simplified setting.

Fed-CO2: Cooperation of Online and Offline Models for Severe Data Heterogeneity in Federated Learning
Zhongyi Cai ShanghaiTechh University caizhy@shanghaiteec techh edu. on
Ye Shi ShanghaiTTech University shiye@shanghaitech edu, on
Wei Huang
Jingya Wang Shanghai Tech University wangjiingyyaShannhha. tecc. edu, cn
RIKEN Center for Advanced Intelligence Project wei. huang. vrōriken.Jj
Abstract
Federated Learning (FL) has emerged as promising distributed learning paradigm that enables multiple clients to learn global model collaboratively without sharing their private data. However, the effectiveness of FL is highly dependent on the quality of the data that is being used for training. In particular, data heterogeneity issues, such as label distribution skew and feature skew, can significantly impact the performance of FL. Previous studies in FL have primarily focused on addressing label distribution skew data heterogeneity, while only a few recent works have made initial progress tackling feature skew issues. Notably, these two forms of data heterogeneity have been studied separately and have not been well explored within a unified FL. framework. To address this gap, we propose Fed-CO2, universal FL. framework that handles both label distribution skew and feature skew within Cooperation mechanism between the Online and Offline models. Specifically, the online model learns general knowledge that is shared among all clients, while the offline model is trained locally to learn the specialized knowledge of each individual client. To further enhance model cooperation in the presence of feature shifts, we design an intra-client knowledge transfer mechanism that reinforces mutual learning between the online and offline models, and an inter-client knowl- edge transfer mechanism to increase the models' domain generalization ability. Extensive experiments show that our Fed-CO, outperforms wide range of existing personalized federated learning al algorithms in terms of handling label distribution skew and feature skew, both individually and collectively. The empirical results supported by our convergence analyses in a simplified setting.
1 Introduction
Federated Learning (FL) 11 2 is a distributed learning framework that involves collaboratively training global model with multiple clients while ensuring privacy protection. The pioneering work FedAvg 2 learns the global model by aggregating the local client models and obtains satisfactory performance when the client data are independently and identically distributed (IID). However, when the data are heterogeneous and non-lID among clients, performance with the global model can degrade substantially 3 4. Common data heterogeneity issues include label distribution skew and feature skew. Clients experiencing label distribution skew exhibit varying class distributions within the same domain, while clients encountering feature skew maintain consistent class distributions
Coorespponiing author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

but belong to different domains. As a promising solution to data heterogeneity issues, Personalized Federated Learning (PFL) has emerged, where personalized models are trained for individual clients. Most previous studies have focused on addressing label distribution skew [5 6 7 while only few recent works 8 have made initial progress tackling feature shift issues. So far, these two forms of data heterogeneity have been studied separately and have not been addressed within unified FL framework. Therefore, in this paper, we propose a universal FL framework that effectively handles data heterogeneity issues arising from label distribution skew, feature skew, or their combination.
Several algorithms personalized some parts of the model in FL with label distribution imbalance or feature shifts to retain and leverage some of the local offline information [10] [1 [3 g). However, in realistic FL scenarios, where extreme label distribution skew, severe feature skew, or even both are present, these algorithms fail to effectively harness local specialized knowledge for satisfactory adaptation. fact, some cases of extreme heterogeneity, models trained by those personalized algorithms may even perform worse than the locally- model, as the locally-trained model excels in capturing offline specialized knowledge. On the other hand, in FL. scenarios with milder heterogeneity, partially personalized models trained by PFL algorithms perform better due to their ability to access online general information from other clients.
So, given that the model with partially personalized parameters and the locally trained model each perform better in different cases, the question is: there a more effective approach fuse the online general knowledge and the offline specialized knowledge for better performance? From our investigations, the answer is yes. Accordingly, we propose a novel universal cooperation framework with these two models to address this challenge for both label distribution skew and feature skew data heterogeneity, referring to the model with partially personalized parameters as the online model, and the locally trained model as the offline model. Specifically, we personalize Batch Normalization layers in the online model and fuse the online and offline models' predictions as the final prediction. The prediction fusion between the online and offline models rectifies errors in their respective individual predictions and exhibits better performance.
In FL scenarios with feature skew, the general knowledge learned by the online model is domain- invariant, whereas the specialized knowledge learned by the offline model is Nomain-speciff Co- operation that occurs solely by fusing predictions is not sufficient as this process does not let the online and offline models communicate during the training phase. In turn, this impedes the transfer of comain-invarian and domain-sseccio cooperation between the models, we propose two novel knowledge transfer mechanisms one intra- client and one inter-client- which work at the model and client level, respectively. The intra-client knowledge transfer mechanism facilitates mutual learning between the online and offline models via knowledge distillation, which enables the online and offline models to benefit from both online nomain-invarian and offline domain- -speific knowledge. Conversely, the inter-client knowledge transfer mechanism enhances the model's domain generalization ability by introducing classifiers from the offline models of other clients to each local client.
knowledge between the models. Hence, to further encourage
Contribution. We propose Fed-CO2, a universal cooperative FL. framework for severe data hetero- geneity including both label distribution skew and feature skew. By simply fusing the online and offline models, Fed-CO, can handle severe label distribution skew effectively. To enhance model per- formance in the presence of severe feature skew, Fed-CO, involves an intra-client knowledge transfer mechanism that improves model cooperation and an inter-client knowledge transfer mechanism that increases client cooperation. We theoretically show that Fed-CO, has a faster convergence rate than FedBN [10 in a simplified setting. Besides, extensive experiments on five benchmark datasets demonstrate that our Fed-CO2 framework has a prominent edge over a range of state-of- -fat--of-the-att -htth-rrtttt algorithms where the data contain label distribution skew, feature skew, both!
2 Related Work
Federated Learning for Label Distribution Skew Data Heterogeneity. In real-world FL. scenarios, label distribution imbalance among clients is a common phenomenon that poses challenges to learning a single model that can effectively cater to all clients. A straightforward approach to personalizing the global model is fine-tuning it on local datasets 14
13
16 17 Other approaches attempt to
overcome distribution heterogeneity by exerting proximal regularization term on the global model,
'Our codes are publicly available at /ttttps://githuu.oo http://hhhwu.bome/hhccc/////eeee

---

as seen in examples like FedProx A PFedMe [18) MOON 77 and Ditto [19] Instead of adapting the single global model, learning personalized models is another main track. Here, the more explicit personalize some of the model parameters while leaving other parameters
methods [11 20 21 22
for aggregation. Similarly, FedMask [23] learns distinct binary masks for the last several layers
of the local models. Further, inspired by Hypernetworks (24), some methods 66] 13 25 apply
hypernetwork to produce client-specific parameters for participants. Beyond directly personalizing parts of the model parameters, some of the newer methods involve two-branch architecture where
the aim to calibrate the model's predictions 12 26 [27]. For example, FedRoD 12 uses a personal head to strengthen local learning, while PerFCL 26 splits the features in the local clients into shared parts and personalized parts. FedProe [27] uses prototypes as global knowledge to correct local training. However, unlike these methods, Fed-CO, maintains two separate personalized models for each client- - one that learns online general knowledge and the other that learns offline specialized knowledge with a focus on facilitating cooperation between the two models.
Additionally, knowledge distillation, which transfers dark knowledge between models, has played an important role in addressing label distribution skew 28 29
32 22]. Most of previous works in this arena rely heavily on a public dataset to transfer knowledge between the server and the local clients (29] (30 [T. Only limited number of studies have delved into the synergistic interplay between learning global knowledge and learning local knowledge, as well as strategies to optimize the mutual benefits of these two processes. In FML [28), the global model and the local model engage in mutual learning, but only the local model is used for personalized predictions. A very recent study, CD2-pFed [22]. employs cyclical distillation
30
31
regularization term in conjunction with the local training cross-entropy loss to mitigate the gaps between the learned representations from local weights and those from global weights. By contrast, our method utilizes knowledge distillation to facilitate the exchange of beneficial knowledge between the online and offline models before local training, as well as ensure effective and adequate intra-client knowledge transfer.
Federated Learning for Feature Skew Data Heterogeneity. While various methods have shown promising results on label distribution data heterogeneity, they often suffer from significant perfor- mance degradation when confronted with the more challenging feature shift data heterogeneity. To adapt to the feature shifts among different domains, FedBN [10] personalizes Batch Normalization (BN) layers for each participant. Beyond personalizing BN layers, PartialFed [8] explores the strategy of selecting personalized parameters based on the feature characteristics of different clients. To handle the feature skew, FedAP [9] learns the similarities between participants by analyzing their private parameters in the BN layers. In a recent work called TrFedDis (33], the concept of feature disentangling is used to capture both domain-invariant and domain-specifie knowledge. Our Fed-CO, deviates from such disentangling approaches and instead focuses on facilitating mutual learning between the online and offline models.
3 METHOD
In this section, we begin by formulating the research problem and subsequently introduce Fed-CO2. our novel universal FL framework. Finally, we will present the intra-client and inter-client knowledge transfer mechanisms designed to further enhance model cooperation in FL. with feature skew.
3.1 Problem Formulation
We aim to train a set of N models for each client to fit local data distribution. Each client has its own data distribution P. with its private dataset D; = {( v/² ) : k {1...., m, E {I..... N and m; is the sample number in the private dataset. Due to label distribution skew or feature skew, the data distribution Pz(x,, y) differs client-by-chle by-clieent basis. Our goal is to learn good model F.(-) with parameters 9; for each client according to information in all clients:
1 IL J=1 T(Fr(0z; x(), v!))
min (0,) N
where .) sample-wise loss function, F, is constructed by a feature extractor fi(+) with parameters 77 and a classifier C)(-) with parameters di. Accordingly, 02 = (m, Φ.).
3

---

(a) Cooperation of Online and Offline Models
Server
Model
Online Model
Online Modsl
Offline Mocel
Offine Model
Offine Model
Client
Client 2
Client N
Shard
Perscnalized
(b) Intra-Client Knowledge Transfer
(c) Inter-Cliene Knowledge Transfer
Offine Model
Classifery
gens
Online Model
Knowtedge Transfer
Onuire Model
LEB
Ofline Model
Omine Model....
LCE
Online Motiel lest roand
Growledge Tranefer
offf
Food Cless other ciinns
The Fed-CO, framework. Fig. aa) shows overview of the cooperation between the online and offline models in Fed-CO2. Figures [b) [[c] respectively illustrate the intra-client and inter-client knowledge transfer mechanisms FL with feature skew.
Figure 1:
3.2 Cooperation of Online and Offline Models
As shown in Fig. a propose Fed-CO2: novel universal collaboration framework between the online and offline models to adapt to local data distribution the presence of label distribution skew, feature skew, or both. Specifically, for each client i, we train a partially personalized model, referred to as the online model frnm, and a secally-trained model, referred to as the offline model Poff With two models in each client F: {F moft), ). cooperation is achieved through prediction fusion of the online and offline models. In this manner, our framework aims exploit both online general and offline specialized knowledge, enabling consistent high performance in FL. across various cases of local data heterogeneity.
For the online model, we personalize only few critical parameters to enable the online model to learn knowledge from other clients through model aggregation the server, while still retaining essential local knowledge. Prior work FedBN [10] has shown that Batch Normalization (BN) layers capture feature distribution in local clients. Inspired by such discovery, personalize the BN layers in the online model and split its parameters 80mh into {6on 6om) ). where gon denotes all BN layer parameters and 0g" denotes other parameters. Model aggregation done the server obtain the shared gon among all clients with the formula:
gon Σ
(2)
1=1 N 9.
For the offline model, all its parameters are personalized and not engage in server aggregation. As result, the offline model is able to learn local specialized knowledge without forgetting or being contaminated by other irrelevant information. With online and offline models, we employ a simple fusion technique and obtain the final prediction for model F;:
F, ; x) F (gon 6en x)) + Frfft (0off x).
(3)

---

It's worth noting that the ofline-offline model cooperation does not require extra communication costs compared to standard federated aggregation operations since offline models are not uploaded to the server.
When feature skew data heterogeneity is present, relying solely on prediction fusion for cooperation is insufficient. This approach fails the communication between the online and offline models during the training phase, impeding the transfer of online domain-invariant and offline domain-speeific knowledge. Therefore, to enhance collaboration among the models in FL with feature skew, we propose novel intra-client and inter-client knowledge transfer mechanisms. In the next two parts, we will introduce these mechanisms and discuss their role.
3.3 Intra-Client Knowledge Transfer
In our framework, each client i is equipped with two models: an online model for capturing gen- eral domain- knowledge and an offline model for capturing specialized domain-specine knowledge. Unlike previous algorithms that utilized a two-branch framework, aiming to make the knowledge learned by each branch as disjoint possible through methods such as Contrastive Learning [26] or Feature Disentangling (33), we introduce novel intra-client knowledge transfer mechanism to enhance the collaboration of the two models. This mechanism employs knowledge distillation to facilitate mutual learning between the two models during the training phase. However, transferring online general knowledge to the offline model could potentially conflict with the local training objective. The former emphasizes learning global information among {D, }D, } and avoiding over-fitting local data, whereas the latter focuses fitting local information in D;- As result, neither of the two goals can be fully accomplished. Furthermore, during local training, there is a risk of the online model losing general domain-invarian knowledge, which leads a reduction in the amount of valuable information accessible to the offline model.
To address these challenges, we divide the local training process into two distinct phases: a mutual learning phase and a local adaptation phase to achieve their respective goals. Concretely, in the mutual learning phase, personalized BNs from the last round and other parameters updated with the global aggregation model. Additionally, we retain copy of the initial offline model from the last round. As illustrated in Fig. bb). these duplicated models are frozen and utilized as teacher networks, with the copied online model serving teacher for the offline model, and the copied offline model guiding the online model. We denote the duplicated initial online model and copied initial offline model as Fr and Foff respectively, with frozen parameters 8:t and 0.ft. In training step s, given input sample Zj.- online and offline models conduct intra-client knowledge transfer and update their parameters in the mutual learning form:
create duplicate of the initial online model, which consists of its
000 1,8-1
V fom 1
Foll KT. (0:0 TK), F xx)))
(4)
0off goff a gull KL (0.h: Iw), F:
( (0cfh-1 xx)),
(5)
where KL .) denotes the Kullback-Lein mutual learning, the online model and the offline model engage in model-level collaboration and share the knowledge acquired in the previous round.
(KL) divergence and is the learning rate. Through
3.4 Inter-Client Knowledge Transfer
After the intra-client knowledge transfer in the first phase, both the online and offline models have gained beneficial knowledge from each other, and now it is opportune to let them adapt to the local data distributions. The pertinent question is whether other clients to bridge the domain gaps in the local training. Currently, the approach of acquiring knowledge from other clients is limited to aggregating shared parameters on the server. In comparison to various advanced approaches used in Domain Generalization (DG) research, parameter aggregation alone may struggle to learn effective commiin-ivvariann features. The primary challenge stems from the isolation of private data, as FL prohibits cross-client access to data from other domains. Without data from multiple domains, it is tough to directly apply techniques in DG.
can leverage additional knowledge from
Considering that the offline model in each client trained to capture local specialized domain-specific knowledge, propose to leverage these lightweight classifiers of the offline model to transfer domain
5

---

knowledge among clients, illustrated in Fig. Tc.. The classifiers of the offline model from other clients are introduced to each local client, enabling the feature extractors of its online and offline models to generate robust and well-generalized features that can be effectively recognized by these introduced classifiers. We share similar design intuitions with COPA 34 but here our inter-client knowledge transfer mechanism stresses more on making use of knowledge from other clients to benefit the personalized model for each client rather than serving unseen novel clients.
To be specific, in Fed-CO2, each client uploads its classifier C from its offline model to the server, along with its shared parameters (BNs are not included) from its online model. These uploaded heads construct classifier set for inter-client knowledge transfer: {C₃
fj=1 {C " with parameter set
10g ooff IN and are transmitted to each client in the next round. In communication round t, each client has its own personalized model F, = {FTm, Fr and the downloaded knowledge-transter
classifier set {C,y 1 Similar teacher models in the mutual learning phase, we freeze these sewwledge-transeee classifiers to keep their knowledge unchanged. With imput data Ty and its label yk our inter-client knowledge transfer loss function Lyen for models on client is formulated as:
Lgen (Ih, yh Σ jatt
LOE (c,
(0z off fe (Nias au)) M1
Combined with its classification loss to adapt to local private data, the total loss in local adaptation phase for each model is:
= LCB : yx) + μ Ligen(r yk),
where y is a penalized factor set as by default. After local training, we upload parameters the server and aggregate them with Eq. 2 Algorithm 1 in Appendix demonstrates the procedures of our Fed-CO, algorithm.
Theoretical Analysis
Here, we provide convergence analyses, which compare our Fed-CO,, and FedBN [0] with the neural tangent kernel (NTK) (35] theory, to illustrate the effectiveness of our algorithm. For simplification, only the cooperation of online and offline models is considered while the intra-client and inter-client knowledge transfer mechanisms are ignored.
4.1 Formulation
Before our analyses, we clarify the simplified setup and basic assumptions. Suppose we have N clients, each with M training examples, and aim to jointly train them for T communication rounds. In each round, each model is locally trained for one epoch. For simplification, we assume that all clients share identical two-layer neural network for regression task, with the only source of heterogeneity being the feature skew. Namely, each client i has its private dataset DD; = (xx), 1 with x) ga and y IR. For convenience, we adopt the same data distribution assumption as in 10
Assumption 4.1 For each client E (1...., N). the inputs x), are centered, meaning that T[x')] and they have a covariance matrix S; = [x{x S, is independent of the label y and varies for each client i. Not all S, matrices are identity matrices. For any index pair p and q (p & q), we have Xp # w. Xy for any non-zero w.
Let Ve Rd represent the parameters of the first layer, where & E [m] and m is the width of the hidden layer. We define ||v [|ss VvTSv Sv the induced vector norm for a positive definite matrix and a l2 vector norm ||v||2 The projections of x onto v and v are defined as x" : vv
x
x.
vIx The shift
Based on Assumption 4.1 for client i, the output of the first layer is normalized as T lls, parameter of BN omitted. Further, we denote the scaling parameter of BN 7 E R and the second layer parameter 1.

---

With these parameters, we proceed to train the online model:
N
F V, 7. 7, Vm
Tk
V
|s.
1(x
Di),
(8)
where o()) is the ReLU activation function and 7he model with all parameters personalized, we train:
is personalized BN parameters. For the offline
D-ffT X
Foft x V
1 Vm
off kiz
off The
R
1(x
(9)
veoff k.s
With online and offline models, form Fed-CO2 model F:
F(x; V, 7.
(x; Von 7on con) + poft (x Vofl o0t (oft)
(10)
In our analysis, we adopt one random strategy (36 to initialize parameters:
| (0)ll2
(0) - v NO,, a I); C.i = ck Unif(-1, 1}; Nk. 7k
(11)
where a controls the magnitude of va and val at initialization. The initialization of the BN
parameters 7h,t and off Thtt are independent of a. We use the MSE loss to train our model F:
L(F) NM
j=l (F(x))
w³) -
2
(12)
Fon + Foff (x!)
NM Σ 1=1
j=1
2
4.2 Convergence Analysis
We employ NTK (35] to analyze the trajectory of networks F learned by Fed-COp and networks Fon learned by FedBN (the online model in Fed-CO2). Existing studies [37 38 have validated that the convergence rate of finite-width over wer-parameteriieed networks is controlled by the least eigenvalue of the induced kernel in the training process. Following the discovery in 38), we can decompose the NTK into direction component (tt//a² and magnitude component G(t):
dF
V(t) a
dt (()((F(( y), A(I)
+ G(I)
(13)
The specific forms of (tt) and Gtt) are provided in the Appendix. Here, let Amin (H) denote the minimum eigenvalue of matrix H. is worth noting that both matrices (t) and G(t) are positive semi- cani-definite as they can be interpreted as covariance matrices. Based on this, we can deduce that Amin (t)) 2 max max(Amin (V())aa²), Amin )min(G(())), From the NTK theory, the value of Amin (!()) controls the convergence rate. Then, considering that a is the pre-defined parameter, for E (0. 1), convergence is dominated by (t). Let A(),, (t) and A A (t) denote the evolution dynamics of Fed-CO,, the online model and the offline model, respectively. Based on the work in [10) the auxiliary version of the Gram matrices y V and VoG for three models are strictly positive definite. Let the least cigenvalues Amin (Voe) := = Amim (Voc == of and (min((*)) := where n offff and are all positive values. To be specific, we define Gram matrices Voc and V in Definition
Definition 4.2 Given sample points [xpINM LNM we define the auxiliary Gram matrices V NNMNNM and Vee NNMMN
(ac)² x, v = Ev-N(0.0P1) (ac)² X, x 1(1p = ig), (Offline Model).
00pa
=
E Ev
9. (Online Model)
(14)
0 olp
v
(15)
With the prerequisite provided in Appendix, the convergence performance of the online model, the offline model, and our Fed-CO, can be analyzed by comparing Amin Amin(Ve VVoo, Amin (Voe Amin (V)). Our main thcoretical result is given in Theorem 4.3
Vor and

---

Theorem 4.3 For the Y-dominated convergence, the convergence rate of Fed-COF CO2 is faster than that of FedBN (the online model in Fed-CO2).
Proof sketch The core is to show Amin (Von) I Amin V
To prove the theorem, we first
show that ( (Von) Amin (Von )- Comparing Eq. 14 and 15 Vog takes the M M block
matrices on the diagonal of on Let V be the i-th M
x M block matrices on the diagonal
of V
According linear algebra, we have Amin (V:
Amin Von), for i [N.. With Z Amin (Von). Then, based on + Vof)). Therefore, we have
diag(Vi
VN). , it can be obtained that Amin off
off
Eq.
10
12
and
13 we can obtain A = Amin
(V
on
Amin
Aminn (Von) + 1 Amiia (Voo) )). Since we have proven that (Voo 2 Amin (Von).
the result Amin
Amiu on) can be achieved.
5 EXPERIMENTS
5.1 Experiment Settings
Dataset and Data Heterogeneity. We followed the footstep of prior research 33 7 13 to study the label distribution skew heterogeneity with image datasets: CIFARIO and CIFAR 100 [40]. Specifically, we considered two different label distributions among participants: 1) Pathological Distribution, each client is randomly assigned 2 classes per client in CIFARIO (10 classes per client in CIFAR100); 2) Dirichlet Distribution, each client gets its private data through partitioning of the datasets using a symmetric Dirichlet distribution with a default parameter = 0.3.
For feature skew heterogeneity, we conducted extensive experiments on three datasets: Digits, Office- Caltech10 [4]] and DomainNet 42) In this non-llD) data setting, each domain serves as a client, with each client having access to all the data from its respective domain. Digits is composed of 44] SynthDigits 45] MNIST-M 45 and MNIST [46]. Office-Caltech10 [41] owns data from four different domains including Amazon, DSLR, WebCam, and Caltech-225 -566. DomainNet 42) is a challenging dataset that comprises six distinct domains, namely Clipart, Infograph, Painting, Quickdraw, Real, and Sketch. To explore the more realistic FL scenarios with both label distribution skew and feature skew, we exert Dirichlet Distribution to datasets with feature shift. More details are supplemented Appendix.
different digit datasets with feature shift: SVHN 43), USPS
Compared Benchmarks. We compared our universal framework Fed-CO, with fundamental algo- rithms including SingleSet, FedAvg (2) and FedProx 4, together with some refined FL algorithms:
FedPer [1I. MOON 77) FedBN 10), and FedRoD 12 Here, SingleSet means each client trains their own model on their private data. We applied the linear version for FedRoD. For FL. with feature skew issues, further compared COPA (34), which was originally proposed to solve FDG and FDA problems. We now train the model on all domains in the dataset and evaluate its performance on each individual training client using the learned universal model.
5.2 Performance Evaluation
Model Evaluation on Feature Skew. Experiments are conducted datasets Digits, Office- Caltech10, and DomainNet. Here, we present the results of experiments on CalicechC,,,ch0,, and DomainNet in Table and Table Full experiments shown in Appendix. The performance of our Fed-CO, exhibits dominant edge over state-of-the-an algorithms in all experiments, where the average accuracy each domain increases by nearly 4%. The results confirm that our novel collaboration framework between online and offline models effectively addresses FL challenges associated with severe feature skew data heterogeneity.
Remarkably, when compared to algorithms FedBN 10 and COPA 34] which are explicitly designed to handle feature shift issues, method consistently outperforms them across all sub-datasets. This phenomenon provides compelling evidence that our cooperation mechanism adapts to local feature shifts more effectively by leveraging both general knowledge.
and specialized domain-specine
Model Evaluation on Label Distribution Skew. As shown in Table 3 our Fed-CO, outperforms variety of state-of-the-art PFL algorithms terms of average test accuracy across almost all cases in experiments with label distribution skew. Our state-of-the-an results across almost all cases
8

---

Table 1: Experiment results for FL with Feature Skew on Office-Caltech1...
Cattce-Caltech00
Methods
Amazon
Caltech
DSLR
WebCam
Avg
SingleSet
54,9#1.5
40.2+1.6
78.7+1.3
86.4+2.4
65.1+1.7
FedAvg
54.1#1.1
44.8#1.0
66.9+1.5
85.1+2.9
62.7+1.6
FedProx
54.2#2.5
44.5+0.5
65.0+3.6
8..4+1.7
62.0+2.1
FedPer [
49,0#1.2
37.1+2.4
57.743.7
79.7+2.1
56.0+1.1
MOON 7
57.3=0.7
44.4+0.5
76.2+2.5
83.1+1.1
65.2+0.5
FedRoD
12
60,4+2.3
45.320.9
73.742.5
83.7+2.3
65.8+1.4
COPA
34
51.9=2.5
46.720.8
65.642.0
85.0+1.3
62.340.9
FedBN 10
63,021.6
45.3
83.142.5
90.542.3
70.5+2.0
Fed-CO2
63.0#1.6
49.1
89,4+2.5
96.6+1.5
74.5+0.3
Table Experiment results for FL with Feature Skew on DomainNet.
DomainNet
Methods
Clipart
Inlograph
Painting
Quickdraw
Real
Skeich
Avg
SingleSet
41,0=0.9
23.8+1.2
36.2+2.7
73.1
48.5#1.9
34.0+1.1
42.8+1.5
FedAvg 2
48.8+1.9
24.9+0.7
36.++1.1
56.141.6
46.3+1.4
36.642.5
41.5+1.5
FedProx
48.9=0.8
24.9=1.0
36.6#1.8
54.4+3.1
47.8+0.8
36.942.1
41.6+1.6
FedPer [1|
40.420.8
25.7=0.6
37.320.6
62.5+1.2
47.4=0.5
32.8+0.8
41.0=0.3
MOON 7
$2.5.1.1
25.7+0.6
39.4+1.7
50.8+4.7
48.8+0.8
40.1+4.1
42.9+1.5
FedRoD 12
50.8+1.6
26.340.2
40.1+1.8
66.8+1.8
$1.5+1.1
39.142.0
45.7+0.7
COPA
51.1=10
24.7=1.2
36.820.8
51.81.6
47.
41.0144
42.6H0.4
FedBN 10
51.2+1.4
26.840.5
41.5+1.4
71.340.7
$4.840.8
42.141.3
48.0+1.0
Fed-CO,
55.02111
28.6211
44.320.6
75.120.6
62.420.8
45.7219
51.+0.22
prove that our method can more effectively excavate and utilize local information to overcome data heterogeneity caused by label distribution imbalance than previous methods. It is worth highlighting that our Fed-CO, outperforms both SingleSet and FedBN, regardless of which one performs better individually. This observation confirms that Fed-CO, effectively integrates the general knowledge obtained by the online model and the specialized knowledge acquired by the offline model, resulting in enhanced performance.
Table 3: Experiment results for FL with Label Distribution Skew on CIFARIO and CIFARI00, Experiments are conducted with two kinds of label distribution data heterogeneity: Pathological setting and Dirichlet setting.
CIFARTO
CIFART00
Methods
Pathological
Dirichlet
Pathological
Dirichlet
SingleSet
85.85+0.05
68.3840.06
49.54=0.05
21.3940.05
FedAvg 2
44.12+3.11 57.52+1.01
14.59+0.40
20.34+1.34
FedProx 4
57.38+1.08 56.46+0.66
21.32+0.71
19.40+1.76
FedPer
80.9940.71
74.2140.07
42.08+0.18
20.0610.26
MOON 7
48,4 18.43+3.18
54.49+1.87
17.8940.76
19.73+0.71
FedRoD
12
89.05=0.04
73.99+0.09
54.96+1.30
28.29+1.53
FedBN [T)
86.71+0.56
75.41+0.37
48.3740.56
28.7040
Fed-CO2
88.7940.25
77.45+0.30
58.5010.43
32.43+0.37
Model Evaluation on Label Distribution Skew and Feature Skew. Here, we evaluate our Fed-CO and benchmark algorithms in FL scenarios with label distribution skew and feature skew. We exhibit the experiment results on Digits in Table where we exerted a Dirichlet Distribution on each client with = 0.3. From the results, it is evident that, even in the presence of two types of data heterogeneity, our Fed-CO, outperforms various state-of-the-art algorithms with a significant margin in every sub-dataset. Therefore, we can conclude that under our universal cooperation framework, local clients not only make the most use of their local offline information but also benefit from general knowledge shared by other clients for a better adaptation to local data, resulting in ota the-art performance in FL. with both label distribution skew and feature skew. More experiments are provided in Appendix.

---

Table 4: Experiment results for FL with both Label Skew and Feature Skew on Digits.
Digits
Methods
MNIST
SVHN
USPS
SynthDigits
MNIST-M
Avg
SingleSct
83.75+7.58
74.63+0.31
97.14+0.06
87,95+0.46
80.55+0.26
84,80# .44
FedAvg 2
89.27=6.39
57.23+5.43
94.60+1.05
81.30+2.51
71.71+7.59
78.82+4.03
FedProx 4
87.0927.83
53.40+7.38
90.55+7.47
78.40+7.05
69.9826.55
75.88+6.90
FedPer [1]
96,92=0.02
72.86+0.11
97.09+0.12
87.82+0.02
83.69#0.07
87.68+0.02
MOON 7
96.58=0.07
74.10+0.22
96.19+0.10
88.15=0.04
85.05=0.14
88.01+0.07
FedRoD 12
96.65=0.06
77.05=0.16
96.8810.13
89.5920.06
86.1810.11
89.2740.05
COPA
14
96.82+0.08
78.3240.12
06.5440.15
89.3640.04
87.4620.11
89.7080.06
FedBN 10
92.68+3.45
70,2644.38
9140+8..2
83.1724.95
77.9843.84
83.1014.96
Fed-CO2
97.17=0.67
83.1620.23
98.12+0.13
93,0410.11
91.45+0.14
92.5940.15
5.3 Component Analysis
In the presence of feature skew, we add intra-client and inter-client knowledge transfer mechanisms to foster the collaboration between the online and offline models for better local adaptation. Here, we evaluate the efficacy of these two knowledge transfer mechanisms dataset through three additional experiments, as presented in Table 5 In these experiments, we selectively removed specific mechanisms from Fed-CO2 to investigate their individual contributions and functionalities.
the challenging DomainNet
According to the experimental results, we can observe that: 1) In the absence of intra-client and inter-client knowledge transfer mechanisms, Fed-CO, exhibits significant decline performance, resulting in performance level comparable to that of benchmark methods. Therefore, cooperation relying solely on prediction fusion proves to be inadequate for FL with feature skew. 2) The removal of either the intra-client the inter-client knowledge transfer mechanisms will also result in an average decline in performance. This phenomenon validates the effectiveness of our intra-client and inter-client knowledge transfer mechanisms, both of which contribute to enhancing the models' capability to adapt to local data distributions. 3) Compared with all three ablation experiments, our Fed-CO, consistently achieves the highest performance average and across the majority of sub- datasets. These results firmly demonstrate that our cooperation framework effectively leverages both lodel-level and client-level collaboration through the intra-client knowledge transfer and inter-client knowledge transfer mechanisms, respectively, to adapt to feature shifts resulting from domain gaps. Further analysis of these two cooperation mechanisms is supplemented in the Appendix.
Table 5: Ablation study of Intra-client and Inter-client knowledge transfer mechanisms for FL with Feature Skew Data Heterogeneity on DemainNet.
Meihods
Clipurt
Inlograph
Painting
Quickulraw
Reul
Sketch
Avg
Fed-CO, wio Intra and Inter Transfer
48.75a0.94
26.1922.05
42. 12.1021.05
72.8610.8 80
57.12a1.08
99.9660..9
47.8840.70
Fed-CO; w/o Intra Transfer
53.88::063
26. 18.071
42.9420.92
75.10#0.28
61. 1..040070
46.68:50.75
S1.12:0.1
Fed-CO, Transler
50.42a0.72
26.97a1.07
43.9420.69
74.1420.64
58.1420.85
49.2740.46
Ced
35.02a1.13
28.58=1.10
44.2750.62
75.0840622
62.3720.76
45.6751.95
318310.25
6 CONCLUSIONS
In this work, we proposed a new universal FL framework called Fed-CO, that capable of handling label distribution skew and feature skew even when both are present in the same data. The core of our approach is to foster cooperation between the online and offline models, leveraging the benefits of online general knowledge and offline specialized knowledge to effectively adapt to local data distribution. To improve model collaboration in FL with feature shifts, we designed two novel knowledge transfer mechanisms: one intra-client and the other inter-client. These mechanisms facilitate mutual learning between the online and offline models, concurrently enhancing the model's capacity generalize across different domains. Comparisons with wide range of tf-t-oo-thhe-ar art methods on five benchmark datasets consistently show that Fed-CO, yields superior performance in addressing both label distribution skew and feature skew challenges, both individually and in combination. Extending Fed-CO, to scenarios with noisy training data is under consideration in our future work.
10

---

Acknowledgement
This work was supported by NSFC (No.62303319), Shanghai Sailing Program (2IYF1429400. 22YF1428800), Shanghai Local College Capacity Building Program 130105531000, Shanghai Fron- tiers Science Center of Human-centeres Intelligent Perception and Macchne Collaboration (ShanghaiTech University), and Shanghai Engineering Research Center of Intelligent Vision and Imaging.
Artificial Intelligence (ShangHAD, MoE Key Laboratory of
References
[1] Qiang Yang, Yang Liu, Tianjian Chen, and Yongxin Tong. Federated machine learning: Concept and applications. ACM Transactions on Intelligent Systems and Technology (TIST), 10(2):1-19, 2019.
[2] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication -ffficitttinnefiiieentt learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273-1282. PMLR, 2017.
[3] Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Ar- jun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning, Foundations and Trends in Machine Learning, 14(1-2):1-210, 2021.
[4] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization heterogeneous networks. Proceedings of Machine learning and systems, 2:429-455 450, 2020.
[5] Othmane Marfoq, Giovanni Neglia, Richard Vidal, and Laetitia Kameni. Personalized federated learning through local memorization. In International Conference on Machine Learning, pages 15070-15092. PMLR, 2022.
[6] Aviv Shamsian, Aviv Navon, Ethan Fetaya, and Gal Chechik. Personalized federated learning using hypernetworks. In International Conference Machine Learning, pages 9489-9502. PMLR, 2021.
[7] Qinbin Li, Bingsheng He, and Dawn Song. Model-contrastive federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10713-10722, 2021.
[8] Benyuan Sun, Hongxing Huo, Yi Yang, and Bo Bai. Partialfed: federated learning via partial initialization. Advances in Neural Information Processing Systems, 0-22309--33221 2021.
personalized
[9] Wang Lu, Jindong Wang, Yiqiang Chen, Xin Qin, Renjun Xu, Dimitrios Dimitriadis, and Tao Qin. Personalized federated learning with adaptive batchnorm for healthcare. IEEE Transactions on Big Data, 2022.
[10] Xiaoxiao Li, Meirui Jiang, Xiaofei Zhang, Michael Kamp, and Qi Dou. Fedbn: Federated learning non-iid features via local batch normalization. International Conference Learning Representations, 2021.
[11] Manoj Ghuhan Arivazhagan, Vinay Aggarwal, Aaditya Kumar Singh, and Sunav Choudhary. Federated learning with personalization layers. arXiv preprint arXiv:1912.00 1912.00818, 2019.
[12] Hong-You You Chen and Wei-Lun Chao. On bridging generic and personalized federated learning for image classification. International Conference on Learning Representations, 2022.
[13] Hongxia Li, Zhongyi Cai, Jingya Wang, Jiangnan Tang, Weiping Ding, Chin-Teng Lin, and Ye Shi. FedTP: Federated Learning by Transformer Personalization. IEEE Transactions on Neural Networks and Learning Systems, pages 1-15, 2023.

---

[14] Kangkang Wang, Rajiv Mathews, Chloé Kiddon, Hubert Eichner, Françoise Beaufays, and Daniel Ramage. Federated evaluation of on-device personalization. arXiv preprint arXiv: 1910. 10252, 2019.
[15] Yishay Mansour, Mehryar Mohri, Jae Ro, and Ananda Theertha Suresh. Three approaches for personalization with applications to federated learning. arXiv preprini (rXi::2002./0619 2020.
[16] Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Personalized federated learning with theoretical guarantees: A model-agnostic meta-learning approach. Advances in Neural Information Processing Systems, 33:3557-3568, 2020.
[17] Mikhail Khodak, Maria-Florina F Balcan, and Ameet S Talwalkar. Adaptive gradient-based meta-lcarning methods. Advances in Neural Information Processing Systems, 32, 2019.
[18] Canh T Dinh, Nguyen Tran, and Josh Nguyen. Personalized federated learning with moreau envelopes. Advances in Neural Information Processing Systems, 33:21394-21405, 2020.
[19] Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. Ditto: Fair and robust federated learning through personalization. In International Conference on Machine Learning, pages 6357-6368. PMLR, 2021.
[20] Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai. Exploiting shared representations for personalized federated learning. In International Conference Machine Learning, pages 2089. PMLR, 2021.
[21] Paul Pu Liang, Terrance Liu, Liu Ziyin, Nicholas B Allen, Randy P Auerbach, David Brent, Ruslan Salakhutdinov, and Louis-PPilippe Morency. Think locally, act globally: Federated learning with local and global representations. arXiv preprint arXiv:2001.01523, 2020.
[22] Yiqing Shen, Yuyin Zhou, and Lequan Yu. Cd2-pfed: Cyclic Eestillation-uudder channel decoupling for model personalization in federated learning. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10041-10050, 2022.
[23] Ang Li, Jingwei Sun, Xiao Zeng, Mi Zhang, Hai Li, and Yiran Chen. Fedmask: Joint computa- tion and communication--ffiienn personalized federated learning via heterogeneous masking. In Proceedings of the 19th ACM Conference on Embedded Networked Sensor Systems, pages 42-55, 2021.
[24] David Ha, Andrew Dai, and Quoe V Le. Hypernetworss. arXiv preprint arXiv:1609.09106 2016.
[25] Xiaosong Ma, Jie Zhang, Song Guo, and Wenchao Xu. Layer-wised model aggregation for personalized federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10092-10101, 2022.
[26] Yupei Zhang, Yunan Xu, Shuangshuang Wei, Yifei Wang, Yuxin Li, and Xuequn Shang, Doubly contrastive representation learning for federated image recognition. Pattern Recognition, 139:109507, 2023.
[27] Xutong Mu, Yulong Shen, Ke Cheng, Xueli Geng, Jiaxuan Fu, Tao Zhang, and Zhiwei Zhang. Fedproc: Prototypical contrastive federated learning on non-iid data. Future Generation Computer Systems, 143:93-104, 2023.
[28] Tao Shen, Jie Zhang, Xinkang Jia, Fengda Zhang, Gang Huang, Pan Zhou, Kun Kuang, Fei Wu, and Chao Wu. Federated mutual learning. arXiv preprint arXiv:2006. 16765, 2020.
[29] Daliang Li and Junpu Wang. Fedmd: Heterogenous federated learning via model distillation. arXiv preprint arXiv: 191000358, 2019.
[30] Tao Lin, Lingjing Kong, Sebastian U Stich, and Martin Jaggi. Ensemble distillation for robust model fusion federated learning. Advances Neural Information Processing Systems, 33:2351-2363, 2020.
12

---

[31] Jie Zhang, Song Guo, Xiaosong Ma, Haozhao Wang. Wenchao Xu, and Feijie Wu. Parameterized knowledge transfer for personalized federated learning. Advances in Neural Information Processing Systems, 34:10092-10104, 2021.
[32] Huancheng Chen, Haris Vikalo, et al. The Best of Both Worlds: Accurate Global and Personal- Distillation. arXiv
ized Models through Federated Learning with Data-free preprint arXiv:2301.0890 2023.
[33] Meng Wang, Kai Yu, Chun-Mei Feng, Yiming Qian, Ke Zou, Lianyu Wang, Rick Siow Mong Goh, Xinxing Xu, Yong Liu, and Huazhu Fu. TrFedDis: Trusted Federated Disentangling Network for Non-IID Domain Feature. arXiv preprint 230.:::331112799 2023.
[34] Guile Wu and Shaogang Gong. Collaborative optimization and aggregation for decentralized do- main generalization and adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6484-6493, 2021.
[35] Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
[36] Tim Salimans and Durk P Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. Advances in neural information processing systems, 29, 2016.
[37] Simon S Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes wver-parameterized neural networks. arXiv preprint arXiv:1810.02004, 2018
[38] Yonatan Dukler, Quanquan Gu, and Guido Montúfar. Optimization theory for relu neural networks trained with normalization layers. In International conference on machine learning, PMLR, 2020.
pages
[39] Sashank Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečný, Sanjiv Kumar, and Brendan McMahan. Adaptive federated optimization. arXiv preprint arXiv:2003.00295, 2020.
[40] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[41] Boqing Gong, Yuan Shi, Fei Sha, and Kristen Grauman. Geodesic flow kernel for unsupervised domain adaptation. In 2012 IEEE conference computer vision and pattern recognition, pages 2066-2073. IEEE, 2012.
[42] Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1406-1415, 2019.
[43] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
[44] Jonathan J. Hull. A database for handwritten text recognition research. JEEE Transactions on pattern analysis and machine intelligence, 16(5):550-554, 1994.
[45] Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In International conference on machine learning, pages 1180-1189. PMLR, 2015.
[46] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, (6(11):2278-2)) 1998.
[47] Durmus Alp Emre Acar, Yue Zhao, Ramon Matas Navarro, Matthew Mattina, Paul N What- mough, and Venkatesh Saligrama. Federated learning based on dynamic regularization. arXiv preprint arXiv:2111.04266 2021.
[48] Idan Achituve, Aviv Shamsian, Aviv Navon, Gal Chechik, and Ethan Fetaya. Personalized federated learning with gaussian processes. Advances Neural Information Processing Systems, 34:8392-844 8406, 2021.
13

---

[49] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 60(6):84-90, 2017.
[50] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-perfiee begh-performacc deep learning library. Advances in neural information processing systems, 32, 2019.
14

---

A Dataset Setup Details
The details of non-IlI settings of mentioned datasets are presented in the section. Our two settings of label distribution skew, namely the Pathological setting and the Dirichlet setting are similar to preceding works [ 13). In the Pathological setting, where each client is randomly assigned two (ten) classes from CIFARIO CIFARI00), we first sampled qe UO0.4, 0.6) for the selected class
and then calculated the sample rate for this class с on client i with sr 2h In the Dirichlet Σ; ct
setting, we partitioned the datasets randomly utilizing a symmetrie Dirichlet distribution with default parameter a 0.3. Specifically, for CIFARIO0, we leveraged its coarse labels following previous works 5 39 In detail, a two-stage Pachinko allocation method was adopted to partition samples in CIFARTO0. For each client, this method first generates a Dirichlet distribution with default parameter = 0.3 over the coarse labels and then generates another Dirichlet distribution with parameter
= 10 over the corresponding fine labels. The local training batch size was set to 64. In both non-IID partitions, the classes and their distribution in each client's 's training and test sets remain consistent.
We adopted the setting used in FedBN I0 for feature skew heterogeneity, which involved adjusting the local training batch size to 32 maintain consistency. Additionally, we followed the implementa- tion of selecting only the top 10 classes based on data amount from DomainNet to create the dataset for our experiments. For experiments with label distribution skew and feature skew, we partitioned each sub-dataset into several parts with Dirichlet Distribution and selected one part as the new training set. Then we adjusted the label distribution in the test set to the same distribution the new training set. Specifically, based on the number of training images in each dataset, we divided each sub-dataset in Digits into three parts, each sub-dataset in Office-Calteche sub-dataset in DomainNet into five parts. Table 6 summarizes the datasets and the number of clients.
into two parts, and each
Table 6: Datasets Statistics.
Dataset CIFARIO
Label Distribution Skew
Feature Skew
Client Number
100
CIFAR100
Digits Office-Caltech10
4
DomainNet
B
Implementation Details
Network Modules. Similar to prior works 47 48
we adopted a ConvNet [46] with two convo-
lutional layers and three fully-connected layers for methods experiments conducted under label distribution skew setting. With regards to feature skew setting, experiments were performed with backbone similar to FedBN [I]. In other words, for the Digits datasct, we applied a ConvNet [46] and DomainNet dataset,
with four convolutional layers and three fc layers. For applied one Alexnet 49 as the default model backbone.
In our experimental evaluations of model performance under label distribution skew heterogeneity. we added an additional BN layer after each convolutional layer and fully connected (fc) layer, except for the last layer to algorithms: FedBN [0] and Fed-CO2. Meanwhile, in experiments pertaining to feature skew heterogeneity, as our default network incorporated BN layer after each convolutional and fc layer (except for the last layer), we omitted the BNs in our classifiers for methods other than FedBN [10] and Fed-CO2. All other Convnet network architectures used for the Digits dataset, as well as the Alexnet architecture employed for used in FedBN [TO)
and DomainNet, are identical to those
Concrete Implementation. For experiments evaluating model performance on label distribution skew, we set up 100 local clients and randomly sampled 5% of them to participate in each communication round. We trained every algorithm with 1500 communication rounds. Meanwhile, in experiments about feature skew non-iid heterogeneity, all clients participated each communication round and every algorithm was trained with 300 communication rounds. All training tasks were optimized with an SGD optimizer with the default learning rate 0.01. We used PyTorch (0) to implement our algorithms with an RTX 2080 Ti GPU.
15

---

For experiments on CIFARIO and CIFARIO0, we tested the model performance every 5 rounds during its last 200 communication rounds. The average and variance of test accuracy were reported. For all experiments with feature skew, we run 5 times and reported its mean and variance.
Evaluation Metries. We evaluated the learned personal models the private test data of each client. We examined the model's performance on each domain using the same method as FedBN [10] in FL with feature skew. Furthermore, for experiments label distribution skew, calculated the accuracy using the same protocol as FedTP 13
C Convergence
In this section, we first provide the preliminary and complete theorem proof. Then, we empirically demonstrate that our Fed-CO2 shows faster and more robust convergence performance than FedBN and other benchmark methods on FL issues with Feature Skew.
C.1 Preliminary
Following the V-dominated convergence rate for FedAvg (Theorem C.I in 38
derive the
convergence rate for the online model, the offline model, and Fed-CO, Corollary C.2
Theorem C.1 (V -dominated convergence for FedAvg (38) Suppose the network is initialized as in Eq. 77 with a < trained using gradient descent and Assumption 4.1 holds. Assuming the loss function used for training the neural network is the square loss, and the target values y satisfy lly 1l 0(1). If m - log N M/ / pâ), then with probability 6,
1. for iterations 0, 1, ...., the evolution matrix (t) satisfies Amia (A(t)) 2 2
2. training with gradient descent of step-size = 0 TV=
converges linearly as
llf(t) - -11 -
(1
17100 2a
[|f(0) -112.
Corollary C.2 Under the same assumption in Theorem C.1 with probability 1 8, for iterations = 0, 1, we have
1. The evolution matrix (f) satisfies Amin(A) (t)())
20 and training with gradi-
ent descent of step-size = O |
a²
converges linearly as [[Fo(() - y11 <
(1 (1-10) I/u
FOo (0) y
2n
2. The evolution matrix Afl (1) satisfies Amin (A (A (f)) a² ent descent of step-size O (IIv converges linearly as | (t) yl12 <
>
2
off
and training with gradi-
(1-18) Tiu 2n
Foft (0) -yll
3. The evolution matrix A(t) step-size 1
satisfies Amin (A(t)) 2 x and training with gradient descent of
a
converges linearly as [t)) y11
2
[F y
Therefore, the exponential factor of convergence for the online model, the offline model, and Fed-CO are controlled by the smallest eigenvalue of Von (t), voff (t), and (tt). respectively. The convergence performance of the online model, the offline model, and our Fed-CO, can analyzed by comparing Xuni (Vo)), Amin (Voc ) and Amiu (V
C.2 Theorem Proof
To prove Theorem 4.3 we will first prove SingleSet (the offline model in Fed-CO2) converges faster than FedBN (the online model in Fed-CO,). Gram matrix (t) and (t) of FedBN can be
16

---

obtained from work [10 as:
m
x x2 (e)a
xxprr (1)
Von pg
(ack)) )
Thiig (1) on (t) lIv (t)] [v]=(())ls))
on
x
(16)
{px(t)
Gpg (t)
IIvg (t)ls, [vg a (v2 T xp) a (v) xg) 1lip in}o (17)
ck
where 1p (t)
(velt) xxp00)
Similarly, the Gram matrix Voft (t) for method SingleSet with poff can be computed as:
Vodl (1) Σ
col off ofl
C.,, C.,y Th (1)
off
(1)
||]]] V2 sio
k=1
(18)
cffthe Khig (1))
(e)
[na(t) = in3
To compare the convergence rates of SingleSet and FedBN, compare the exponential factor in the convergence rates, which are (1
respectively. Then, considering that is the pre-defined parameter, it reduces to comparing jeth Ami (Von) and μ Xmin (V 18.
nu
and (1
nu 2
2a
Comparing Eq. 16
and
V takes the M M block matrices on the diagonal of Vos.
V
Vi
LN
V
1.2
VOG
2,N
2
0
V00
V off
where V is the i-th M x M block matrices the diagonal of Voe Therefore, have:
Ami (V?") 2 Amin (Vo), Vi E [N].
Since the eigenvalues of V a are exactly the union of eigenvalues of V we get:
Amin off
min (Amin (VP)), (E(N)
Amin )
Hence, (1
(1
nu and we get the first-stage conclusion that the convergence rate of 2a
2
method SingleSet is faster than that in FedBN.
Then we go to prove Fed-CO, is faster than FedBN, as well. With Eq. 10 12
and
13
we get:
dF
doo
dFoff dt
dt
1 A (t) + =A) (5 (F Foll Fo)(t)) y
(19)
2
1 on (t)
101()) (F(t) y).
Based on Eq.
19
we get the following relationships among three models:
A(t)
2 (A (n (t)++ + A A (t))
Von
oof (t)
Gom
+ Goff (1)
(20)
Von + Voff (t)
1
Gon (t) + off (t)
2
a²
2
17

---

To compare convergence rates of Fed-CO2 and FedBN, we need to compare their exponential factor (1 2n and (1 2n ) well. Similar to the former proof, the problem reduces to compare
== = Amin ( (Von Vorl)) ) with pmm Ami (Von). With previous proof result Amin (Vor) off 2
Amin Von), get
1
Amin
(Vor + Vodt
Amin (Von
+ Amin (Voft
2
on
2
Amin (Von)
Amiu (Von
Ami on
Therefore, we have (1
This theoretical conclusion guarantees a faster convergence our Fed-CO 202 than FedBN.
1/
(1
2n
2na
C.3 Experimental Performance
Here, we empirically analyze the convergence of test accuracy for various algorithms, well as the convergence hehavior of both the online and offline models. The results for sub-dataset WebCam and SVHN are illustrated in Fig. as examples. As Fig. []xxhibits, among various FL algorithms, our Fed- CO2 achieves the highest accuracy with significantly faster convergence speed and demonstrates much more robust behavior. Compared with its online and offline models, Fed-COp consistently surpasses both models, exhibiting smoother curve, which proves the success in fusing Nomain-invariin and domain-specific knowledge.
(b)
1.
WWNN
Notiog
HOON
Comman
Ohline Madr)
Calis Nold
246 se 1ia 220
123 aoo 200 260 280
Figure Convergence test accuracy on sub-datasets. Fig. a and Fig. (b) illustrate convergence behavior on WebCam from Office-Caltedd and SVHN from Digits, respectively. In each sub- sub-graph, we exhibit convergence performance comparison among different algorithms (Top) and comparison with online and offline models (Bottom). Fed-CO, exhibits faster and more robust convergence.
D Fed-CO, Algorithm
Here, we provide detailed description of the algorithm for our universal FL framework, Fed-CO2. The learning process for FL. with and without Feature Skew presented in Algorithm Algorithm respectively.
18

---

Algorithm 1: Fed-CO2 for FL. with Feature Skew
Input: T- number of communication rounds, N number clients.
Set parameters gg,0> p p.i.o. 6
for each communication rounds & {1...., T do
Communicate {Ø, \N
g to all clients;
1j=1:
for each client E [1, N] do
Initialize 00 = g, 0g p..t-1) 0g 0 f11-1. 000= 0 0001= 02:0
(0,, 0off < Intra-client Knowledge Transfer(OPth Ogt 0gn, 0g. [Eq. and Eq.
off (0r,, 0,,7) < Inter-elient Knowledge Transfer(O;F) ogg (Ø) } [Eq.
doff Communicate g,,,t to the server
end
10
Aggregate
gon n. ofl N Lim1 gon [Eq. giitt
11
Construct
[0; l==1 = { i,t li=1;
N
12 end
return Jon Oon and 0oft 9.1*
Algorithm 2: Fed-CO2 for Fl. without Feature Skew
Input: T- number of communication rounds, N- number of clients.
Set parameters gg, p...0- 8.ff.
for each communication rounds & (1,...TT do
Communicate gm g.t-1 to all clients;
for each client E [1, N] N] do
Initialize 0. = ( g. gon p.i.t-1,, o = i,1-13
(0en 0off) < Local Training (0 0 );
Communicate g. to the server
end
Aggregate foo
Σ
Oon g.e.t
[Eq.
N
end
11 return gon g,ts gom. and 8:0f
Table 7: Experiment results for FL with Feature Skew on Digits.
Digits
Methods
MNIST
SVHN
USPS
SynthDigits
MNIST-M
Avg
SingleSet
94,3820.07
65.25+1.07
95.1620.12
80.3140.38
77.77+0.47
82.00+0.40
FedAvg 2
95.87+0.20
62.86+1.49
95.56+0.27
82.27+0.44
76.85+0.54
82.70+0.60
FedProx
95.75+0.21
63.08+1.62
95.5810.31
82,3420.37
76.6420.55
82.70+0.60
FedPer 11
96.21+0.02
67.6140.04
96.53+0.02
83.88+0.02
81.89+0.03
85.2240.01
MOON 7
96.25+0.04
65.4820.49
95.0540.12
82,8920.15
80.5740.23
84.0540.16
FedRoD 2
96.0920.08
51.50...25
96.4210.08
85.5110.04
81.7920.09
86.2640.07
COPA
34
96.27+0.05
72.90+0.17
95. 95,9940.06
85,5240.04
83.0840.22
86.75+0.02
FedBN
96.57=0.13
71.04+0.31
96.97+0.32
13.19+0.42
78.33+0.66
85.20+0.40
Fed-CO,
97.6620.07
77.56+0.60
97.78+0.13
88.68+0.08
87.8420.12
89.9140.11
19

---

E More Experimental Results On Benchmark Datasets
Here, we supplement more experimental results on benchmark datasets. Firstly, we present the experimental results on Digits dataset for FL with feature skew in Table 7 As the table shows, our Fed-CO2 surpasses other various state-of-the-an aae-thh-rtt algorithms with substantial margin, which further proves the effectiveness of our framework in adapting local feature shifts.
Then, we show experimental results in FL with both label distribution skew and feature skew on DomainNet and Office-Caltech1 in Table 8 and Table We applied a Dirichlet Distribution with 2.0 to DomainNet and another Dirichlet Distribution with 1.0 to Office-Caltech 10. It can be observed that when facing two severe forms of data heterogeneity, algorithms demonstrated significantly lower performance on Office-Caltech10 and DomainNet compared to FL scenarios with feature skew alone. Even in such a challenging task, our Fed-CO2 still achieves the highest average accuracy and outperforms locally-traing -triined models. This result proves that Fed-CO, succeeds in making use of both general domain- invariant and specialized Nomain-specinf knowledge to adapt extreme local data heterogeneity.
Table 8: Experiment results for FL with both Label Skew and Feature Skew on DomainNet.
DomainNet Quickdraw
Methods
Clipart
Intograph
Painting
Real
Sketch
Avg
SingleSet
16.342.5
17.4+1.5
18.8+3.7
12.6+1.1
12.3+1.6
12.424.3
15.040.6
FedAvg
9.4+1.7
9.2+1.4
15.2+1.3
10.0+1.7
9.1+0.7
10.2+1.4
10.5+0.3
FedProx 4
9.6+1.4
10.8+1.6
12,2+1.2
10.1+0.7
10.3+1.5
11.0+2.7
10,741.1
FedPer 1
16.6+1.2
15.3+1.3
18.6#2.4
14.4+1.0
11.9+1.7
15.3+2.9
15.4+0.5
FedRoD [12
13.6+1.1
15.4+2.1
15.9
10.6#2.3
11.8+1.0
11.4+1.4
13.1+0.6
COPA 34
8.440.9
10.242.5
13.242.5
10.2+1.9
I1.11133
10.9E1.3
10.7#0.9
FedBN [0
11.0+1.1
12.911.3
13.0#2.0
11.840.6
10.4114
11.0#2,2
11.740.4
Fed-CO2
18.8+2.4
17.9+1.8
19.542.5
14.1+1.1
12.2+2.2
14.6#4.4
16.2+0.5
Table 9: Experiment results for FL with both Label Skew and Feature Skew Office-Caltech10.
Office-Caltechll
Methods
Amazon
Caltech
DSLR
WebCam
Avg
SingleSet
14.6+2.3
15.2+0.3
10.0#1.2
3.4+0.0
10.8+0.6
FedAvg 2
15.9+1.7
15.0+2.4
11.2+1.5
3.0+2.7
11.3+0.7
FedProx 4
14.0#1.3
13.0+1.0
9.4+2.0
5.1+2.4
10,4+0.8
FedPer [1]
16.5+0.6
16.140.3
11.9+1.2
1.7+0.0
11.5+0.4
FedRoD)
12
14.7+0.9
12.1+1.0
6.25+0.0
6.420.7
9.940.5
COPA
34
17.5+1.7
11.5+0.7
8.122.5
4.742.7
10.4&1.0
FedBN 10
19.5+2.5
12.6+1.9
11.9+2.3
2.4+0.8
11.620.4
Fed-CO, 17.5#0.9
16.6+1.2
12.5+0.0
3.7+1.3
12.6+0.5
Additionally, to explore broader real-life data heterogeneity issues with both feature skew and label distribution skew, conducted further experiments by exerting different levels of label distribution skew on the Digits dataset. Specifically, we partitioned each sub-dataset of Digits using the Dirichlet Distribution, with different values of E {0.3, 0.7, 1.0, 1.5, 2.0). A smaller a value indicates more severe label distribution imbalance. The experimental results are demonstrated in Fig. 3 It evident that every instance of label distribution imbalance. our Fed-CO, exhibits significant advantage over other benchmark methods in its sub- dataset as well as the overall average performance. Therefore, we can firmly conclude that our universal FL framework, Fed-COg, effectively addresses a wide range of real-life data heterogeneity issues.
F
Additional Experiments and Analyses
F.1
Effectiveness of Cooperation Framework
The experimental results of testing accuracy on five benchmark datasets have validated that our Fed-CO, effectively combines online general knowledge and offline specialized knowledge, resulting in improved adaptation to local data distribution in FL scenarios with label imbalance and feature
20

---

Test Acturay SynthDigits sub--datzeit
(b) Average Test Accutioy on Dig dataset
0.95
0.95
0.90
0.50
Tes: Accuroc
0.85
Singleset Fedffgg
SingleSet
Fedlin
0.75
ledrer
0.75
redrer
RodBM
0.70
COPA
0.70
CORA
Red C02
Fed C02
Drichiet Distibution
Dirchlet Distribution
Figure 3: The test accuracy of our Fed-CO, and benchmark algorithms in Federated Learning with Feature Skew and varying levels of Label Distribution Skew. Fig. a depicts the model performance on the SynthDigits sub-dataset, while Fig. b) demonstrates the average model performance across all sub-datasets in Digits.
shifts. To further confirm the effectiveness of the cooperation between the online and offline models, we conducted additional experiments for FL with label imbalance and feature shifts in the following parts.
Online and Offline Models in FL with Label Distribution Skew. For FL with label imbalance, we split the CIFARI0 dataset for 20 clients with various types and degrees of label distribution skew. In the Pathological case, each client i was randomly assigned two classes following the strategies mentioned in Appendix A In the Dirichlet case, we partitioned the dataset randomly utilizing the Dirichlet distribution with different parameters a E {0.3, 0.7, 1.0) to evaluate model performances under various extents of label distribution imbalance. In these experiments, we compared the model trained by our CO2 with its online and offline models.
The experimental results have been exhibited in Fig. is evident that the online model outperforms the offline model when the label distribution imbalance is moderate, while the offline model excels when the imbalance is severe. Across all extensive experiments, Fed-CO, consistently surpasses both the standalone online model and the standalone offline model in terms of test accuracy statistics, exhibiting higher average and median values. At the client level, Fed-CO, improves prediction accuracy for the majority of local clients. Although Fed-CO, may not surpass the superior model between the online and offline models on limited number of clients, it consistently outperforms the weaker model and achieves performance levels approaching that of the superior model. Therefore, we can conclude that our collaborative FL framework, Fed-CO2, which combines the strengths of online and offline models, exceptionally effective. successfully integrates online general knowledge and offline specialized knowledge, resulting in an improved adaptation to local data distribution under various cases.
Online and Offline Models in FL. with Feature Skew. For FL with feature skew, we conducted experiments that evaluate the performance of the online and offline models on DomainNet. Based on the experimental results shown in Table IO] we have the following observations: First, the online model outperforms the offline model on most clients, except for the Quickdraw client. This phenomenon validates our hypothesis that when the feature shift is severe, the model aggregation will lead the loss of important local offline information, resulting in the model's failure to adapt to such significant data heterogeneity. Second, prediction fusion is even inferior to the online and offline models for some clients. This result reflects that prediction fusion is not sufficient to fuse online general knowledge and offline specialized knowledge in FL with severe data heterogeneity issues. Therefore, for FL. with feature skew, intra-client and inter-client knowledge transfer mechanisms are required to boost model and client cooperation under novel framework.
21

---

(o)Pathological cal
Ditiibttion
Karieties
100
8:90
8.80-
1.75
Ds
(b)Dirichlet 0.3
Dattrbation
Accurncy Kastetts
Perforrance
2700
0.4
250
Clent IDs
Ofhice Mold
Diricchte Cr-0.7
Dotrbbtton
Aecurnc Santetos
0.5
0.1
0.1
Dr
Client IDs
[d)iirichlet a-1.0
Loved Distauatio
Accuracy Suttakis
800
500
042
Clant
Figure 4: Comparison results among the online model, the offline model, and our Fed-CO, with different kinds of label distribution skew. In each subgraph, illustrate label distribution among 20 clients (Left), the classification test accuracy statistics (Middle), where the full line represents the median accuracy and the dotted line represents the average accuracy among the 20 clients, and test accuracy in each client (Right).
Table 10: Performance of Online and Offline Models in FL with Feature Skew on DomainNet
Methods
Clipart
Inlograph
Painting
Quickdraw
Real
Skeuch
Avg
Online Modal
51.211.4
26.82033
41.5g1.4
71.340.7
54.3008
42.121.
48.011.0
Offine Model
41.040.9
23.8+1.2
36.2+2.7
73.140.9
48.5+1.9
34.0+1.1
42.8+..5
Fed. CO2 (no intra and inter knowledge transfer)
48.740.9
26.5+2.0
42.1a1.0
72.910.8
57.1=1.1
40000.
47.940.7
Fed-COp
35.0El.1
28.8111
44.320.6
75.1446
62.450.
45.7219
31.80.2
22

---

F.2 Knowledge Transfer Mechanisms in Feature Skew
As mentioned the main paper, intra-client and inter-client knowledge transfer mechanisms foster model-level and client-level collaboration to better make use of general domain-invariant and special- ized domain-specific knowledge for a better adaptation to local data distribution. In this section, we did further ablation studies the challenging DomainNet dataset to investigate two mechanisms from a line-grained angle.
A Ablation Study in Intra-client Knowledge Transfer Mechanism. In this section, we investigated the impact of the intra-client knowledge transfer mechanism on the online and offline models by excluding the inter-client knowledge transfer mechanism from Fed-CO2- Specifically, we performed separate intra-client knowledge transfers: from the offline model to the online model and from the online model to the offline model. The results are demonstrated in Table [ and we can discover that knowledge transfer is beneficial to both online and offline models. By transferring specialized domain-specific knowledge to the online model, we can observe an improvement in the performance of the Quickdraw client, which has a more extreme non-IID data distribution and relies more on local offline information. Conversely, when transferring general offline model, we could notice enhanced performance in the remaining clients that present milder data heterogeneity and require more global information. Hence, by incorporating the full intra-client knowledge transfer mechanism, our Fed-COp effectively utilizes both online domain- invviiiinn and offline domain-specift knowledge, resulting in improved average test accuracy.
knowledge to the
Table 11: Ablation Study of Intra-client Knowledge Transfer mechanism on DomainNet
Method
Clipart
Infograph
Painting
Quickdraw
Real
Sketch
Avg
No intra transter
48. 78.9
26.542.0
42,1+1.0
72.920.8
57.111.1
40.020.8
47.920.7
Intra transfer online
18.1
26.2+1.3
42.1+1.2
74.020.5
57.0+1.0
40.4+1.0
48.0+0.3
Intra transfer offline
50.9+0.7
26.6+0.7
42.1+1.1
72.8+0.5
59.1+0.3
41.3+18
48.8
Full intra transfer
50.420.7
27.0#1.1
43.9+0.7
74.1+0.6
58.1
42.0+0.8
49.3+0.5
A Ablation Study in Inter-elient Knowledge Transfer Mechanism. In this part, we removed the intra-client knowledge transfer mechanism from Fed-CO, and did experiments to investigate the impact of inter-client knowledge transfer mechanism on online and offline models. The experiments included exerting inter-client knowledge transfer mechanism on the online model, the offline model, and both models. Table 12 presents the experimental results. The results demonstrate that knowledge from other domains bencñits both the online and offline models, with the online model showing more remarkable improvement. This observation aligns consistently with the purpose of the online model, which focuses on acquiring general domain-invariie domains. Furthermore, the application of the inter-client knowledge transfer mechanism to both the online and offline models yields optimal performance, which means simultaneously enhancing domain generalization ability for online and offline models can exploit domain-invarian knowledge more effectively. In summary, our inter-client knowledge transfer mechanism enables each local client to acquire and exploit more Nomain-invarian knowledge resulting in a better adaptation to local data distribution.
knowledge that can be applied across various
Table 12: Ablation Study of Inter-client Knowledge Transfer Mechanism on DomainNet
Methods
Clipart
Infograph
Painting
Quickdraw
Real
Sketch
Avg
No inter transfer
48.720.9
26.542.0
42.1=1.0
72.920.8
57.1
40.020.8
47.920.7
Online model with inter transfer
$1.3+0.77
27.1+1.0
44.2+1.3
74.1
61.4x0.7
43.5x1.5
50.320.5
Offline model with inter transfer
11.8+1.3
26.3+1.8
43.3+1.3
74.2=1.0
59.0:2.2
43.121.6
49.620.6
Full inter transler
53.9+0.6
26.240.7
42.9=0.9
75.1+0.3
61.9+0.7
46.7+0.7
51.1=0.2
F.3
Data Heterogeneity of Noise
Noise is another common factor causing data heterogeneity issues in real FL scenarios, where each client has the same label distribution but is subject to various degrees of noise. Therefore, we implemented additional experiments on this new scenario to make the initial attempt. detail, we divided the dataset CIFAR10 10 into 10 clients in a way to make them independently and identically distributed. Then we applied the increasing level of random Gaussian noise with mean 14; = 0 and standard deviation o; - i, where i E {0, ............ ..N 1}. In this series of initial experiments,
23

---

we let the client number and the sample rate be 0.5 with aM E {0, 5, 10, 15, 20). The backbone of each method the same as that in other experiments.
As presented in Fig. 5 the experimental results reveal that our Fed-CO, consistently exhibits better and more robust performance across nearly all experiments, particularly in scenarios with severe noise. These experiments will also serve as source of inspiration for our future work.
Accuracy
Levels
redwng
MDON
FedProx
0.55
tedbi
CD2
2.50
0.45
0.40
325
020
013
010
Figure 5: Test accuracy on dataset CIFARIO for Fed-CO, and some benchmark methods with different degrees of data heterogeneity caused by noise.
F.4 Effects of Data Number and Client Number
The sample size of the training set and the client number are significant factors. Therefore, in this section, we conducted series of experiments explore their influence.
To investigate the effect of the training sample number, we utilized dataset Digits and set 10% of its training data as the original training dataset. Then, we selected certain ratio 7 of the original training dataset as a new training dataset for each client, where 7 E {0.2, 0.4, 0.6, 1.0, 2.0}. The results shown in Fig. [6] demonsttraee that our Fed-CO, have edge over various benchmark methods in almost every case, especially when the number of training data very limited.
(e) Test Accuracy on MN ST-M sub-detaset
(b) Average Test Accuracy on Digits diaaas::
0.95
0.95
0.50
0.50
0.85
That Accuney
0.75
0.80
0.70
Singleeet
0.75
Singleset
0.65
Redkig
Rediig
MOON
MCON
0.5
COPA
0.5
CONA
Fed-C02
Red-C02
0.0
maining sample
Training sample
Figure 6: The effect of the training number ratio on Fed-CO, and benchmark algorithms for Federated Learning with Feature Skew. Fig. [] a) depicts the model performance on the MNIST-M sub-dataset, while Fig. (bb) demonstrates the average model performance across all sub-datasets in Digits.
As for the effect of client number, we employed dataset CIFAR10 with Dirichlet setting (a: 0.3) to conduct series of experiments on benchmark methods with client number N E (50, 100, 150, 300, 500]. The experimental results shown in Table 13 demonstrate that our Fed- CO2 consistently outperforms a bunch of SOTA methods in every case even if the client number is substantial. The results in Table T3 and Fig. 6 prove that our Fed-CO, owns higher scalability than benchmark methods.
24

---

Table 13: The influence of client numbers on CIFARIO
#client
50
100
150
300
500
SingleSet
71.28+0.08
68.6920.07
70.05:40.06
64.62+0.04
60.3140.07
FedAvg 2
41.2043.13
52.85+2.30
33.34=11.15
42.63+3.85
34.31+5.68
FedProx
45.35=t6.4
36.00+4.42
34.53+6.59
37.44+6.23
34.58+5.18
MOON
7
40.25+3.31
52.38+2.16
34.9349.74
49.58+3.93
37.35+5.80
FedRoD
12
65.18+1.14
66.08+0.92
67.00+1.43
65.46+1.50
64.68+4.30
FedBN
10
74.40+0.61
75.43+0.39
78.38+0.44
76.06+0.46
69.24+1.21
Fed-CO2
77.14+0.37
77.21
80.06:0.25
78.53+0.31
75.59+0.50
F.5 Personalized Parts in the Online model
In order enhance the adaptation to local data distribution, necessary personalize certain critical parameters in the online model to preserve the essential offline specialized knowledge. In this section, we investigated and analyzed which specific parts of online model should be personalized for optimal performance in FL with feature skew. The most common strategy is to personalize Batch Normalization layers or the Classifier Head. Therefore in our Fed-CO, approach, we conducted three experiments with different personalized components in the online model to compare and determine the optimal strategy. The results for three datasets, namely DomainNet, Office-Calle Afiie-CaltecchO, and Digits, are illustrated in Fig. 7 It is evident from the observations that personalizing the Classification Head has a negative impact on almost every sub-dataset, resulting lower classification accuracy. Based on the results, we can conclude that personalizing Batch Normalization layers in our online model is the best strategy to adapt to feature shifts.
(c) Paterrann Concartoe Of Deuts
10"
Porfermenc Comcer son Donmanttet
1.
Porformance Comcerton on De CaRcettD
1.8"
Nesuenalrr
Hoad
Perseraber
Figure 7: The test accuracy for Fed-CO, with different personalized parts in the online model on Net,inNet, and Digits.
25


Reviews:
[{"review_id": "uAqsRT34pR", "reviewer": "Reviewer_xxVq", "summary": "This paper introduces an algorithm to address both label and feature heterogeneity in federated learning. The algorithm entails that each client learns two models: one that has personalized batch normalization parameters and otherwise global parameters, and another that is fully personalized. To learn the parameters, a two-stage knowledge distillation procedure is proposed, where in the first stage knowledge is distilled between the two models on each client, then in the second stage knowledge is distilled across clients. Experiments study the performance of the proposed model against a variety of baselines on image classification in the presence of label heterogeneity, feature heterogeneity, and both label and feature heterogeneity.", "questions": "n/a", "limitations": "See Weaknesses", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The experimental results show convincing improvement over a variety of relevant baselines in multiple settings with different types of heterogeneity. The procedure is rigorous. \n\n2. The related works are well-discussed.\n\n3. The paper addresses a relevant topic.", "weaknesses": "1. The writing can be improved. There are numerous sentences with improper grammar/semantics, e.g. “Federated learning for Data Heterogeneity of Feature Skew”.  The model parameter notation is not consistent as both \\theta= \\{\\phi, \\eta\\} and \\theta = \\{\\phi, \\xi\\} are used. Label and feature heterogeneity/skew are never defined. The motivation for personalizing specifically the batch normalization parameters is not clear whatsoever, and seems to be simply because it worked well in [10].\n\n2. Label and feature heterogeneity/skew are never defined.\n\n3. The proposed method entails learning many parameters per client. This is likely to degrade performance in settings with few samples per client, but the effect of the number of samples on algorithm performance is not investigated.\n\n4. The experiments are limited to image classification on relatively simple datasets.\n\n5. The idea to encourage retaining global information locally by adding a regularizer that penalizes loss on local samples of local models with concatenated heads of all other clients is interesting, but introduces additional communication and privacy concerns (due to all clients sharing each of their heads with each other) that are not addressed.\n\n-----------\n\nPost-rebuttal: I have raised my overall score from 4 to 5, and Contribution score from 2 to 3, please see comment below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">**Q1**: The writing can be improved. There are numerous sentences with improper grammar/semantics, e.g. “Federated learning for Data Heterogeneity of Feature Skew”.\n\n**R1**: We will improve improper grammar/semantics in the final version.\n\n\n\n>**Q2**: The model parameter notation is not consistent as both $\\theta= $\\{ $\\phi, \\eta$ \\} and $\\theta = $\\{ $\\phi, \\xi$ \\} are used.\n\n**R2**: We will rectify the confusing parameter notation in our final version. The parameter notation you are confused is actually two types of parameter division. The former parameters $\\theta=$\\{ $\\phi, \\eta$ \\} represent parameters of the feature extractor and the classifier (defined in section 3.1). The latter parameters $\\theta=$\\{$ \\psi_i, \\xi$ \\} represents the personalized parameters and the global parameters of the model (defined in section 3.2). In the final version, we will rectify this latter notation into $\\theta=$\\{ $\\theta_{p,i}, \\theta_g$ \\} to avoid misunderstanding.\n\n\n>**Q3**: Label and feature heterogeneity/skew are never defined.\n\n**R3**: Actually, we have briefly explained label distribution skew and feature skew in the Related Work (the former in line 77 and the latter in line 112). Label distribution skew is \"label distribution imbalance among clients\" and feature skew is \"feature shift among clients (domains)\". \n\nSpecifically, for FL with label distribution skew, data in each client has different label distributions. For FL with feature skew, data in each client comes from different domains. In the final version, we will add more illustrations to explain label distribution skew and feature skew in the Introduction.\n\n\n>**Q4**: The motivation for personalizing specifically the batch normalization parameters is not clear whatsoever, and seems to be simply because it worked well in [10].\n\n**R4**: We respectively disagree with this comment. [10] is a classic method in addressing feature skew issues and provides a significant insight that BN helps harmonizing local feature distributions. As we mentioned in line 143, we also need to capture feature distribution in local clients. Therefore, we adopted BN personalization in our online model. Moreover, we did an ablation study in Appendix F.3, where we evaluated three personalization strategies including personalizing BNs, personalizing classification head, and personalizing BNs and classification head. The experimental results persuasively convinced us to personalize BNs. \n\n\n>**Q5**: The effect of the number of samples on algorithm performance is not investigated.\n\n**R5**: Here, we have supplemented experiments exploring the effect of the number of samples. For convenience, we utilize the Digits dataset and set 10\\% of its training data as the original training dataset. Then, we select a certain ratio $\\gamma$ of the original training dataset as a new training dataset for each client, where $\\gamma \\in $ \\{0.2, 0.4, 0.6, 1.0, 2.0\\}. The results are shown in Fig. 1 (in our new supplementary) and demonstrate that our Fed-CO$_2$ have an edge over various benchmark methods in almost every case, especially when the number of training data is very limited. We will add this experiment in our final version.\n\n\n>**Q6**: The experiments are limited to image classification on relatively simple datasets.\n\n**R6**: We respectively hold a different view from this comment. Actually, we utilized five distinct datasets: CIFAR-10, CIFAR-100, Digits, Office-Caltech10, and DomainNet under several non-I.I.D. settings including FL with label distribution skew, FL with feature skew, FL with label distribution skew and feature skew. Notably, most existing methods utilized even fewer datasets under more limited non-I.I.D. setting. \n\nAs revealed in our paper (in line 32), prior research either concentrated on addressing label distribution skew [2, 4, 7, 11, 12, 13] with datasets CIFAR-10 and CIFAR-100, or alternatively, attempted to mitigate feature skew [8, 10, 32] with datasets Digits, Office-Caltech10, and DomainNet. Some other algorithms researched the effect of noise on FL including pFedGP [50] and [13] with datasets CIFAR-10 and CIFAR-100.\n\nHere are more details: \n1. Experiment results for FL with label distribution skew with datasets CIFAR-10 and CIFAR-100 are shown in Table 3 (section 4.2) and Figure 4 (Appendix F.1); \n1. Experiment results for FL with feature skew with datasets Digits, Office-Caltech10, and DomainNet are shown in Table 1, 2 (section 4.2) and Table 7 (Appendix E); \n3. Experiment results for FL with label distribution skew and feature skew with datasets Digits, Office-Caltech10, and DomainNet are shown in Table 4 (section 4.2), Table 8, 9 (Appendix E);\n4. We supplemented experiment results of our Fed-CO$_2$ for FL with noise on CIFAR-10 in respondence to Reviewer 5E9k Q7. \n\n[50] Achituve, Idan, et al. \"Personalized federated learning with gaussian processes.\" Advances in Neural Information Processing Systems 34 (2021): 8392-8406.\n\n>**Q7**: The idea ... introduces additional communication and privacy concerns (due to all clients sharing each of their heads with each other) that are not addressed. \n\n**R7**: We partially disagree with this opinion. (1) Privacy concerns. In the context of traditional FedAvg [2], communicating clients' models to the server remains necessary for model parameter aggregation, meaning that model communication will not cause severe privacy concerns.\n(2) Additional communication. Our Fed-CO$_2$ merely communicates parts (the classifier) of the offline model. Thus, the additional communication overhead is relatively modest.\n\nMoreover, the inter-client knowledge transfer, which causes these two concerns, is only employed in extreme data heterogeneity issues. As our ablation study shows in Table 5, our method still achieves SOTA performance without Inter-Client Knowledge Transfer. In most non-I.I.D. cases, our framework already works."}, {"review_id": "Trt2flpYgl", "reviewer": "Reviewer_5E9k", "summary": "The paper presents Fed-CO2, a novel framework for Federated Learning (FL), a distributed learning paradigm that allows multiple clients to collectively learn a global model without sharing their private data. FL's effectiveness depends heavily on the quality of the data used for training. Data heterogeneity issues, like label distribution skew and feature skew, can significantly impact FL's performance. Traditionally, most studies have focused on dealing with label distribution skew, while a few recent ones have started addressing feature skew. These forms of data heterogeneity have been studied separately and have not been integrated within a unified FL framework. \n\nFed-CO2 aims to address this gap by developing a framework that handles both label distribution skew and feature skew. The framework utilizes a cooperation mechanism between the Online and Offline models: the online model learns general knowledge shared among all clients, and the offline model is trained locally to learn each individual client's specialized knowledge.  To further enhance model cooperation in the presence of feature shifts, the authors design an intra-client knowledge transfer mechanism to reinforce mutual learning between the online and offline models. Additionally, they introduce an inter-client knowledge transfer mechanism to enhance the models' domain generalization ability. Through extensive experiments, Fed-CO2 outperforms a wide range of existing personalized federated learning algorithms, handling label distribution skew and feature skew, both individually and collectively.", "questions": "- What are the computational costs associated with the proposed intra-client and inter-client knowledge transfer mechanisms?\n- How does Fed-CO2 handle potential security and privacy concerns associated with inter-client knowledge transfer?\n- How does Fed-CO2 handle scenarios where the label distribution skew and feature skew are extreme?\n- Can the framework handle other types of data heterogeneity beyond label distribution skew and feature skew?", "limitations": "See weaknesses and questions", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The cooperation mechanism between the online and offline models is a novel and effective way to deal with data heterogeneity issues. This mechanism allows for general learning shared among all clients, as well as specialized learning unique to each client.\n- The Fed-CO2 framework represents a comprehensive approach to addressing data heterogeneity in Federated Learning. It doesn't just focus on label distribution skew or feature skew, but rather addresses both these challenges within a unified framework.\n- The extensive experiments show that the Fed-CO2 model performs better than a wide range of current personalized federated learning algorithms. This indicates the effectiveness and potential of the Fed-CO2 model.", "weaknesses": "- The model may be complex to implement, especially in environments where the number of clients is very large, due to the dual model structure (online and offline) and the knowledge transfer mechanisms.\n- Despite being designed to handle heterogeneity, the performance of the framework is still reliant on the quality of the data used for training, which may not always be guaranteed, especially in a federated setting.\n- The paper does not clearly articulate how the proposed method scales with increased data, feature dimensions, or numbers of clients. Understanding the scalability of the method is crucial in real-world applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">**Q1**: The model may be complex to implement, especially in environments where the number of clients is very large, due to the dual model structure (online and offline) and the knowledge transfer mechanisms.\n\n**R1**: We partially disagree with this opinion. Our method will be easy to implement if we remove the inter-client knowledge transfer mechanism. As mentioned in response to Reviewer xxVq Q7, Fed-CO$_2$ is able to address numerous non-I.I.D. challenges without the need for the inter-client knowledge transfer mechanism, and in such cases, no additional communication overhead arises as the offline model remains consistently local. \n\n>**Q2**: Despite being designed to handle heterogeneity, the performance of the framework is still reliant on the quality of the data used for training, which may not always be guaranteed, especially in a federated setting.\n\n**R2**: We respectively disagree with this comment. We believe every Deep Learning method (not limited to federated learning) relies on the quality of the training data. While our method does not rely much on the training data quality as we study very challenging data heterogeneity cases. \n\n>**Q3**: The paper does not clearly articulate how the proposed method scales with increased data, feature dimensions, or numbers of clients. Understanding the scalability of the method is crucial in real-world applications\n\n**R3**: We appreciate your advice for supplementing experiments discussing the influence of increased data, feature dimensions, and number of clients. Considering the influence of feature dimensions is rarely discussed in former FL algorithms, we have only supplemented experiments concerning increased data and client number. \n\nAs for the experiments of increased data, please refer to the response to Reviewer xxVq Q5. Here, we provide the experiment results of our method with the increased client number. To be specific, we employ dataset CIFAR-10 with Dirichlet setting ($\\alpha=0.3$) to conduct a series of experiments on benchmark methods with client number N $\\in $ \\{50, 100, 150, 300, 500\\}. As Table 1 (in our new supplementary) demonstrates, our Fed-CO$_2$ consistently outperforms a bunch of SOTA methods in every case. The results in this Table and Fig. 1 (in our new supplementary) prove that our Fed-CO$_2$ owns higher scalability than benchmark methods. These supplemented experimental results will be included in our final version. \n\n\n>**Q4**: What are the computational costs associated with the proposed intra-client and inter-client knowledge transfer mechanisms?\n\n**R4**: Intra-client knowledge transfer mechanism will add one epoch in the local training phase of each client. In this epoch, the online and offline models learn from each other via knowledge distillation. Inter-client knowledge transfer mechanism does not introduce extra data epochs. The extra computation cost is the calculation of cross-entropy loss between the prediction of frozen classifiers from other clients and the labels. \n\n>**Q5**: How does Fed-CO2 handle potential security and privacy concerns associated with inter-client knowledge transfer?\n\n**R5**: Our inter-client knowledge transfer only communicates partial model parameters without exposing client's data. The traditional FL algorithm FedAvg [2] communicates clients' models to the server as well to do model parameter aggregation on the server. Therefore, model communication will not cause serious security and privacy issues.\n\n>**Q6**: How does Fed-CO2 handle scenarios where the label distribution skew and feature skew are extreme?\n\n**R6**: We enhance model cooperation via intra-client and inter-client knowledge transfer mechanisms based on the prediction fusion framework. The intra-client knowledge transfer mechanism is applied to facilitate mutual learning between the online and offline models and the inter-client knowledge transfer mechanism is utilized to leverage additional knowledge from other clients.\n\n>**Q7**:\nCan the framework handle other types of data heterogeneity beyond label distribution skew and feature skew?\n\n**R7**: Yes, our framework can handle other types of data heterogeneity. Apart from label distribution skew and feature skew, we also studied the heterogeneity case with mixed label distribution skew and feature skew. The experiment results were shown in Table 4, Table 8, 9 in the main paper.\n\nMoreover, we have added additional experiments on the new non-I.I.D. scenarios, where each client has the same label distribution but is subject to various degrees of noise. In detail, we divide the dataset CIFAR-10 into 10 clients in a way to make them independently and identically distributed. Then we apply the increasing level of random Gaussian noise with mean $\\mu_{i}=0$ and standard deviation $\\sigma_{i}=\\frac{\\sigma_M}{N-1}*i$, where $i\\in$ \\{0, 1,$\\cdots$, N-1\\}. In this series of initial experiments, we let the client number $N=10$ and the sample rate be 0.5 with $\\sigma_{M}\\in $ \\{0, 5, 10, 15, 20\\}. The backbone of each method is the same as that in other experiments. \n\nAs presented in Fig. 2 (in our new supplementary), the experimental results reveal that our Fed-CO$_2$ consistently exhibits better and more robust performance across nearly all experiments, particularly in scenarios with severe noise. These experiments will be incorporated into the Appendix of our final version and will also serve as a source of inspiration for our future work."}, {"review_id": "5L1Z41Tbld", "reviewer": "Reviewer_y4gh", "summary": "The paper proposes a personalized federated learning (PFL) method to handle both label and feature distribution skew.Specifically, each clients holds a partially perosnalized model (personalization happens at the level of batch-normaizliation layers), and a fully personalized local model.  The paper conducts a series of numerical sumilations to evaluate the performance of the proposed method and to compares with state-of-the-art PFL methods.", "questions": "- Could you please provide more intuition on the particular choice of the loss function in (6) and (7)?\n- Could you please double check (3)?", "limitations": "- The proposed method induces a memory overhead in comparison to  standard federated averaging.\n- The paper does not provide any theoretical guarantees.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper is overall well-written and easy to follow. \n- The proposed method shows promising performance and outperforms competitors in most cases. \n- The proposed method does not induce any communication overhead compared to standard federated aggregation operations since the fully personalized offline models are not uploaded to the server.", "weaknesses": "- The paper does not provide stong explantion to motivate the proposed method. In particular, it is unclear where (6) and (7) come from, and why they take this specific form. \n- The proposed method induces a memory overhead in comparison to  standard federated averaging.\n- Apart from proposing a PFL with good performance, the paper does not provide any insights helping to understand personalization.\n- The paper does not provide any theoretical guarantees. \n- I think that (3) is not accurate. Maybe a division by 2 in the RHS of (3) is missing. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">**Q1**: The paper does not provide stong explantion to motivate the proposed method. In particular, it is unclear where (6) and (7) come from, and why they take this specific form.\n\n**R1**: The motivation of our proposed method is answered together with Q3. Formulas (6) and (7) describe the generalization loss function $L_{gen}$ for the online and offline models separately, which force feature extractor $f_{i}$ to produce robust and well-generalized features that can be recognized by frozen classifiers $\\overline{C}\\_{j}^{\\rm off} (j\\neq i)$ introduced from other clients' offline models. \n\nThis function facilitates the acquisition of domain-invariant knowledge for the online and offline models from other clients. Our design is inspired by the belief that if the feature extractor owns general knowledge from other clients then the image features extracted by it should be well recognized by classifiers trained on other clients.\n\nTake formula (6) as an example, which transfers general knowledge from other clients to the online model. $f_{i}^{\\rm on}(\\eta_{i}^{\\rm on}; x_{k})$ represents image features extracted by the feature extractor of the online model. Then we give these features to frozen classifiers from other clients. $\\overline{C}\\_{j}^{\\rm off}(\\overline{\\phi}\\_{j}^{\\rm off}; f\\_{i}^{\\rm on}(\\eta\\_{i}^{\\rm on}; x\\_{k}))$ represents the prediction logits provided by the classifier from client j. \n\n>**Q2**: The proposed method induces a memory overhead in comparison to standard federated averaging.\n\n**R2**: Yes, we do need more memory overhead but our method achieves SOTA in cases with severe data heterogeneity, where standard federated averaging shows a quite poor performance. In addition, some published methods induced a memory overhead (even more than ours) as well. For example, KNN-Per [51] employed a local memory bank to save local image features; PerFCL [52] kept an extra local feature extractor to obtain local information; [12] utilized an extra personalized classifier or hyper network to deal with heterogeneity issues.\n\n[51] Marfoq, Othmane, et al. \"Personalized federated learning through local memorization.\" International Conference on Machine Learning. PMLR, 2022.\n\n[52] Zhang, Yupei, et al. \"Doubly contrastive representation learning for federated image recognition.\" Pattern Recognition 139 (2023): 109507.\n\n>**Q3**: Apart from proposing a PFL with good performance, the paper does not provide any insights helping to understand personalization.\n\n**R3**: The key of Fed-CO$_2$ relies on the fusion of offline specialized knowledge and online general knowledge that is crucial for federated personalization. Our insight comes from the observation (as mentioned in line 40) that in certain instances of extreme data heterogeneity, models trained by existing personalized algorithms may exhibit inferior performance compared to the locally-trained model. Conversely, in FL scenarios with milder heterogeneity, partially personalized models trained by PFL algorithms perform better. \n\nBased on this observation, we raised the question (as mentioned in line 46): Is there a more effective approach to fuse the online general knowledge and the offline specialized knowledge to achieve better performance? To answer this question, we conducted a series of experiments and designed our effective universal framework Fed-CO$_2$.\n\n>**Q4**: The paper does not provide any theoretical guarantees.\n\n**R4**: Actually, we have provided a detailed theoretical analysis from the perspective of Neural Tangent Kernel [47] in Appendix C. As reviewer NQAC suggests in Q4, we will present the main theorem in the main body of the paper in our final version.\n\n>**Q5**: I think that (3) is not accurate. Maybe a division by 2 in the RHS of (3) is missing.\n\n**R5**: In fact, whether (3) is divided by 2 or not remains consistent for the final prediction result. This is because Formula (3) is only utilized during the testing phase for class prediction. In our framework, the online model $F^{\\rm on}$ and offline model $F^{\\rm off}$ are trained separately. We will add this illustration in the final version. \n"}, {"review_id": "dJLBi1kyWw", "reviewer": "Reviewer_NQAC", "summary": "In this paper, the authors focus on tackling the feature skew issue, and introduces a new approach that utilizes knowledge distillation. Specifically, the proposed method transfers the knowledge between the FL online model and the local offline model to enable them to have both domain invariant and domain specific knowledge. ", "questions": "1. Refer weakness 1 and weakness 2. \n\n2. Regarding the order of two knowledge transfers:  Why is Intra-Client Knowledge Transfer executed before Inter-Client Knowledge Transfer？\n\n3. Why is the theoretical analysis part only included in the appendix? In my opinion, at least the main theorem should be presented in the main body of the paper, while the assumptions, lemmas, and proofs could be included in the appendix.", "limitations": "NA", "rating": 6, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "Strength:\n\n1. The proposed approach of mutual learning between the offline and online models is novel.\n\n2. Theoretical analysis of the proposed method is provided (though they are all in the appendix)\n\n3. Experimental results on multiple datasets with both label distribution skew and feature skew demonstrate promising outcomes.", "weaknesses": "1. Lack of important baselines: \n\t\n\ta. Ditto: Since Ditto is a very classic pFL method, it should be included as the baseline. \n\n\tb. FedMask [1]: There is no hypernetwork based method that directly personalizing parts of the model parameters. FedMask is one of the SOTA method, which learns distinct binary masks for the last several layers of the local models, and aggregates the masks with an intersection operation. \n\n2. Lack of some experiments:\n\n\tThe proposed method actually contains three components, intra-client, inter-client knowledge distillation, and the fuse of online, offline models. In the ablation study, the authors focus on the experiment of the previous two parts, and the fuse part is only mentioned in the label distribution skew setting (Figure 4 in the appendix). How about conducting experiments to evaluate the performance of the online and offline models in the feature distribution skew setting, i.e., the DomainNet dataset? \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">**Q1**: Lack of important baselines:\na. Ditto: Since Ditto is a very classic pFL method, it should be included as the baseline.\nb. FedMask [1]: There is no hypernetwork based method that directly personalizing parts of the model parameters. FedMask is one of the SOTA method, which learns distinct binary masks for the last several layers of the local models, and aggregates the masks with an intersection operation.\n\n**R1**: Thank you for providing us with two important baselines.  We will cite them in Related Work of our final version but will not compare our method with them. Generally, it is hard to compare all the existing methods in federated learning. \n\nFor Ditto, one of our baseline methods FedRoD [12] has shown higher performance than it in the case of label distribution skew, while our Fed-CO$_2$ outperforms FedRoD[12] on FL with label distribution skew, feature skew, and both. This can clearly prove our method is better than Ditto on PFL issues. For FedMask, this algorithm is designed to address label distribution skew issues with no special design to overcome domain gaps. Moreover, we have not yet found published code online. It is hard to reimplement this algorithm. \n\n>**Q2**: Lack of some experiments:\nThe proposed method actually contains three components, intra-client, inter-client knowledge distillation, and the fuse of online, offline models. In the ablation study, the authors focus on the experiment of the previous two parts, and the fuse part is only mentioned in the label distribution skew setting (Figure 4 in the appendix). How about conducting experiments to evaluate the performance of the online and offline models in the feature distribution skew setting, i.e., the DomainNet dataset?\n\n**R2**: Here we supplement experiments that evaluate the performance of the online and offline models in FL with the feature skew on DomainNet. The experimental results and relevant analyses will be added in the Appendix of our final version.\n\nBased on the experimental results shown in the Table below, we have the following observations:\nFirst, the online model outperforms the offline model on most clients, except for the Quickdraw client. This phenomenon validates our hypothesis that when the feature shift is severe, the model aggregation will lead to the loss of important local offline information, resulting in the model's failure to adapt to such significant data heterogeneity. Second, prediction fusion is even inferior to the online and offline models for some clients. This result reflects that prediction fusion is not sufficient to fuse online general knowledge and offline specialized knowledge. \n\nTherefore, for FL with feature skew, intra-client and inter-client knowledge transfer mechanisms are required to boost model and client cooperation under our novel framework.\n\n\n| Method                                        | Clipart  | Infograph | Painting | Quickdraw | Real     | Sketch   | Avg     |\n| -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- |\n| Online Model                                  | 51.2±1.4 | 26.8±0.5  | 41.5±1.4 | 71.3±0.7  | 54.8±0.8 | 42.1±1.3 | 48.0±1.0 |\n|Offline Model                                 | 41.0±0.9 | 23.8±1.2  | 36.2±2.7 | 73.1±0.9  | 48.5±1.9 | 34.0±1.1 | 42.8±1.5 |\n|Fed-CO2 (no intra and inter knowledge transfer) | 48.7±0.9 | 26.5±2.0  | 42.1±1.0 | 72.9±0.8  | 57.1±1.1 | 40.0±0.8 | 47.9±0.7 |\n|Fed-CO2                                       | 55.0±1.1 | 28.6±1.1  | 44.3±0.6 | 75.1±0.6  | 62.4±0.8 | 45.7±1.9 | 51.8±0.2 \n\n>**Q3**: Regarding the order of two knowledge transfers: Why is Intra-Client Knowledge Transfer executed before Inter-Client Knowledge Transfer?\n\n**R3**: If the Inter-Client Knowledge Transfer is executed before the Intra-Client Knowledge Transfer, the global general information in the online model will be damaged and the initial model for the local training process will be weaker. \n\nThe online model gets global general information after parameter aggregation on the server. But parameter aggregation will also cause the online model to forget some local specialized information. Meanwhile, the offline model keeps all learned local specialized information but cannot access any global information.\n\nIntra-Client Knowledge Transfer not only aids the online model in compensating for lost local information but also equips the offline model with some global information. This approach initializes the offline and online models better for the next local training. \n\nInter-client Knowledge Transfer utilizes local data to not only adapt to local data distribution but also concurrently enhance the domain generalization ability for the online and offline models during the local training process. When the Inter-Client Knowledge Transfer is executed ahead of the Intra-Client Knowledge Transfer, the global general knowledge in the online model will unavoidably be covered by the learned local knowledge. In addition, the initial online model will lose local knowledge and the initial offline model will lose global knowledge in this case.\n\n>**Q4**: Why is the theoretical analysis part only included in the appendix? In my opinion, at least the main theorem should be presented in the main body of the paper, while the assumptions, lemmas, and proofs could be included in the appendix.\n\n**R4**: We appreciate your valuable advice and will present the Main Theorem in the main body of the paper in our final version. "}]

Author rebuttal:
We thank the reviewers for their careful reading of our paper and help with improving our manuscript. We sincerely appreciate that you find our work proposes 'a novel and effective approach' (Reviewer 5E9k, Reviewer NQAC), addresses 'both label distribution skew and feature skew within a unified framework' (Reviewer 5E9k), shows "promising performance" (Reviewer y4gh and Reviewer NQAC) and "convincing improvement over a variety of relevant baselines in multiple settings with different types of heterogeneity" (Reviewer xxVq), and "provided theoretical analysis" (Reviewer NQAC).

In what follows, we try to address your concerns/questions and provide a detailed item-by-item response to your comments.  According to the reviewers ' advice, we have supplemented four extra experiments to make our work more comprehensive. Specifically, we study the effect of the training number ratio and the client number on our Fed-CO$_2$ and some benchmark methods. **The results are provided in Fig.1 and Table 1 in our newly submitted attachment**. We also provide an additional ablation study about prediction fusion on FL with feature skew in our response to Reviewer NQAC Q2. Finally, we further explore the performance of our Fed-CO$_2$ and some baseline methods on FL with a new type of data heterogeneity issue caused by noise and demonstrate **the results in Fig. 2 in our newly submitted attachment**.

Meta-review: The authors study the issue of data heterogeneity in Federated Learning (FL) and propose a universal FL framework that effectively handles data heterogeneity issues arising from label distribution skew, feature skew, or their combination. Specifically, they propose a novel universal cooperation framework with two models to address this challenge for both label distribution skew and feature skew data heterogeneity, referring to the model with partially personalized parameters as the online model, and the locally trained model as the offline model. They personalize Batch Normalization layers in the online model and fuse the online and offline models’ predictions as the final prediction.

The paper addresses an important problem, and the proposed framework is of interest to the FL community. In particular, the cooperation mechanism between the online and offline models is a novel and effective way to deal with data heterogeneity issues. That said, the reviewers had some concerns and questions about the paper, but they were all satisfied by the authors' responses, and they all raised their scores post-rebuttal or maintained their positive evaluations. While I think the presentation of the paper can be improved, and in particular, the motivation for the batch normalization can be further expanded in the paper, I believe the paper would be a valuable addition to the NeurIPS conference.


AC
Decision: Accept (poster)