Title: Eliminating Domain Bias for Federated Learning in Representation Space
Venue: NeurIPS 2023
Accepted: yes
Keywords: Federated Learning

Personalized Federated Learning

Representation

Knowledge Transfer
Abstract: Recently, federated learning (FL) is popular for its privacy-preserving and collaborative learning abilities. However, under statistically heterogeneous scenarios, we observe that biased data domains on clients cause a representation bias phenomenon and further degenerate generic representations during local training, i.e., the representation degeneration phenomenon. To address these issues, we propose a general framework Domain Bias Eliminator (DBE) for FL. Our theoretical analysis reveals that DBE can promote bi-directional knowledge transfer between server and client, as it reduces the domain discrepancy between server and client in representation space. Besides, extensive experiments on four datasets show that DBE can greatly improve existing FL methods in both generalization and personalization abilities. The DBE-equipped FL method can outperform ten state-of-the-art personalized FL methods by a large margin. Our code is public at https://github.com/TsingZ0/DBE.

Eliminating Domain Bias for Federated Learning in Representation Space

Abstract

Recently, federated learning (FL) is popular for its privacy-preserving and collaborative learning abilities. However, under statistically heterogeneous scenarios, we observe that biased data domains on clients cause a representation bias phenomenon and further degenerate generic representations during local training, i.e ., the representation degeneration phenomenon. To address these issues, we propose a general framework Domain Bias Eliminator ( DBE ) for FL. Our theoretical analysis reveals that DBE can promote bi-directional knowledge transfer between server and client, as it reduces the domain discrepancy between server and client in representation space. Besides, extensive experiments on four datasets show that DBE can greatly improve existing FL methods in both generalization and personalization abilities. The DBE -equipped FL method can outperform ten state-of-the-art personalized FL methods by a large margin. Our code is public at https://github.com/TsingZ0/DBE .

1 Introduction

As a popular distributed machine learning paradigm with excellent privacy-preserving and collaborative learning abilities, federated learning (FL) trains models among clients with their private data kept locally [37, 56, 79]. Traditional FL ( e.g ., the famous FedAvg [56]) learns one single global model in an iterative manner by locally training models on clients and aggregating client models on the server. However, it suffers an accuracy decrease under statistically heterogeneous scenarios, which are common scenarios in practice [47, 56, 67, 86].

Due to statistical heterogeneity, the data domain on each client is biased, which does not contain the data of all labels [37, 45, 67, 69, 79, 84]. As the received global model is locally trained on individual clients' biased data domain, we observe that this model extracts biased ( i.e ., forming client-specific clusters) representations during local training. We call this phenomenon ' representation bias ' and visualize it in Figure 1. Meanwhile, by training the received global model with missing labels, the generic representation quality over all labels also decreases during local training [45]. Furthermore, we ob-

(a) Before.

(b) After.

1

2

3

4

4.4

4.6

4.8

5.0

5.2

5.4

5.6

MDL (bits)

1e3

before

after

(c) MDL change.

serve that this ' representation degeneration ' phenomenon happens at every layer, as shown in Figure 1(c). We estimate the representation quality via minimum description length (MDL) [62, 65, 74], a metric independent of data and models, measuring the difficulty of classifying target labels according to given representations.

To tackle the statistical heterogeneity, unlike traditional FL that learns a single global model, personalized FL (pFL) comes along by learning personalized models (or modules) for each client besides learning a global model among clients [20, 22, 69]. Typically, most of the existing pFL methods train a personalized classifier 2 for each client [3, 14, 20, 61], but the feature extractor still extracts all the information from the biased local data domain, leading to representation bias and representation degeneration during local training.

To address the representation bias and representation degeneration issues in FL, we propose a general framework Domain Bias Eliminator ( DBE ) for FL including two modules introduced as follows. Firstly, we detach the representation bias from original representations and preserve it in a Personalized Representation Bias Memory ( PRBM ) on each client. Secondly, we devise a Mean Regularization ( MR ) that explicitly guides local feature extractors to extract representations with a consensual global mean during local training to let the local feature extractor focus on the remaining unbiased information and improve the generic representation quality. In this way, we turn one level of representation between the feature extractor and the classifier on each client into two levels of representation with a client-specific bias and a client-invariant mean, respectively. Thus, we can eliminate the conflict of extracting representations with client-specific biases for clients' requirements while extracting representations with client-invariant features for the server's requirements in the same representation space. Our theoretical analysis shows that DBE can promote the bi-directional knowledge transfer between server and client with lower generalization bounds.

We conduct extensive experiments in computer vision (CV) and natural language processing (NLP) fields on various aspects to study the characteristics and effectiveness of DBE . In both generalization ability (measured by MDL) and personalization ability (measured by accuracy), DBE can promote the fundamental FedAvg as well as other representative FL methods. Furthermore, we compare the representative FedAvg+ DBE with ten state-of-the-art (SOTA) pFL methods in various scenarios and show its superiority over these pFL methods. To sum up, our contributions are:

We observe the representation bias and per-layer representation degeneration phenomena during local training in the representative FL method FedAvg.

We propose a framework DBE to memorize representation bias on each client to address the representation bias issue and explicitly guide local feature extractors to generate representations with a universal mean for higher generic representation quality.

We provide theoretical analysis and derive lower generalization bounds of the global and local feature extractors to show that DBE can facilitate bi-directional knowledge transfer between server and client in each iteration.

Weshow that DBE can improve other representative traditional FL methods including FedAvg at most -22.35% in MDL (bits) and +32.30 in accuracy (%), respectively. Furthermore, FedAvg+ DBE outperforms SOTA pFL methods by up to +11.36 in accuracy (%).

2 Related Work

Traditional FL methods that focus on improving accuracy under statistically heterogeneous scenarios based on FedAvg including four categories: update-correction-based FL [25, 38, 60], regularizationbased FL [1, 17, 40, 46], model-split-based FL [35, 45], and knowledge-distillation-based FL [27, 33, 88, 96]. For pFL methods, we consider four categories: meta-learning-based pFL [13, 22], regularization-based pFL [47, 67], personalized-aggregation-based pFL [21, 52, 87, 89], and modelsplit-based pFL [3, 14, 20, 61, 85]. Due to limited space, we only introduce the FL methods that are close to ours and leave the extended version of this section to Appendix A.

Traditional FL methods. MOON[45] utilizes contrastive learning to correct the local training of each client, but this input-wise contrastive learning still relies on the biased local data domain, so it still suffers from representation skew. Although FedGen [96] learns a shared generator on the server

and reduces the heterogeneity among clients with the generated representations through knowledge distillation, it only considers the local-to-global knowledge for the single global model learning. On the other hand, FedGen additionally brings non-negligible communication and computation overhead for learning and transmitting the generator.

pFL methods. FedPer [3] and FedRep [20] keep the classifier locally, but the feature extractor still learns biased features without explicit guidance. Besides, their local feature extractors are trained to cater to personalized classifiers thus losing generality. FedRoD [14] reduces the discrepancy of local training tasks among clients by using a balanced softmax (BSM) loss function [64], but the BSM is useless for missing labels on each client while label missing is a common situation in statistically heterogeneous scenarios [50, 86, 89]. Moreover, the uniform label distribution modified by the BSM cannot reflect the original distribution. Differently, FedBABU [61] trains a global feature extractor with a naive and frozen classifier, then it fine-tunes the classifier for each client to finally obtain personalized models. However, the post-FL fine-tuning study is beyond the FL scope, as almost all the FL methods have multiple fine-tuning variants, e.g ., fine-tuning the whole model or only a part of the model. Like FedAvg, FedBABU still locally extracts biased features during the FL process.

3 Notations and Preliminaries

3.1 Notations

In this work, we discuss the statistically heterogeneous scenario in typical multi-class classification tasks for FL, where N clients share the same model structure. Here, we denote notations following FedGen [96] and FedRep [20]. The client i, i ∈ [ N ] , has its own private data domain D i , where the data are sampled from D i . All the clients collaborate to train a global model g parameterized by θ without sharing their private local data.

Since we focus on representation learning in FL, we regard g as the sequential combination of a feature extractor f that maps from the input space X to a representation space Z , i.e ., f : X ↦→ Z parameterized by θ f and a classifier h that maps from the representation space to the output space △ Y , i.e ., h : Z ↦→ △ Y parameterized by θ h . Formally, we have g := h ◦ f , θ := [ θ f ; θ h ] , X ⊂ R D and Z ⊂ R K . △ Y is the simplex over label space Y ⊂ R . With any input x ∈ X , we obtain the feature representation by z = f ( x ; θ f ) ∈ Z .

3.2 Traditional Federated Learning

With the collaboration of N clients, the objective of traditional FL, e.g ., FedAvg [56], is to iteratively learn a global model that minimizes its loss on each local data domain:

where ℓ : △ Y ×Y ↦→ R is a non-negative and convex loss function. Following FedGen, we assume that all clients share an identical loss function ℓ and a virtual global data domain D , which is the union of all local domains: D := ⋃ N i =1 D i . In practice, traditional FL methods [45, 46, 56] optimize Eq. (1) by min θ ∑ N i =1 n i n L ˆ D i ( θ ) , where ˆ D i is an observable dataset, n i = | ˆ D i | is its size, and n = ∑ N i =1 n i .

In each communication iteration, clients conduct local updates on their private data to train the global model θ by minimizing their local loss. Formally, for client i , the objective during local training is min θ L D i ( θ ) . The empirical version of L D i ( θ ) is L ˆ D i ( θ ) := 1 n i ∑ n i j =1 ℓ ( h ( f ( x ij ; θ f ); θ h ) , y ij ) , which is optimized by stochastic gradient descent (SGD) [56, 90] in FedAvg.

4 Method

4.1 Problem Statement

pFL iteratively learns a personalized model or module for each client with the assistance of the global model parameters from the server. Our objective is (with a slight reuse of the notation L D i )

where θ i is a model consisting of global and personalized modules. The global modules are locally trained on clients and shared with the server for aggregation like traditional FL, but the personalized modules are preserved locally on clients. Following traditional FL, we empirically optimize Eq. (3) by min θ 1 ,..., θ N ∑ N i =1 n i n L ˆ D i ( θ i ) .

4.2 Personalized Representation Bias Memory ( PRBM )

Due to the existence of statistical heterogeneity in FL, the local feature extractor intends to learn biased representations after being overwritten by the received global model parameters. To detach and store the representation bias locally, we propose a personalized module PRBM that memorizes representation bias for client i . Originally, the feature representation z i ∈ R K is directly fed into the predictor in Eq. (2). Instead, we consider z i as the combination of a global z g i ∈ R K and a personalized ¯ z p i ∈ R K , i.e .,

We let the feature extractor output z g i instead of the original z i , i.e ., z g i := f ( x i ; θ f ) and keep the trainable vector ¯ z p i locally. ¯ z p i is specific

⨁

Feature

Extractor

Classifier

Feature

Extractor

Classifier

(a) Local model (original).

⨁

Feature

Extractor

Classifier

Feature

Extractor

Classifier

among clients but identical for all the local data on one client, so it memorizes client-specific mean. The original feature extractor is trained to capture the biased features for z i . Instead, with the personalized mean stored in ¯ z p i , the feature extractor turns to capture z g i with less biased feature information. We illustrate the difference between the original approach and our method in Figure 2 ( PRBM ). Then, we define the local objective as min θ i L D i ( θ i ) , where θ i := [ θ f ; ¯ z p i ; θ h ] ,

From the view of transformation, we rewrite Eq. (5) to

where PRBM : Z ↦→ Z a personalized translation transformation [78] parameterized by ¯ z p i . Formally, PRBM ( z g i ; ¯ z p i ) = z g i + ¯ z p i , ∀ z g i ∈ Z . With PRBM , we create an additional level of representation z g i besides the original level of representation z i . We call z g i and z i as the first and second levels of representation , respectively. For the original local model (Figure 2(a)), we have z g i ≡ z i .

4.3 Mean Regularization ( MR )

Without explicit guidance, it is hard for the feature extractor and the trainable PRBM to distinguish between unbiased and biased information in representations automatically. Therefore, to let the feature extractor focus on the unbiased information and further separate z g i and ¯ z p i , we propose an MR that explicitly guides the local feature extractor to generate z g i with the help of a client-invariant mean, which is opposite to the client-specific mean memorized in ¯ z p i , as shown in Figure 2 ( MR ). Specifically, we regularize the mean of z g i to the consensual global mean ¯ z g at each feature dimension independently. We then modify Eq. (6) as

where ¯ z g i = E ( x i ,y i ) ∼D i [ f ( x i ; θ f )] . Weobtain the consensus ¯ z g = ∑ N i =1 ¯ z g i during the initialization period before FL (see Algorithm 1) . We measure the distance of ¯ z g i and ¯ z g by mean squared error (MSE) [72], and κ is a hyperparameter to control the importance of MR for different tasks. Empirically,

which is also optimized by SGD following FedAvg.

In Eq. (8), the value of the MR term is obtained after calculating the empirical version of ¯ z g i : ˆ ¯ z g i = 1 n i ∑ n i j =1 f ( x ij ; θ f ) over the entire local data, but the loss computing in SGD cannot see all the local data during one forward pass in one batch. In practice, inspired by the widely-used moving average [48, 90] in approximating statistics over data among batches, in each batch, we obtain

where ˆ ¯ z g i,old and ˆ ¯ z g i,new are computed in the previous batch and current batch, respectively. µ is a hyperparameter called momentum that controls the importance of the current batch. The feature extractor is updated continuously during local training but discontinuously between adjacent two iterations due to server aggregation. Thus, we only calculate ˆ ¯ z g i via Eq. (9) during local training and recalculate it in a new iteration without using its historical records. We consider the representative FedAvg+ DBE as an example and show the entire learning process in Algorithm 1.

Algorithm 1 The Learning Process in FedAvg+ DBE

Input: N clients with their local data; initial parameters θ f, 0 and θ h, 0 ; η : local learning rate; κ and µ : hyperparameters; ρ : client joining ratio; E : local epochs; T : total communication iterations. Output: Global model parameters { θ f , θ h } and personalized model parameters { ¯ z p , . . . , ¯ z p } .

1: Server sends { θ f, 0 , θ h, 0 }

▷ Initialization Period to all clients to initialize their local models.

2: N clients train their local models without DBE for one epoch and collect client-specific mean { ¯ z g 1 , . . . , ¯ z g N } over their data domain.

4: Client i initializes ¯ z p, 0 i , ∀ i ∈ [ N ] .

5: for communication iteration t = 1 , . . . , T do

6: Server samples a client subset I t based on ρ .

7: Server sends { θ f,t -1 , θ h,t -1 } to each client in I t .

8: for Client i ∈ I t in parallel do

9: Initialize f and h with θ f,t -1 and θ h,t -1 , respectively.

10: Obtain { θ f,t i , ¯ z p,t i , θ h,t i } using SGD for min θ i L ˆ D i ( θ i ) with η , κ and µ for E epochs.

11: Upload { θ f,t i , θ h,t i } to the server.

12: Server calculates n t = ∑ i ∈I t n i and obtains

13: θ f,t = ∑ i ∈I t n i n t θ f,t i ;

14: θ h,t = ∑ i ∈I t n i n t θ h,t i .

15: return { θ f,T , θ h,T } and { ¯ z p,T 1 , . . . , ¯ z p,T N }

4.4 Improved Bi-directional Knowledge Transfer

In the FL field, prior methods draw a connection from FL to domain adaptation for theoretical analysis and consider a binary classification problem [21, 55, 69, 96]. The traditional FL methods, which focus on enhancing the performance of a global model, regard local domains D i , i ∈ [ N ] and the virtual global domain D as the source domain and the target domain, respectively [96], which is called local-to-global knowledge transfer in this paper. In contrast, pFL methods that focus on improving the performance of personalized models regard D and D i , i ∈ [ N ] as the source domain and the target domain, respectively [21, 55, 69]. We call this kind of adaptation as global-to-local knowledge transfer. The local-to-global knowledge transfer happens on the server while the global-to-local one occurs on the client. Please refer to Appendix B for details and proofs.

4.4.1 Local-To-Global Knowledge Transfer

Here, we consider the transfer after the server receives a client model. We guide the feature extractor to learn representations with a global mean and gradually narrow the gap between the local domain and global domain at the first level of representation ( i.e ., z g i ) to improve knowledge transfer:

Corollary 1. Consider a local data domain D i and a virtual global data domain D for client i and the server, respectively. Let D i = ⟨U i , c ∗ ⟩ and D = ⟨U , c ∗ ⟩ , where c ∗ : X ↦→ Y is a ground-truth

3: Server generates a consensual global mean ¯ z g through weighted averaging: ¯ z g = ∑ N i =1 n i n ¯ z g i .

▷ Federated Learning Period

1 N

labeling function. Let H be a hypothesis space of VC dimension d and h : Z ↦→ Y , ∀ h ∈ H . When using DBE , given a feature extraction function F g : X ↦→ Z that shared between D i and D , a random labeled sample of size m generated by applying F g to a random sample from U i labeled according to c ∗ , then for every h g ∈ H , with probability at least 1 -δ :

where L ˆ D i is the empirical loss on D i , e is the base of the natural logarithm, and d H ( · , · ) is the H -divergence between two distributions. λ i := min h g L D ( h g ) + L D i ( h g ) , ˜ U g i ⊆ Z , ˜ U g ⊆ Z , and d H ( ˜ U g i , ˜ U g ) ≤ d H ( ˜ U i , ˜ U ) . ˜ U g i and ˜ U g are the induced distributions of U i and U under F g , respectively. ˜ U i and ˜ U are the induced distributions of U i and U under F , respectively. F is the feature extraction function in the original FedAvg without DBE .

As shown in Figure 2, given any x i on client i , one can obtain z i via F in original FedAvg or obtain z g i via F g in FedAvg+ DBE . With d H ( ˜ U g i , ˜ U g ) ≤ d H ( ˜ U i , ˜ U ) holds, we can achieve a lower generalization bound in local-to-global knowledge transfer than traditional FL, thus training a better global feature extractor to produce representations with higher quality over all labels. A small gap between the local domain and global domain in Z promotes the knowledge transfer from clients to the server [82, 92, 94].

4.4.2 Global-To-Local Knowledge Transfer

The global-to-local knowledge transfer focuses on the assistance role of the global model parameters for facilitating local training, i.e ., the transfer ability from D to D i . After the client receives the global model and equips it with PRBM , for the second level of representation ( i.e ., z i ), we have

Corollary 2. Let D i , D , F g , and λ i defined as in Corollary 1. Given a translation transformation function PRBM : Z ↦→ Z that shared between D i and virtual D , a random labeled sample of size m generated by applying F ′ to a random sample from U i labeled according to c ∗ , F ′ = PRBM ◦ F g : X ↦→ Z , then for every h ′ ∈ H , with probability at least 1 -δ :

where d H ( ˜ U ′ , ˜ U ′ i ) = d H ( ˜ U g , ˜ U g i ) ≤ d H ( ˜ U , ˜ U i ) = d H ( ˜ U i , ˜ U ) . ˜ U ′ and ˜ U ′ i are the induced distributions of U and U i under F ′ , respectively.

Given x i on client i , we can obtain z i via F ′ in FedAvg+ DBE . h g = h ′ ◦ PRBM , so PRBM does not influence the value of d H ( · , · ) for the pair of h g and h ′ (see Appendix B.3), then we have d H ( ˜ U ′ , ˜ U ′ i ) = d H ( ˜ U g , ˜ U g i ) . The inequality d H ( ˜ U ′ , ˜ U ′ i ) ≤ d H ( ˜ U , ˜ U i ) shows that the information aggregated on the server can be more easily transferred to clients with our proposed DBE than FedAvg. We train PRBM on the local loss and preserve it locally, so the local feature extractors can generate representations suitable for clients' personalized tasks. According to Corollary 1 and Corollary 2, adding DBE facilitates the bi-directional knowledge transfer in each iteration, gradually promoting global and local model learning as the number of iterations increases.

4.5 Negligible Additional Communication and Computation Overhead

DBE only modifies the local training, so the downloading, uploading, and aggregation processes in FedAvg are unaffected. In FedAvg+ DBE , the communication overhead per iteration is the same as FedAvg but requires fewer iterations to converge (see Appendix D). Moreover, PRBM only introduces K additional trainable parameters, and the MSE value in the parameterless MR is computed for two representations of K dimension. K is the representation space dimension, typically a smaller value than the dimension of data inputs or model parameters [8, 83]. Thus, DBE introduces no additional communication overhead and negligible computation overhead for local training in any iteration.

4.6 Privacy-Preserving Discussion

Compared to FedAvg, using DBE requires client i to upload one client-specific mean ¯ z g i (one K -dimensional vector) to the server only once before FL , which solely captures the magnitude of the

mean value for each feature dimension within the context of the given datasets and models. Thanks to this particular characteristic, as shown in Section 5.1.4, the performance of FedAvg+ DBE can be minimally affected while enhancing its privacy-preserving capabilities by introducing proper Gaussian noise with a zero mean to ¯ z g i during the initialization phase.

5 Experiments

Datasets and models. Following prior FL approaches [14, 20, 45, 50, 56], we use four public datasets for classification problems in FL, including three CV datasets: Fashion-MNIST (FMNIST) [77], Cifar100 [42], and Tiny-ImageNet (100K images with 200 labels) [19], as well as one NLP dataset: AG News [91]. For three CV datasets, we adopt the popular 4-layer CNN by default following FedAvg, which contains two convolution layers (denoted by CONV1 and CONV2) and two fully connected layers (denoted by FC1 and FC2). Besides, we also use a larger model ResNet-18 [29] on Tiny-ImageNet. For AG News, we use the famous text classification model fastText [36].

Statistically heterogeneous scenarios. There are two widely used approaches to construct statistically heterogeneous scenarios on public datasets: the pathological setting [56, 66] and practical setting [45, 50]. For the pathological setting, disjoint data with 2/10/20 labels for each client are sampled from 10/100/200 labels on FMNIST/Cifar100/Tiny-ImageNet with different data amounts. For the practical setting, we sample data from FMNIST, Cifar100, Tiny-ImageNet, and AG News based on the Dirichlet distribution [50] (denoted by Dir ( β ) ). Specifically, we allocate a q c,i ( q c,i ∼ Dir ( β ) ) proportion of samples with label c to client i , and we set β = 0 . 1 / β = 1 by default for CV/NLP tasks following previous FL approaches [50, 75].

Implementation Details. Following pFedMe and FedRoD, we have 20 clients and set client participating ratio ρ = 1 by default unless otherwise stated. We measure the generic representation quality across clients and evaluate the MDL [65, 74] of representations over all class labels. To simulate the common FL scenario where data only exists on clients, we split the data among each client into two parts: a training set (75% data) and a test set (25% data). Following pFedMe, we evaluate pFL methods by averaging the results of personalized models on the test set of each client and evaluate traditional FL methods by averaging the results of the global model on each client. Following FedAvg, we set the batch size to 10 and the number of local epochs to 1, so the number of local SGD steps is ⌊ n i 10 ⌋ for client i . We run three trials for all methods until empirical convergence on each task and report the mean value. For more details and results ( e.g ., fine-tuning FedAvg on new participants and a real-world application), please refer to Appendix D.

5.1 Experimental Study for Adding DBE

5.1.1 How to Split the Model?

A model is split into a feature extractor and a classifier, but there are various ways for splitting, as each layer in a deep neural network (DNN) outputs a feature representation and feeds it into the next layer [8, 44] [8, 44]. We focus on inserting DBE between the feature extractor and the classifier, but which splitting way is the best for DBE ? Here we answer this question by comparing the results regarding MDL and accuracy when the model is split at each layer in the popular 4-layer CNN. We show the MDL of the representation z g i outputted by the prepositive layer of DBE (with underline here) and show MDL of z i for other layers. Low MDL and high accuracy indicate superior generalization ability and superior personalization ability, respectively.

In Table 1, the generic representation quality is improved at each layer for all splitting ways, which shows that no matter how the model is split, DBE can enhance the generalization ability of the global feature extractor. Among these splitting ways, assigning all FC layers to the classifier, i.e ., CONV2 → DBE → FC1, achieves almost the lowest MDL and highest accuracy. Meanwhile, FC1 → DBE → FC2 can also achieve excellent performance with only 4 . 73% trainable parameters for DBE .

Since FedRep, FedRoD, and FedBABU choose the last FC layer as the classifier by default, we follow them for a fair comparison and insert DBE before the last FC layer ( e.g ., FC1 → DBE → FC2). In Table 1, our FC1 → DBE → FC2 outperforms FedPer, FedRep, FedRoD, FedBABU, and FedAvg with lower MDL and higher accuracy. Since feature extractors in FedPer and FedRep are locally trained to cater to personalized classifiers, they extract representations with low quality.

5.1.2 Representation Bias Eliminated for the First Level of Representation

Figure 3: t-SNE visualization for representations on Tiny-ImageNet (200 labels). 'B' and 'A' denote 'before local training' and 'after local training', respectively. We use color and shape to distinguish labels and clients , respectively. Best viewed in color and zoom-in.

Wevisualize the feature representations using t-SNE [73] in Figure 3. Compared to the representations outputted by the feature extractor in FedAvg, z g i in FedAvg+ DBE is no longer biased to the local data domain of each client after local training. With the personalized translation transformation PRBM , z i can fit the local domain of each client either before or after local training. According to Figure 3(b), Figure 3(e) and Figure 3(f), z i in FedAvg+ DBE can fit the local domain better than FedAvg.

5.1.3 Ablation Study for DBE

We further study the contribution of MR and PRBM in terms of generalization and personalization abilities by applying only one of them to FedAvg. From Table 2, we find that for 4-layer CNN and ResNet-18, + DBE gives a larger improvement in both MDL and accuracy than just using MR or PRBM , which suggests that MR and PRBM can boost each other in bi-directional knowledge transfer. The contribution of MR is greater than that of PRBM in improving the generalization ability in MDL, while + PRBM gains more accuracy improvement for personalization ability than MR .

5.1.4 Privacy-Preserving Ability

Following FedPAC [70], we add Gaussian noise to client-specific means ¯ z g 1 , . . . , ¯ z g N with a scale parameter ( s ) for the noise distribution and perturbation coefficient ( q ) for the noise. Adding the unbiased noise sampled from one distribution is beneficial for representation bias elimination and can further improve the performance of DBE to some extent, as shown in Table 3. Besides, adding too much noise can also bring an accuracy decrease. However, setting s = 0 . 05 and q = 0 . 2 is sufficient to ensure privacy protection according to FedPCL.

5.1.5 DBE Improves Other Traditional Federated Learning Methods

Alarge number of FL methods design algorithms based on the famous FedAvg [37, 56, 69]. Although we describe DBE based on FedAvg for example, DBE can also be applied to other traditional FL methods to improve their generalization and personalization abilities. Here, we apply DBE to another four representative traditional FL methods: SCAFFOLD [38], FedProx [46], MOON [45], and FedGen [96]. They belong to four categories: update-correction-based FL, regularization-based FL, model-split-based FL, and knowledge-distillation-based FL, respectively. In Table 4, DBE promotes traditional FL methods by at most -22.35% in MDL (bits) and +32.30 in accuracy (%), respectively. Based on the results of Table 2 and Table 4 on Tiny-ImageNet, FedAvg+ DBE achieves lower MDL and higher accuracy than close methods MOON and FedGen.

5.2 Comparison with Personalized Federated Learning Methods

5.2.1 Personalization Ability on Various Datasets

To further show the superiority of the DBE -equipped traditional FL methods to existing pFL methods, we compare the representative FedAvg+ DBE with ten SOTA pFL methods, as shown in Table 5. Note that APPLE is designed for cross-silo scenarios and assumes ρ = 1 . For Per-FedAvg and FedBABU, we show the test accuracy after post-FL fine-tuning. FedAvg+ DBE improves FedAvg at most +47.40

on Cifar100 in the pathological setting and outperforms the best SOTA pFL methods by up to +11.36 on Cifar100 † including the fine-tuning-based methods that require additional post-FL effort.

5.2.2 Personalization Ability Under Various Heterogeneous Degrees

Following prior methods [45, 50], we also evaluate FedAvg+ DBE with different β on Tiny-ImageNet using 4-layer CNN to study the influence of heterogeneity, as shown in Table 6. Most pFL methods are specifically designed for extremely heterogeneous scenarios and can achieve high accuracy at β = 0 . 01 , but some of them cannot maintain the advantage compared to FedAvg in moderate scenarios. However, FedAvg+ DBE can automatically adapt to all these scenarios without tuning.

5.2.3 MR Improves Personalized Federated Learning Methods

Since pFL methods already create personalized models or modules in their specific ways, applying personalized PRBM to the local model might be against their philosophy. To prevent this, we only apply the MR to pFL methods. Besides, the local training schemes ( e.g ., meta-learning) in Per-FedAvg, pFedMe, and APPLE are different from the simple SGD in FedAvg, which requires modification of the mean calculation in MR , so we do not apply MR to them. According to Corollary 1, MR can promote the local-to-global knowledge transfer between server and client. Therefore, pFL methods can benefit more from a better global model achieving higher accuracy on Tiny-ImageNet with the 4-layer CNN, as shown in Table 6. However, their MR -equipped variants perform worse than FedAvg+ DBE (Table 5, TINY) since the representation bias still exists without using PRBM .

5.2.4 Computation Overhead

We evaluate FedAvg+ DBE in total time and time per iteration on Tiny-ImageNet using ResNet-18, as shown in Table 6. The evaluation task for one method monopolizes one identical machine. FedAvg, FedBABU, and FedAvg+ DBE cost almost the same and have the lowest time per iteration among these methods, but FedAvg+ DBE requires less total time than FedAvg and FedBABU. Note that the fine-tuning time for FedBABU is not included in Table 6. Since pFedMe and Ditto train an additional personalized model on each client, they cost plenty of time per iteration.

6 Conclusion

Due to the naturally existing statistical heterogeneity and the biased local data domains on each client, FL suffers from representation bias and representation degeneration problems. To improve the generalization and personalization abilities for FL, we propose a general framework DBE including two modules PRBM and MR , with a theoretical guarantee. Our DBE can promote the bi-directional knowledge transfer in each iteration, thus improving both generalization and personalization abilities. Besides, we conduct extensive experiments to show the general applicability of DBE to existing FL methods and the superiority of the representative FedAvg+ DBE to ten SOTA pFL methods in various scenarios.

Acknowledgments and Disclosure of Funding

This work was partially supported by the Program of Technology Innovation of the Science and Technology Commission of Shanghai Municipality (Granted No. 21511104700 and 22DZ1100103). This work was also supported in part by the Shanghai Key Laboratory of Scalable Computing and Systems, National Key R&D Program of China (2022YFB4402102), Internet of Things special subject program, China Institute of IoT (Wuxi), Wuxi IoT Innovation Promotion Center (2022SP-T13-C), Industry-university-research Cooperation Funding Project from the Eighth Research Institute in China Aerospace Science and Technology Corporation (Shanghai) (USCAST2022-17), Intel Corporation (UFunding 12679), and the cooperation project from Ant Group ('Wasm-enabled Managed language in security restricted scenarios').

References

Durmus Alp Emre Acar, Yue Zhao, Ramon Matas, Matthew Mattina, Paul Whatmough, and Venkatesh Saligrama. Federated Learning Based on Dynamic Regularization. In International Conference on Learning Representations (ICLR) , 2021.

Davide Anguita, Alessandro Ghio, Luca Oneto, Xavier Parra, and Jorge L Reyes-Ortiz. Human activity recognition on smartphones using a multiclass hardware-friendly support vector machine. In Ambient Assisted Living and Home Care: 4th International Workshop, IWAAL 2012, Vitoria-Gasteiz, Spain, December 3-5, 2012. Proceedings 4 , pages 216-223. Springer, 2012.

Manoj Ghuhan Arivazhagan, Vinay Aggarwal, Aaditya Kumar Singh, and Sunav Choudhary. Federated Learning with Personalization Layers. arXiv preprint arXiv:1912.00818 , 2019.

Suresh Balakrishnama and Aravind Ganapathiraju. Linear discriminant analysis-a brief tutorial. Institute for Signal and information Processing , 18(1998):1-8, 1998.

Shai Ben-David, Nadav Eiron, and Philip M Long. On the difficulty of approximately maximizing agreements. Journal of Computer and System Sciences , 66(3):496-514, 2003.

Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Pereira. Analysis of Representations for Domain Adaptation. In International Conference on Advances in Neural Information Processing Systems (NeurIPS) , 2006.

Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. A theory of learning from different domains. Machine learning , 79:151-175, 2010.

Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation Learning: A Review and New Perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence , 35(8):1798-1828, 2013.

John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman. Learning bounds for domain adaptation. Advances in neural information processing systems , 20, 2007.

John Blitzer, Mark Dredze, and Fernando Pereira. Biographies, bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification. In Proceedings of the 45th annual meeting of the association of computational linguistics , pages 440-447, 2007.

José-Ramón Cano. Analysis of data complexity measures for classification. Expert systems with applications , 40(12):4820-4831, 2013.

Xiaoyu Cao, Jinyuan Jia, and Neil Zhenqiang Gong. Provably secure federated learning against malicious clients. In AAAI Conference on Artificial Intelligence (AAAI) , 2021.

Fei Chen, Mi Luo, Zhenhua Dong, Zhenguo Li, and Xiuqiang He. Federated Meta-Learning With Fast Convergence and Efficient Communication. arXiv preprint arXiv:1802.07876 , 2018.

Hong-You Chen and Wei-Lun Chao. On Bridging Generic and Personalized Federated Learning for Image Classification. In International Conference on Learning Representations (ICLR) , 2021.

Yiqiang Chen, Xin Qin, Jindong Wang, Chaohui Yu, and Wen Gao. Fedhealth: A federated transfer learning framework for wearable healthcare. IEEE Intelligent Systems , 35(4):83-93, 2020.

Yiqiang Chen, Wang Lu, Xin Qin, Jindong Wang, and Xing Xie. MetaFed: Federated Learning Among Federations With Cyclic Knowledge Distillation for Personalized Healthcare. arXiv preprint arXiv:2206.08516 , 2022.

Anda Cheng, Peisong Wang, Xi Sheryl Zhang, and Jian Cheng. Differentially Private Federated Learning with Local Regularization and Sparsification. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2022.

Lenaic Chizat and Francis Bach. On the global convergence of gradient descent for over-parameterized models using optimal transport. Advances in neural information processing systems , 31, 2018.

Patryk Chrabaszcz, Ilya Loshchilov, and Frank Hutter. A Downsampled Variant of Imagenet as an Alternative to the Cifar Datasets. arXiv preprint arXiv:1707.08819 , 2017.

Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai. Exploiting Shared Representations for Personalized Federated Learning. In International Conference on Machine Learning (ICML) , 2021.

Yuyang Deng, Mohammad Mahdi Kamani, and Mehrdad Mahdavi. Adaptive Personalized Federated Learning. arXiv preprint arXiv:2003.13461 , 2020.

Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Personalized Federated Learning with Theoretical Guarantees: A Model-Agnostic Meta-Learning Approach. In International Conference on Advances in Neural Information Processing Systems (NeurIPS) , 2020.

Minghong Fang, Xiaoyu Cao, Jinyuan Jia, and Neil Gong. Local Model Poisoning Attacks to ByzantineRobust Federated Learning. In USENIX Security , 2020.

Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks. In International Conference on Machine Learning (ICML) , 2017.

Liang Gao, Huazhu Fu, Li Li, Yingwen Chen, Ming Xu, and Cheng-Zhong Xu. FedDC: Federated Learning With Non-IID Data Via Local Drift Decoupling and Correction. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2022.

Bimal Ghimire and Danda B Rawat. Recent Advances on Federated Learning for Cybersecurity and Cybersecurity for Federated Learning for Internet of Things. IEEE Internet of Things Journal , 09(11): 8229 - 8249, 2022.

Xuan Gong, Abhishek Sharma, Srikrishna Karanam, Ziyan Wu, Terrence Chen, David Doermann, and Arun Innanje. Preserving Privacy in Federated Learning With Ensemble Cross-Domain Knowledge Distillation. In AAAI Conference on Artificial Intelligence (AAAI) , 2022.

Meng Hao, Hongwei Li, Xizhao Luo, Guowen Xu, Haomiao Yang, and Sen Liu. Efficient and PrivacyEnhanced Federated Learning for Industrial Artificial Intelligence. IEEE Transactions on Industrial Informatics , 16(10):6532-6542, 2019.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2016.

Tin Kam Ho and Mitra Basu. Complexity measures of supervised classification problems. IEEE transactions on pattern analysis and machine intelligence , 24(3):289-300, 2002.

Rui Hu, Yuanxiong Guo, Hongning Li, Qingqi Pei, and Yanmin Gong. Personalized federated learning with differential privacy. IEEE Internet of Things Journal , 7(10):9530-9539, 2020.

Tiansheng Huang, Weiwei Lin, Wentai Wu, Ligang He, Keqin Li, and Albert Y Zomaya. An efficiencyboosting client selection scheme for federated learning with fairness guarantee. IEEE Transactions on Parallel and Distributed Systems , 32(7):1552-1564, 2020.

Wenke Huang, Mang Ye, and Bo Du. Learn From Others and Be Yourself in Heterogeneous Federated Learning. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2022.

Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems , 31, 2018.

Meirui Jiang, Zirui Wang, and Qi Dou. Harmofl: Harmonizing Local and Global Drifts in Federated Learning on Heterogeneous Medical Images. In AAAI Conference on Artificial Intelligence (AAAI) , 2022.

Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. Bag of Tricks for Efficient Text Classification. In Conference of the European Chapter of the Association for Computational Linguistics (EACL) , 2017.

Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and Open Problems in Federated Learning. arXiv preprint arXiv:1912.04977 , 2019.

Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic Controlled Averaging for Federated Learning. In International Conference on Machine Learning (ICML) , 2020.

Daniel Kifer, Shai Ben-David, and Johannes Gehrke. Detecting change in data streams. In VLDB , pages 180-191, 2004.

Jinkyu Kim, Geeho Kim, and Bohyung Han. Multi-Level Branched Regularization for Federated Learning. In International Conference on Machine Learning (ICML) , 2022.

Jakub Koneˇ cn` y, H Brendan McMahan, Felix X Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. Federated Learning: Strategies for Improving Communication Efficiency. arXiv preprint arXiv:1610.05492 , 2016.

Alex Krizhevsky and Hinton Geoffrey. Learning Multiple Layers of Features From Tiny Images. Technical Report , 2009.

Max Kuhn, Kjell Johnson, Max Kuhn, and Kjell Johnson. Discriminant analysis and other linear classification models. Applied predictive modeling , pages 275-328, 2013.

Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep Learning. Nature , 521(7553):436-444, 2015.

Qinbin Li, Bingsheng He, and Dawn Song. Model-Contrastive Federated Learning. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2021.

Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated Optimization in Heterogeneous Networks. In Conference on Machine Learning and Systems (MLSys) , 2020.

Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. Ditto: Fair and Robust Federated Learning Through Personalization. In International Conference on Machine Learning (ICML) , 2021.

Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. On the Convergence of FedAvg on Non-IID Data. In International Conference on Learning Representations (ICLR) , 2020.

Zengpeng Li, Vishal Sharma, and Saraju P Mohanty. Preserving Data Privacy via Federated Learning: Challenges and Solutions. IEEE Consumer Electronics Magazine , 9(3):8-16, 2020.

Tao Lin, Lingjing Kong, Sebastian U Stich, and Martin Jaggi. Ensemble Distillation for Robust Model Fusion in Federated Learning. In International Conference on Advances in Neural Information Processing Systems (NeurIPS) , 2020.

Yang Liu, Yan Kang, Chaoping Xing, Tianjian Chen, and Qiang Yang. A secure federated transfer learning framework. IEEE Intelligent Systems , 35(4):70-82, 2020.

Jun Luo and Shandong Wu. Adapt to Adaptation: Learning Personalization for Cross-Silo Federated Learning. In International Joint Conference on Artificial Intelligence (IJCAI) , 2022.

Xinjian Luo, Yuncheng Wu, Xiaokui Xiao, and Beng Chin Ooi. Feature inference attack on model predictions in vertical federated learning. In 2021 IEEE 37th International Conference on Data Engineering (ICDE) , pages 181-192. IEEE, 2021.

WANG Luping, WANG Wei, and LI Bo. Cmfl: Mitigating communication overhead for federated learning. In 2019 IEEE 39th international conference on distributed computing systems (ICDCS) , pages 954-964. IEEE, 2019.

Yishay Mansour, Mehryar Mohri, Jae Ro, and Ananda Theertha Suresh. Three Approaches for Personalization with Applications to Federated Learning. arXiv preprint arXiv:2002.10619 , 2020.

Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-Efficient Learning of Deep Networks from Decentralized Data. In International Conference on Artificial Intelligence and Statistics (AISTATS) , 2017.

Sebastian Mika, Gunnar Ratsch, Jason Weston, Bernhard Scholkopf, and Klaus-Robert Mullers. Fisher discriminant analysis with kernels. In Neural networks for signal processing IX: Proceedings of the 1999 IEEE signal processing society workshop (cat. no. 98th8468) , pages 41-48. Ieee, 1999.

Viraaji Mothukuri, Reza M Parizi, Seyedamin Pouriyeh, Yan Huang, Ali Dehghantanha, and Gautam Srivastava. A Survey on Security and Privacy of Federated Learning. Future Generation Computer Systems , 115:619-640, 2021.

Dinh C Nguyen, Ming Ding, Pubudu N Pathirana, Aruna Seneviratne, Jun Li, and H Vincent Poor. Federated Learning for Internet of Things: A Comprehensive Survey. IEEE Communications Surveys & Tutorials , 23(3):1622-1658, 2021.

Yifan Niu and Weihong Deng. Federated Learning for Face Recognition With Gradient Correction. In AAAI Conference on Artificial Intelligence (AAAI) , 2022.

Jaehoon Oh, SangMook Kim, and Se-Young Yun. FedBABU: Toward Enhanced Representation for Federated Image Classification. In International Conference on Learning Representations (ICLR) , 2022.

Ethan Perez, Douwe Kiela, and Kyunghyun Cho. True Few-Shot Learning with Language Models. In International Conference on Advances in Neural Information Processing Systems (NeurIPS) , 2021.

Tao Qi, Fangzhao Wu, Chuhan Wu, Lingjuan Lyu, Tong Xu, Hao Liao, Zhongliang Yang, Yongfeng Huang, and Xing Xie. Fairvfl: A fair vertical federated learning framework with contrastive adversarial learning. Advances in Neural Information Processing Systems , 35:7852-7865, 2022.

Jiawei Ren, Cunjun Yu, Xiao Ma, Haiyu Zhao, Shuai Yi, et al. Balanced Meta-Softmax for Long-Tailed Visual Recognition. In International Conference on Advances in Neural Information Processing Systems (NeurIPS) , 2020.

Anna Rogers, Olga Kovaleva, and Anna Rumshisky. A Primer in BERTology: What We Know about How BERT Works. Transactions of the Association for Computational Linguistics , 8:842-866, 2021.

Aviv Shamsian, Aviv Navon, Ethan Fetaya, and Gal Chechik. Personalized Federated Learning using Hypernetworks. In International Conference on Machine Learning (ICML) , 2021.

Canh T Dinh, Nguyen Tran, and Tuan Dung Nguyen. Personalized Federated Learning with Moreau Envelopes. In International Conference on Advances in Neural Information Processing Systems (NeurIPS) , 2020.

Shauhin A Talesh. Data breach, privacy, and cyber insurance: How insurance companies act as 'compliance managers' for businesses. Law & Social Inquiry , 43(2):417-440, 2018.

Alysa Ziying Tan, Han Yu, Lizhen Cui, and Qiang Yang. Towards Personalized Federated Learning. IEEE Transactions on Neural Networks and Learning Systems , 2022. Early Access.

Yue Tan, Guodong Long, Jie Ma, Lu Liu, Tianyi Zhou, and Jing Jiang. Federated learning from pretrained models: A contrastive learning approach. Advances in Neural Information Processing Systems , 35: 19332-19344, 2022.

Welderufael B Tesfay, Peter Hofmann, Toru Nakamura, Shinsaku Kiyomoto, and Jetzabel Serna. I read but don't agree: Privacy policy benchmarking using machine learning and the eu gdpr. In Companion Proceedings of the The Web Conference 2018 , pages 163-166, 2018.

Michael Tuchler, Andrew C Singer, and Ralf Koetter. Minimum Mean Squared Error Equalization using A Priori Information. IEEE Transactions on Signal Processing , 50(3):673-683, 2002.

Laurens Van der Maaten and Geoffrey Hinton. Visualizing Data Using T-SNE. Journal of Machine Learning Research , 9(11), 2008.

Elena Voita and Ivan Titov. Information-Theoretic Probing with Minimum Description Length. In Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2020.

Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H. Vincent Poor. Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization. In International Conference on Advances in Neural Information Processing Systems (NeurIPS) , 2020.

Suge Wang, Deyu Li, Xiaolei Song, Yingjie Wei, and Hongxia Li. A feature selection method based on improved fisher's discriminant ratio for text sentiment classification. Expert Systems with Applications , 38 (7):8696-8702, 2011.

Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-MNIST: A Novel Image Dataset for Benchmarking Machine Learning Algorithms. arXiv preprint arXiv:1708.07747 , 2017.

Xiaoming Xue, Kai Zhang, Kay Chen Tan, Liang Feng, Jian Wang, Guodong Chen, Xinggang Zhao, Liming Zhang, and Jun Yao. Affine Transformation-Enhanced Multifactorial Optimization for Heterogeneous Problems. IEEE Transactions on Cybernetics , 52(7):6217-6231, 2020.

Qiang Yang, Yang Liu, Tianjian Chen, and Yongxin Tong. Federated Machine Learning: Concept and Applications. ACM Transactions on Intelligent Systems and Technology , 10(2):1-19, 2019.

Han Yu, Zelei Liu, Yang Liu, Tianjian Chen, Mingshu Cong, Xi Weng, Dusit Niyato, and Qiang Yang. A fairness-aware incentive scheme for federated learning. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society , pages 393-399, 2020.

Ming Zeng, Le T Nguyen, Bo Yu, Ole J Mengshoel, Jiang Zhu, Pang Wu, and Joy Zhang. Convolutional neural networks for human activity recognition using mobile sensors. In 6th international conference on mobile computing, applications and services , pages 197-205. IEEE, 2014.

Chao Zhang, Lei Zhang, and Jieping Ye. Generalization Bounds for Domain Adaptation. In International Conference on Advances in Neural Information Processing Systems (NeurIPS) , 2012.

Daokun Zhang, Jie Yin, Xingquan Zhu, and Chengqi Zhang. Network Representation Learning: A Survey. IEEE Transactions on Big Data , 6(1):3-28, 2018.

Jianqing Zhang, Yang Hua, Hao Wang, Tao Song, Zhengui Xue, Ruhui Ma, Jian Cao, and Haibing Guan. Gpfl: Simultaneously learning global and personalized feature information for personalized federated learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 5041-5051, 2023.

Jianqing Zhang, Yang Hua, Hao Wang, Tao Song, Zhengui Xue, Ruhui Ma, and Haibing Guan. Fedcp: Separating feature information for personalized federated learning via conditional policy. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , 2023.

Jianqing Zhang, Yang Hua, Hao Wang, Tao Song, Zhengui Xue, Ruhui Ma, and Haibing Guan. FedALA: Adaptive Local Aggregation for Personalized Federated Learning. In AAAI Conference on Artificial Intelligence (AAAI) , 2023.

Jianqing Zhang, Yang Hua, Hao Wang, Tao Song, Zhengui Xue, Ruhui Ma, and Haibing Guan. Fedala: Adaptive local aggregation for personalized federated learning. In Proceedings of the AAAI Conference on Artificial Intelligence , 2023.

Lin Zhang, Li Shen, Liang Ding, Dacheng Tao, and Ling-Yu Duan. Fine-Tuning Global Model Via Data-Free Knowledge Distillation for Non-IID Federated Learning. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2022.

Michael Zhang, Karan Sapra, Sanja Fidler, Serena Yeung, and Jose M Alvarez. Personalized Federated Learning with First Order Model Optimization. In International Conference on Learning Representations (ICLR) , 2020.

Sixin Zhang, Anna E Choromanska, and Yann LeCun. Deep Learning with Elastic Averaging SGD. International Conference on Advances in Neural Information Processing Systems (NeurIPS) , 2015.

Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-Level Convolutional Networks for Text Classification. In International Conference on Advances in Neural Information Processing Systems (NeurIPS) , 2015.

Yuchen Zhang, Tianle Liu, Mingsheng Long, and Michael Jordan. Bridging Theory and Algorithm for Domain Adaptation. In International Conference on Machine Learning (ICML) , 2019.

Zaixi Zhang, Xiaoyu Cao, Jinyuan Jia, and Neil Zhenqiang Gong. Fldetector: Defending federated learning against model poisoning attacks via detecting malicious clients. In Proceedings of ACM SIGKDD International Conference on Knowledge Discovery &amp; Data Mining , 2022.

Han Zhao, Shanghang Zhang, Guanhang Wu, José MF Moura, Joao P Costeira, and Geoffrey J Gordon. Adversarial Multiple Source Domain Adaptation. In International Conference on Advances in Neural Information Processing Systems (NeurIPS) , 2018.

Wennan Zhu, Peter Kairouz, Brendan McMahan, Haicheng Sun, and Wei Li. Federated Heavy Hitters Discovery with Differential Privacy. In International Conference on Artificial Intelligence and Statistics (AISTATS) , 2020.

Zhuangdi Zhu, Junyuan Hong, and Jiayu Zhou. Data-Free Knowledge Distillation for Heterogeneous Federated Learning. In International Conference on Machine Learning (ICML) , 2021.

We provide more details and results about our work in the appendices. Here are the contents:

Appendix A: The extended version of the Related Work section in the main body.

Appendix B: Proofs of Corollary 1 and Corollary 2.

Appendix C: More details about experimental settings.

Appendix D: Additional experiments ( e.g ., a real-world application).

Appendix E: Broader impacts of our proposed method.

Appendix F: Limitations of our proposed method.

Appendix G: Data distribution visualizations for different scenarios in our experiments.

A Related Work

As the number of users and sensors rapidly increases with massive growing services on the Internet, the privacy concerns about private data also draw increasing attention of researchers [37, 68, 71]. Then a new distributed machine learning paradigm, federated learning (FL), comes along with the privacy-preserving and collaborative learning abilities [37, 56, 79]. Although there are horizontal FL [46, 56, 79], vertical FL [53, 63, 79], federated transfer learning[15, 51], etc ., we focus on the popular horizontal FL and call it FL for short in this paper.

Traditional FL methods concentrate on learning a single global model among a server and clients, but it suffers an accuracy decrease under statistically heterogeneous scenarios, which are common scenarios in practice [47, 56, 67, 86]. Then, many FL methods propose learning personalized models (or modules) for each client besides learning the global model. These FL methods are specifically called personalized FL (pFL) methods [20, 22, 69].

A.1 Traditional Federated Learning

FL methods perform machine learning through iterative communication and computation on the server and clients. To begin with, we describe the FL procedure in one iteration based on FedAvg [56], which is a famous FL method and a basic framework for later FL methods. The FL procedure includes five stages: (1) A server selects a group of clients to join FL in this iteration and sends the current global model to them; (2) these clients receive the global model and initialize their local model by overwriting their local model with the parameters in the global model; (3) these clients train their local models on their own private local data, respectively; (4) these clients send the trained local models to the server; (5) the server receives client models and aggregates them through weighted averaging on model parameters to obtain a new global model.

Then, massive traditional FL methods are proposed in the literature to improve FedAvg regarding privacypreserving [49, 58, 95], accuracy [38, 45, 96], fairness [32, 80], overhead [28, 41, 54], etc . Here, we focus on the representative traditional FL methods that handle the heterogeneity issues in four categories: updatecorrection-based FL [25, 38, 60], regularization-based FL [1, 17, 40, 46], model-split-based FL [35, 45], and knowledge-distillation-based FL [27, 33, 88, 96].

Among update-correction-based FL methods, SCAFFOLD [38] witnesses the client-drift phenomenon of FedAvg under statistically heterogeneous scenarios due to local training and proposes correcting local update through control variates for each model parameter. Among regularization-based FL methods, FedProx [46] modifies the local objective on each client by adding a regularization term to keep local model parameters close to the global model during local training in an element-wise manner. Among model-split-based FL methods, MOON[45] observes that local training degenerates representation quality, so it adds a contrastive learning term to let the representations outputted by the local feature extractor be close to the ones outputted by the received global feature extractor given each input during local training. However, input-wise contrastive learning relies on biased local data domains, so MOON still suffers from representation bias. Among knowledge-distillationbased FL methods, FedGen [96] learns a generator on the server to produce additional representations, shares the generator among clients, and locally trains the classifier with the combination of the representations outputted by the local feature extractor and the additionally generated representations. In this way, FedGen can reduce the heterogeneity among clients with the augmented representations from the shared generator via knowledge distillation. However, it only considers the local-to-global knowledge transfer for the single global model learning and additionally brings communication and computation overhead for learning and transmitting the generator.

A.2 Personalized Federated Learning

Different from traditional FL, pFL additionally learns personalized models (or modules) besides the global model. In this paper, we consider pFL methods in four categories: meta-learning-based pFL [13, 22], regularizationbased pFL [47, 67], personalized-aggregation-based pFL [21, 52, 89], and model-split-based pFL [3, 14, 20, 61].

Meta-learning-based pFL. Meta-learning is a technique that trains deep neural networks (DNNs) on a given dataset for quickly adapting to other datasets with only a few steps of fine-tuning, e.g ., MAML [24]. By integrating MAML into FL, Per-FedAvg [22] updates the local models like MAML to capture the learning trends of each client and then aggregates the learning trends by averaging on the server. It obtains personalized models by fine-tuning the global model for each client. Similar to Per-FedAvg, FedMeta [13] also introduces MAMLon each client during training and fine-tuning the global model for evaluation. However, it is hard for these meta-learning-based pFL methods to find a consensus learning trend through averaging under statistically heterogeneous scenarios.

Regularization-based pFL. Like FedProx, pFedMe [67] and Ditto [47] also utilize the regularization technique, but they modify the objective for additional personalized model training rather than the one for local model training. In pFedMe and Ditto, each client owns two models: the local model that is trained for global model aggregation and the personalized model that is trained for personalization. Specifically, pFedMe regularizes the model parameters between the personalized model and the local model during training while Ditto regularizes the model parameters between the personalized model and the received global model. Besides, Ditto simply trains the local model similar to FedAvg while pFedMe trains the local model based on the personalized model. Although the local model is initialized by the global model, but the initialized local model gradually loses global information during local training. Thus, the personalized model in Ditto can be aware of more global information than the one in pFedMe. Both pFedMe and Ditto require additional memory space to store the personalized model and double the computation resources at least to train both the local model and the personalized model.

Personalized-aggregation-based pFL. These pFL methods adaptively aggregate the global model and local model according to the local data on each client, e.g ., APFL [21], or directly generate the personalized model using other client models through personalized aggregation on each client, e.g ., FedFomo [89] and APPLE [52]. Specifically, APFL aggregates the parameters in the global model and the local model with weighted averaging and adaptively updates the scalar weight based on the gradients. On each client, FedFomo generates the client-specific aggregating weights for the received client models through first-order approximation while APPLE adaptively learns these weights based on the local data. Both FedFomo and APPLE require multiple communication overhead than other FL methods, but FedFomo costs less computation overhead than APPLE attributed to approximation.

Model-split-based pFL. These pFL methods split a given model into a feature extractor and a classifier. They treat the feature extractor and the classifier differently. Concretely, FedPer [3] and FedRep [20] keep the classifier locally on each client. FedPer trains the feature extractor and the classifier together while FedRep first fine-tunes the classifier and then trains the feature extractor in each iteration. For FedPer and FedRep, the feature extractor intends to extract representations to cater to these personalized classifiers, thus reducing the generic representation quality. FedRoD [14] trains the local model with the balanced softmax (BSM) loss function [64] and simultaneously learns an additional personalized classifier for each client. However, the BSM loss is useless for missing labels on each client while label missing is a common situation in statistically heterogeneous scenarios [50, 86, 89]. Moreover, the uniform label distribution modified by the BSM cannot reflect the original distribution. The above pFL methods learn personalized models (or modules) in FL, but FedBABU [61] firstly trains the global feature extractor with the frozen classifier during the FL process, then it fine-tunes the global model on each client after FL to obtain personalized models. However, this post-FL fine-tuning is beyond the scope of FL. Almost all the FL methods have multiple fine-tuning variants, e.g ., fine-tuning the whole model or only a part of the model. Furthermore, training the feature extractor with the naive and randomly initialized classifier in FL has an uncontrollable risk due to randomness.

B Theoretical Derivations

B.1 Notations and Preliminaries

Following prior arts [21, 55, 69, 96], we consider a binary classification problem in FL here. Recall that X ⊂ R D is an input space, Z ⊂ R K is a representation space, and Y ⊂ { 0 , 1 } is a label space. Let F : X ↦→ Z be a representation function that maps from the input space to the representation space. We denote D := ⟨U , c ∗ ⟩ as a data domain where the distribution U ⊆ X and c ∗ : X ↦→ Y is a ground-truth labeling function. ˜ U is the induced distribution of U over the representation space Z under F [6], i.e ., ˜ U ⊆ Z , that satisfies

where B is a probability event. Given fixed but unknown U and c ∗ , the learning task on one domain is to choose a representation function F and a hypothesis class H ⊆ { h : Z ↦→ Y} to approximate the function c ∗ .

Then, we provide the definition and theorem from Ben-David et al. [6, 7], Blitzer et al. [9], Kifer et al. [39] under their assumptions:

Definition 1. If a space Z with ˜ U a and ˜ U b distributions over Z , let H be a hypothesis class on Z and Z h ⊆ Z be the subset with characteristic function h , the H -divergence between ˜ U a and ˜ U b is

where Z h = { z ∈ Z : h ( z ) = 1 } , h ∈ H .

Definition 1 implies that d H ( ˜ U a , ˜ U b ) = d H ( ˜ U b , ˜ U a ) .

Theorem 1. Consider a source domain D S and a target domain D T . Let D S = ⟨U S , c ∗ ⟩ and D T = ⟨U T , c ∗ ⟩ , where U S ⊆ X , U T ⊆ X , and c ∗ : X ↦→ Y is a ground-truth labeling function. Let H be a hypothesis space of VC dimension d and h : Z ↦→ Y , ∀ h ∈ H . Given a feature extraction function F : X ↦→ Z that shared between D S and D T , a random labeled sample of size m generated by applying F to a random sample from U S labeled according to c ∗ , then for every h ∈ H , with probability at least 1 -δ :

where L ˆ D S is the empirical loss on D S , e is the base of the natural logarithm, and d H ( · , · ) is the H -divergence between two distributions. ˜ U S and ˜ U T are the induced distributions of U S and U T under F , respectively, s.t. E z ∼ ˜ U S [ B ( z )] = E x ∼U S [ B ( F ( x ))] given a probability event B , and so for ˜ U T . ˜ U S ⊆ Z and ˜ U T ⊆ Z . λ := min h L D S ( h ) + L D T ( h ) denotes an oracle performance.

The traditional FL methods, which focus on enhancing the performance of a global model, regard local domains D i , i ∈ [ N ] and the virtual global domain D as the source domain and the target domain, respectively [96], which is called local-to-global knowledge transfer in this paper. In contrast, pFL methods that focus on improving the performance of personalized models regard D and D i , i ∈ [ N ] as the source domain and the target domain, respectively [21, 55, 69]. We call this kind of adaptation global-to-local knowledge transfer. The local-to-global knowledge transfer happens on the server while the global-to-local one occurs on the client.

B.2 Derivations of Corollary 1

As we focus on the local-to-global knowledge transfer on the server side , in the FL scenario, we can rewrite Theorem 1 to

Theorem 2. Consider a local data domain D i and a virtual global data domain D . Let D i = ⟨U i , c ∗ ⟩ and D = ⟨U , c ∗ ⟩ , where U i ⊆ X and U ⊆ X . Given a feature extraction function F : X ↦→ Z that shared between D i and D , a random labeled sample of size m generated by applying F to a random sample from U i labeled according to c ∗ , then for every h ∈ H , with probability at least 1 -δ :

where ˜ U i and ˜ U are the induced distributions of U i and U under F , respectively. ˜ U i ⊆ Z and ˜ U ⊆ Z . λ i := min h L D i ( h ) + L D ( h ) denotes an oracle performance.

Corollary 1. Consider a local data domain D i and a virtual global data domain D for client i and the server, respectively. Let D i = ⟨U i , c ∗ ⟩ and D = ⟨U , c ∗ ⟩ , where c ∗ : X ↦→ Y is a ground-truth labeling function. Let H be a hypothesis space of VC dimension d and h : Z ↦→ Y , ∀ h ∈ H . When using DBE , given a feature extraction function F g : X ↦→ Z that shared between D i and D , a random labeled sample of size m generated by applying F g to a random sample from U i labeled according to c ∗ , then for every h g ∈ H , with probability at least 1 -δ :

where L ˆ D i is the empirical loss on D i , e is the base of the natural logarithm, and d H ( · , · ) is the H -divergence between two distributions. λ i := min h g L D ( h g ) + L D i ( h g ) , ˜ U g i ⊆ Z , ˜ U g ⊆ Z , and d H ( ˜ U g i , ˜ U g ) ≤

d H ( ˜ U i , ˜ U ) . ˜ U g i and ˜ U g are the induced distributions of U i and U under F g , respectively. ˜ U i and ˜ U are the induced distributions of U i and U under F , respectively. F is the feature extraction function in the original FedAvg without DBE .

Proof. Computing d H ( · , · ) is identical to learning a classifier to achieve a minimum error of discriminating between points sampled from ˜ U and ˜ U ′ , i.e ., a binary domain classification problem [6, 7]. The more difficult

the domain classification problem is, the smaller d H ( · , · ) is. Unfortunately, computing the error of the optimal hyperplane classifier for arbitrary distributions is a well-known NP-hard problem [5, 6]. Thus, researchers approximate the error by learning a linear classifier for the binary domain classification [5, 9, 10]. Inspired by previous approaches [4, 43, 57], we consider using Linear Discriminant Analysis (LDA) for the binary domain classification. The discrimination ability of LDA is measured by the Fisher discriminant ratio (F1) [11, 30, 76]

where µ k ˜ U a and ( σ k ˜ U a ) 2 are the mean and variance of the values in the k th dimension over ˜ U a . The smaller the Fisher discriminant ratio is, the less discriminative the two domains are. Theorem 2 holds with every h ∈ H , so we omit PRBM here. MR (¯ z g i , ¯ z g ) forces the local domain to be close to the global domain in terms of the mean value at each feature dimension in the feature representation independently, therefore, ∀ k ∈ [ K ] ,

As the feature extractors share the same structure with identical parameter initialization and the feature representations are extracted from the same data domain D i ( D ) [18, 34], we assume that σ ˜ U g i = σ ˜ U i and σ ˜ U g = σ ˜ U . Thus, ∀ k ∈ [ K ] ,

As this inequality is satisfied in all dimensions including the dimension where the maximum value exists, so for the Fisher discriminant ratio, we have

The smaller the Fisher discriminant ratio is, the less discriminative the two domains are. The less discriminative the two domains are, the smaller d H ( · , · ) is. Thus, finally, we have

B.3 Derivations of Corollary 2

When we focus on the global-to-local knowledge transfer on the client side , in the FL scenario, we rewrite Theorem 1 as

Theorem 3. Consider a virtual global data domain D and a local data domain D i . Let D = ⟨U , c ∗ ⟩ and D i = ⟨U i , c ∗ ⟩ , where U ⊆ X and U i ⊆ X . Given a feature extraction function F : X ↦→ Z that shared between D and D i , a random labeled sample of size m generated by applying F to a random sample from U labeled according to c ∗ , then for every h ∈ H , with probability at least 1 -δ :

where ˜ U i and ˜ U are the induced distributions of U i and U under F , respectively. ˜ U i ⊆ Z and ˜ U ⊆ Z . λ i := min h L D ( h ) + L D i ( h ) denotes an oracle performance.

Corollary 2. Let D i , D , F g , and λ i defined as in Corollary 1. Given a translation transformation function PRBM : Z ↦→ Z that shared between D i and virtual D , a random labeled sample of size m generated by applying F ′ to a random sample from U i labeled according to c ∗ , F ′ = PRBM ◦ F g : X ↦→ Z , then for every h ′ ∈ H , with probability at least 1 -δ :

where d H ( ˜ U ′ , ˜ U ′ i ) = d H ( ˜ U g , ˜ U g i ) ≤ d H ( ˜ U , ˜ U i ) = d H ( ˜ U i , ˜ U ) . ˜ U ′ and ˜ U ′ i are the induced distributions of U and U i under F ′ , respectively.

Proof. PRBM is a translation transformation with parameters ¯ z p i , s.t. ∀ x i ∈ U i , z i = z g i + ¯ z p i , where z i = F ′ ( x i ) ∈ ˜ U ′ i and z g i = F g ( x i ) ∈ ˜ U g i . In other words, ∀ z g i ∈ ˜ U g i , ∃ ! z i ∈ ˜ U ′ i . Therefore, we have Pr ˜ U g i [ { z ∈ Z} ] = Pr ˜ U ′ i [ { z ∈ Z} ] and the same applies to the pair of ˜ U g and ˜ U ′ , i.e ., Pr ˜ U g [ { z ∈ Z} ] = Pr ˜ U ′ [ { z ∈ Z} ] . Then the subtraction of the probability on each side is also equal, i.e .,

∀ h ′ ∈ H , h g = h ′ ◦ PRBM ∈ H , so ∀ z a ∈ Z if h g ( z a ) = 1 , then h ′ ( z b ) = 1 , where z b = z a + ¯ z p i . Therefore, we have

where Z h g = { z ∈ Z : h g ( z ) = 1 } , h g ∈ H and Z h ′ = { z ∈ Z : h ′ ( z ) = 1 } , h ′ ∈ H . According to Definition 1, we have

C Detailed Settings

C.1 Implementation Details

We create the datasets for each client using six public datasets: Fashion-MNIST (FMNIST) 3 , Cifar100 4 , TinyImageNet 5 (100K images with 200 labels) and AG News 6 (a news classification dataset with four labels, more than 30K samples per label). The MDL is calculated through the public code 7 . We run all experiments on a machine with two Intel Xeon Gold 6140 CPUs (36 cores), 128G memory, eight NVIDIA 2080 Ti GPUs, and CentOS 7.8.

C.2 Hyperparameters of DBE

For hyperparameter tuning, we use grid search to find optimal hyperparameters, including κ and µ . Specifically, grid search is performed in the following search space:

κ : 0 , 0 . 001 , 0 . 01 , 0 . 1 , 1 , 5 , 10 , 20 , 50 , 100 , 200 , 500

µ : 0 , 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 , 0 . 5 , 0 . 6 , 0 . 7 , 0 . 8 , 0 . 9 , 1 . 0

In this paper, we set κ = 50 , µ = 1 . 0 for the 4-layer CNN, κ = 1 , µ = 0 . 1 for the ResNet-18, and κ = 0 . 1 , µ = 1 . 0 for the fastText. We only set different values for the hyperparameters κ and µ on different model architectures but use identical settings for one architecture on all datasets. Different models exhibit diverse capabilities in both feature extraction and classification. Given that our proposed DBE operates by integrating itself into a specific model, it is crucial to tune the parameters κ and µ to adapt to the feature extraction and classification abilities of different models.

As for the criteria for hyperparameter tuning , κ and µ require different tunning methods according to their functions. Specifically, µ is a momentum introduced along with the widely-used moving average technology in approximating statistics, so for the model architectures that originally contain statistics collection operations ( e.g ., the batch normalization layers in ResNet-18) one can set a relatively small value by tuning µ from 0 to 1 with a reasonable step size. For other model architectures, one can set a relatively large value for µ by tuning it from 1 to 0. The parameter κ is utilized to regulate the magnitude of the MSE loss in MR . However, different architectures generate feature representations with varying magnitudes, leading to differences in the magnitude of the MSE loss. Thus, we tune κ by aligning the magnitude of the MSE loss with the other loss term.

D Additional Experiments

D.1 Convergence

0

100

200

300

400

500

600

Iterations

0.5

1.0

1.5

2.0

Training Loss

0.2

0.4

0.6

0.8

1.0

Test Accuracy

Training Loss

Test Accuracy

Recall that our objective is

and its empirical version is min θ 1 ,..., θ N ∑ N i =1 n i n L ˆ D i ( θ i ) . Here, we visualize the value of ∑ N i =1 n i n L ˆ D i ( θ i ) and the corresponding test accuracy during the FL process. Figure 4 shows the convergence of FedAvg+ DBE and its stable training procedure. Besides, we also report the total iterations required for convergence on Tiny-ImageNet using ResNet-18 in Table 8. Based on the findings from Table 8, we observe that the utilization of DBE can yield a substantial reduction from 230 to 107 (more than 50%) in the total number of communication iterations needed for convergence, as compared to the original requirements of FedAvg.

D.2 Model-Splitting in ResNet-18

In the main body, we have shown that DBE improves the per-layer MDL and accuracy of FedAvg no matter how we split the 4-layer CNN. In Table 7, we report the per-layer MDL and accuracy when we consider model splitting in ResNet-18, a model deeper than the 4-layer CNN. No matter at which layer, we split ResNet-18 to form a feature extractor and a classifier, DBE can also reduce MDL and improve accuracy, showing its general applicability.

D.3 Distinguishable Representations

As our primary goal is to demonstrate the elimination of representation bias rather than improving discrimination in Figure 3 (main body), we present the t-SNE visualization for our largest dataset in experiments, Tiny-ImageNet (200 labels). Given that the 200 labels are distributed around the chromatic circle, adjacent labels are assigned

similar colors, resulting in Figure 3 (main body) being indistinguishable by the label. Using a dataset AG News with only four labels for t-SNE visualization can clearly show that the representations extracted by the global feature extractor are distinguishable in Figure 5.

D.4 A Practical Scenario with New Participants

To simulate a practical scenario with new clients joining for future FL, we perform method-specific local training for 10 epochs on new participants for warming up after their local models are initialized by the learned global model (or client models in FedFomo). Since FedAvg, Per-FedAvg, and FedBABU do not generate personalized models during the FL process, we fine-tune the entire global model on new clients for them to obtain test accuracy. Specifically, using Cifar100 and 4-layer CNN, we conduct FL on 80 old clients ( ρ = 0 . 5 or ρ = 0 . 1 ) and evaluate accuracy on 20 new joining clients after warming up. We utilize the data distribution depicted in Figure 9. According to Table 8, FedAvg shows excellent generalization ability with fine-tuning. However, DBE can still improve FedAvg by up to +6.68 with more stable performance for different ρ .

D.5 Large Local Epochs

We also conduct experiments with more local epochs in each iteration on FMNIST using the 4-layer CNN, as shown in Table 8. All the pFL methods perform similarly with the results for one local epoch, except for Per-FedAvg, which degenerates around 1.18 in accuracy (%).

D.6 Real-World Application

Wealso evaluate the performance of our DBE in a real-world application. Specifically, we apply DBE to the Internetof-Things (IoT) scenario on a popular Human Activity Recognition (HAR) dataset [2] with the HAR-CNN [81] model. HAR contains the sensor signal data collected from 30 users who perform six activities (WALKING, WALKING_UPSTAIRS, WALKING_DOWNSTAIRS, SITTING, STANDING, LAYING) wearing a smartphone on the waist. We show the results in Table 9, where FedAvg+ DBE still achieves superior performance.

E Broader Impacts

The representation bias and representation degeneration naturally exist in FL under statistically heterogeneous scenarios, which are derived from the inherently separated local data domains on individual clients. In the main body, we show the general applicability of our proposed DBE to representative FL methods. More than that, DBE can also be applied to other practical fields, such as the Internet of Things (IoT) [26, 31, 59] and digital health [15, 16]. Furthermore, introducing the view of knowledge transfer into FL sheds light on this field.

F Limitations

Although FL comes along for privacy-preserving and collaborative learning, it still suffers from privacy leakage issues with malicious clients [12, 93] or under attacks [23, 53]. We design DBE based on FL to improve generalization and personalization abilities, and we only modify the local training procedure without affecting the downloading, uploading, and aggregation processes. Thus, the DBE -equipped FL methods still suffer from the originally existing privacy issues like the original version of these FL methods when attacks happen. It requires future work to devise specific methods for privacy-preserving enhancement.

G Data Distribution Visualization

We illustrate the data distributions (including training and test data) in our experiments here.

         

























(a) FMNIST

         

















(b) Cifar100

         















(c) Tiny-ImageNet

         

























(a) FMNIST

         

















(b) Cifar100

         















(c) Tiny-ImageNet

          















(a)

β

= 0

.

01

         















(b)

β

= 0

.

5

         















(c)

β

= 5





























Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the significant challenge of statistical heterogeneity among clients in federated learning. To tackle this issue, the authors propose personalized federated learning, where each client can learn and apply its specific model parameters. Specifically, the authors introduce DBE (Domain Bias Embedding), which comprises two crucial modules. The first module is a client-specific embedding designed to model the domain bias, while the second module is a regularization term incorporated into the objective function to encourage consensus in global representation learning. The effectiveness of these proposed modules is analyzed and demonstrated through theoretical results from domain generalization/adaptation. Moreover, extensive experiments are conducted to verify DBE's efficacy in reducing domain bias and its complementarity with other personalized federated learning methods.\n\nSTRENGTHS:\n1. The paper is well-written, effectively conveying the core idea in a clear and understandable manner.\n2. The authors address the persistent issue of non-iidness in federated learning, which remains a significant challenge. The visualization of embeddings pre- versus post-local updates provides valuable motivation for studying and understanding this problem.\n3. The proposed DBE (Domain Bias Embedding) approach is both simple and effective, demonstrating strong performance from both theoretical and empirical perspectives.\n4. The paper's exploration of the complementary nature of DBE with existing personalized federated learning methods is particularly interesting. This aspect enhances the overall impact and relevance of the paper.\n\nWEAKNESSES:\n1. The introduction of the memory module requires improvement. While the module itself is relatively simple, the presentation of its implementation and functioning could be made clearer. It is challenging to understand the process and how to implement it based on the initial explanation.\n2. The analysis of generalization risk, although provided, may have limited practical utility when it comes to the practical application of DBE. Further clarification or practical implications would enhance the relevance of this analysis.\n3. The experimental evaluation appears to lack diversity in the considered tasks. Including evaluations on tasks such as graph learning would provide additional evidence and make the results more compelling.\n\nQUESTIONS:\n1. What's the difference between the numbers of layers that have memorized domain bias?\n2. Is it possible to show that, in the iid setting, the memory embedding is zero?\n\nLIMITATIONS:\nNone.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the problem of representation bias phenomenon lead by biased domains and proposes Domain Bias Eliminator (DBE), a framework that reduces domain discrepancy between server and client in representation space by Personalized Representation Bias Memory (PRBM) and Mean Regularization (MR). Experimental results show that DBE outperforming ten state-of-the-art methods in both generalization and personalization abilities.\n\nSTRENGTHS:\n-\tIntroduce the PRBM and MR to address the representation bias issue, and the effectiveness of these modules is validated by experimental results.\n-\tProvide theoretical analysis on generalization bounds of the global and local feature extractors.\n-\tThis method has good compatibility and can be combined with other models.\n\nWEAKNESSES:\n-\tIt would be nice to evaluate these methods on real-world distribution shifts.\n-\tThe introduction of MR increases the computational burden.\n-\tSome parts of the writing are not clear.\n\nQUESTIONS:\n1.\tI'm confused about the form of $\\bar{z}_i^p$, is it a trainable embedding vector (since it’s the personalized representation of client) or is it a model parameter (as Algorithm 1 claims)? Please elaborate on how $\\bar{z}_i^p$ is derived. \n2.\t$\\bar{z}^g$ is a consensus obtained during the initialization period, why is $\\bar{z}^g$ not updated during the training process? \n3.\tWhat’s the dimension of the global $z_i^g$ and personalized $\\bar{z}_i^p$ implemented in experiments?\n4.\tAs mentioned in Line 249 “We run three trials for all methods until empirical convergence on each task”, does this mean that all models are trained until they converge and the communication rounds are different for each model (which is rare in federal learning settings)? And what are the specific conditions for convergence?\n5.\tFigure 1 in Appendix only contains the results of FedAvg+DBE, how about other methods? It would be better to compare the results with other methods to judge the convergence rate.\n6.\tThe experimental results of the local model are missing in the paper\n7.\tIn Section E.3, the author evaluated the accuracy on 20 new clients. How is the data of these clients divided, and does the class of these data appear in the 80 old clients?\n\nLIMITATIONS:\nThe limitation of the proposed method has been discussed before.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nTo address the performance drop via heterogeneous data distribution, the proposed method -  Domain Bias Estimator (DBE) - decomposes image features into unbiased (global) and biased (personalized) representations.\n\nTo guide the feature extractor to obtain the unbiased representation, Mean Regularization (MR) is proposed which is a regularization term for local objective functions that enforce to reduce the gap between mean of representations of local data and a mean of representation of all data across clients. Meanwhile, each client has learnable biased representation which is added to unbiased representation from the feature extractor, and then feed to the local classifier. \n\nUnlike the previous works, by considering both unbiased and biased representation, the proposed method could improve bi-directional (local and global) knowledge transfer and could mitigate the performance drop via heterogeneity.\n\nSTRENGTHS:\n- The proposed method handles an important problem of non-IID FL.\n- The paper is clearly written.\n- The paper provides theoretical guarantees and comprehensive experimental results that show the efficiency of the proposed method for the given problem.\n- The proposed method is novel in terms of decomposing representation generated by feature extractor into unbiased and biased representations, and efficient since it requires not much additional computational cost.\n\nWEAKNESSES:\n- The paper does not discuss about privacy issues that could be emerged by collection of client-specific mean over local data (line 2,3 in Algorithm 1, Supple). Since the client-specific mean contains representations from all local data, it is potentially exposed to reconstruct identifiable information of local data or even data itself.\n\nQUESTIONS:\n- Collecting averaged features extracted from all local data seems to be exposed to privacy attacks. It would be helpful to give more evidence (previous works or experiments) that the ‘client-specific mean’ collection is privacy-preserving.\n- Figure 3 shows the role of client-variant (biased) representations and client-invariant (unbiased) representations. However, the representation is not separated according to labels. Are they distinguishable by label for representations extracted by the global feature extractor?\n\nLIMITATIONS:\nYes, the authors well describe the limitation of the proposed framework.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
The paper introduces Domain Bias Eliminator (DBE) to address the representation bias and representation degeneration issues commonly observed in federated learning under statistically heterogeneous scenarios. DBE utilizes a Personalized Representation Bias Memory (PRBM) to preserve representation bias and mean regularization to guide local feature extractors toward producing representations with a consensual global mean. The authors provided a thorough theoretical analysis and empirical studies to validate their method. The results indicated that DBE can enhance the performance of existing FL methods.

STRENGTHS:
1. This paper has solid theoretical analysis.
2. The proposed method is well-motivated.
3. Comparison with existing methods is sufficient.

WEAKNESSES:
1. The explanation on how DBE reduces the domain discrepancy between server and client in the representation space could be more detailed, as it forms a core part of the technique.

2. The authors could also consider exploring DBE's scalability to larger models, to understand the full potential of the method. The models used for experiments are all small, with only million-level parameters.

3. There are two main hyperparameters in the proposed method and it seems that the recommended choice is quite different for different datasets and models. It is difficult for practical scenarios to tune these hyperparameters.

QUESTIONS:
Is there any criteria or empirical methods to choose hyperparameters in this work?

LIMITATIONS:
The authors have addressed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07