Eliminating Domain Bias for Federated Learning in Representation Space
Authors: Jianqing Zhang
Yang Hua
Jian Cao
Hao Wang
Tao Song
Zhengui XUE
Ruhui Ma
Haibing Guan
Venue: NeurIPS 2023 poster (2023)
Keywords: Federated Learning
Personalized Federated Learning
Representation
Knowledge Transfer

Abstract: Recently, federated learning (FL) is popular for its privacy-preserving and collaborative learning abilities. However, under statistically heterogeneous scenarios, we observe that biased data domains on clients cause a representation bias phenomenon and further degenerate generic representations during local training, i.e., the representation degeneration phenomenon. To address these issues, we propose a general framework Domain Bias Eliminator (DBE) for FL. Our theoretical analysis reveals that DBE can promote bi-directional knowledge transfer between server and client, as it reduces the domain discrepancy between server and client in representation space. Besides, extensive experiments on four datasets show that DBE can greatly improve existing FL methods in both generalization and personalization abilities. The DBE-equipped FL method can outperform ten state-of-the-art personalized FL methods by a large margin. Our code is public at https://github.com/TsingZ0/DBE.

Eliminating Domain Bias for Federated Learning in Representation Space
Jianqing Zhang', Yang Hua², JJin Cao'; Hao Wang
Tao Song' Zhengui Xue', Ruhui Ma'*, Haibing Guan
SShanghai Jiao Tong University "Queen's University Belfast Louisiana State University (tsingz, cao-jian, songt333, zhenguixue, ruhuima, hbguan] edu.cn Y. HuHaaQuu..c ac.
ac.uk, uk, haowangolsu.
Abstract
Recently, federated learning (FL) is popular for its preseeeiir and collab- orative learning abilities. However, under statistically heterogeneous scenarios, we observe that biased data domains on clients cause representation bias phe- nomenon and further degenerate generic representations during local training, i.e., the representation degeneration phenomenon. To address these issues, we propose a general framework Domain Bias Eliminator (DBE) for FL. Our theo- retical analysis reveals that DBE can promote bi-directional knowledge transfer between server and client, as it reduces the domain discrepancy between server and client in representation space. Besides, extensive experiments four datasets show that DBE can greatly improve existing FL methods in both generalization and personalization abilities. The DBE-equipped FL method can outperform ten state-of-the-art personalized FL methods by a large margin. Our code is public at https: httpbbb..tt..t..///g/ com wowmTsingg0/DD
Introduction
As popular distributed machine learning paradigm with excellent privacy-preserving and collabora- tive learning abilities, federated learning (FL) trains models among clients with their private data kept locally (37 56 79]. Traditional FL (e.g.. the famous FedAvg (56)) learns one single global model in an iterative manner by locally training models on clients and aggregating client models on the server. However, it suffers an accuracy decrease under statistically heterogeneous scenarios, which are common scenarios in practice [47 [56 67] [6).
Due statistical heterogeneity, the data domain on each client is biased, which does not contain the data of all labels 337 45 67
befoons
5.2
69
79 [4] As
the received global model is locally trained on in- dividual clients' biased data domain, we observe that this model extracts biased (i.e., forming client-specific clusters) representations during local training. We call this phenomenon "rep- resentation bias" and visualize it in Figure [I Mcanwhile, by training the received global model with missing labels, the generic repre- sentation quality over all labels also decreases during local training [45]. Furthermore, we ob-
4.6
Layer Index
(a) Before.
(b) After.
(c) MDL change.
Figure 1: t-SNE 773 visualization and per-layer MDL (bits) for representations before/after local training in FedAvg. We use color and shape to dis- tinguish labels and clients respectively for t-SNE. A large MDL means low representation quality. Best viewed color and zoom-in.
Corresponniigg authors.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

serve that this "representation degeneration" phenomenon happens at every layer, as shown in Fig-
[re[[[c] We estimate the representation quality via minimum description length (MDL) [62 655 77)
metric independent of data and models, measuring the difficulty of classifying target labels according to given representations.
To tackle the statistical heterogeneity, unlike traditional FL that learns a single global model, person- alized FL (pFL) comes along by learning personalized models (or modules) for each client besides learning a global model among clients [20] 22 69 Typically, most of the existing pFL methods train a personalized classifie for each client B
20 61 but the feature extractor still extracts all the information from the biased local data domain, leading to representation bias and representation
degeneration during local training.
To address the representation bias and representation degeneration issues in FL., we propose a general framework Domain Bias Eliminator (DBE) for FL including two modules introduced as follows. Firstly, we detach the representation bias from original representations and preserve it in Personalized Representation Bias Memory (PRBM) on each client. Secondly, we devise a Mean Regularization (MR) that explicitly guides local feature extractors to extract representations with a consensual global mean during local training let the local feature extractor focus on the remaining unbiased information and improve the generic representation quality. In this way, we turn one level of representation between the feature extractor and the classifier on each client into two levels of representation with client-specifer bias and a climinate the conflict of extracting representations with Alient-specife biases for clients' requirements while extracting representations with diient-invarian features for the server's requirements in the same representation space. Our theoretical analysis shows that DBE can promote the bi-directional knowledge transfer between server and client with lower generalization bounds.
mean, respectively. Thus, we can
We conduct extensive experiments in computer vision (CV) and natural language processing (NLP) fields on various aspects to study the characteristics and effectiveness of DBE. In both generalization ability (measured by MDL) and personalization ability (measured by accuracy), DBE can promote the fundamental FedAvg as well as other representative FL methods. Furthermore, we compare the representative FDBAvg+DBE with ten sfate-of-the the-thhe-rt (SOTA) pFL methods various scenarios and show its superiority over these pFL methods. To sum up, our contributions are:
We observe the representation bias and per-layer representation degeneration phenomena during local training in the representative FL method FedAvg.
We propose framework DBE to memorize representation bias cach client to address the representation bias issue and explicitly guide local feature extractors to generate representa- tions with universal mean for higher generic representation quality.
We provide theoretical analysis and derive lower generalization bounds of the global and local feature extractors to show that DBE can facilitate bi-directional knowledge transfer between server and client in each iteration.
We show that DBE can improve other representative traditional FL methods including FedAvg at most -22.35% % in MDL (bits) and +32.30 in accuracy (%), respectively. Furthermore, FedAvg+DBE outperforms SOTA pFL methods by up to +11.36 in accuracy (%).
2 Related Work
Traditional FL methods that focus on improving accuracy under statistically heterogeneous scenarios
based on FedAvg including four categories: codlate-oorrection-be based FL [25 38 60) regularization-
based FL
17 40 46), model-split-based FL [35] 45), and Knowledge-distilaationn
FL [27
33 88, 96
For L methods, we consider four categories: meta-learning-
pFL [3 22).
regularization-basee pFL 47 67 gersonnlizeddaggreggtion
pFL [21 52] 87 [9], and model-
basid pFL 3
20
61 85 Due to limited space. we only introduce the FL methods that are
close to ours and leave the extended version of this section to Appendix A
Traditional FL methods. MOON 45 utilizes contrastive learning to correct the local training of each client, but this input-wise contrastive learning still relies on the biased local data domain, so still suffers from representation skew. Although FedGen [96] learns shared generator on the server
A model is split into feature extractor and classifier. They are sequentially jointed.

---

and reduces the heterogeneity among clients with the generated representations through knowledge distillation, it only considers the local-to-global knowledge for the single global model learning. On the other hand, FedGen additionally brings non-negligible communication and computation overhead for learning and transmitting the generator.
PFL methods. FedPer [3] and FedRep [20] keep the classifier locally, but the feature extractor still learns biased features without explicit guidance. Besides, their local feature extractors are trained to cater to personalized classifiers thus losing generality. FedRoD [4 reduces the discrepancy of local training tasks among clients by using a balanced softmax (BSM) loss function [64], but the BSM is useless for missing labels on each client while label missing is a common situation in statistically heterogeneous scenarios [50] 86 [9]. Moreover, the uniform label distribution modified by the BSM cannot reflect the original distribution. Differently. FedBABU [6] trains global feature extractor with a naive and frozen classifier, then fine-tunes the classifier for each client finally obtain personalized models. However, the post-Fl. fine-tuning study is beyond the FL. scope, almost all the FL methods have multiple fine-tuning variants, e.g., fine-tuning the whole model or only part of the model. Like FedAvg, FedBABU still locally extracts biased features during the FL process.
3 Notations and Preliminaries
3.1 Notations
In this work, we discuss the statistically heterogeneous scenario in typical multi-class classification tasks for FL., where N clients share the same model structure. Here, we denote notations following FedGen [96] and FedRep [20]. The client i, E [NV, has its own private data domain D,, where the data are sampled from D, All the clients collaborate train a global model g parameterized by without sharing their private local data.
Since focus representation learning in FL, regard g the sequential combination of feature extractor f that maps from the input space X' to a representation space Z, i.e. f X + Z parameterized by of and classifier h that maps from the representation space to the output space A i.e., h + A'' parameterized by 0h, Formally, we have g == h ʃ. == (0(:0)) X C RD and 2 C RRK A is the simplex over label space y IR. With any input E X, obtain the feature representation by z = fa: f1) E Z.
3.2 Traditional Federated Learning
With the collaboration of N clients, the objective of traditional FL, e.g., FedAvg [56. is to iteratively learn global model that minimizes its loss on each local data domain:
min (ED,
(0) B, 0), v.)l E (m.an)_-, [e(h(f(x;; 0");))))))))
(2)
where e: A' x + IR is a non-negative and convex loss function. Following FedGen, we assume that all clients share an identical loss function l and a virtual global data domain D, which is the union of all local domains: = U D; In practice, traditional FL methods 45
46
56 optimize Eq. 1
by ming E B.
n Cp, (0), where D, is an observable dataset, n, [D. is its size, and Σ n2.-
In each communication iteration, clients conduct local updates on their private data to train the global model O by minimizing their local loss. Formally, for client i, the objective during local training is ming LD, (0). The empirical version of LD, (0) is L. (0) : Sn;
1i=1 ((h(f((a))) )²:; 0h), Biy).
which is optimized by stochastic gradient descent (SGD) [6 90 in FedAvg.
Method
4.1 Problem Statement
pFL iteratively learns a personalized model or module for each client with the assistance of the global model parameters from the server. Our objective is (with a slight reuse of the notation LD.) EiE[NI[CD, (0:)),
min Oy....ON
(3)
3

---

where 0 is model consisting of global and personalized modules. The global modules are locally trained on clients and shared with the server for aggregation like traditional FL, but the personalized modules are preserved locally on clients. Following traditional FI., we empirically optimize Eq. (3)
by mino........ ON Di=1 D. (0,).
N
4.2 Personalized Representation Bias Memory (PRBM)
Due to the existence of statistical heterogeneity in FL, the local feature extractor intends to learn biased representations after being overwritten by the received global model parameters. To detach and store the representation bias locally, we propose personalized module PREM that memorizes representation bias for client i. Orig- inally, the feature representation 2; TRKK is directly fed into the predictor in Eq. (2) Instead, we consider zi as the combination of global x E IR K and personalized z IR i.e.,
X'
Feature Extractor
1
Classifier
A
(a) Local model (original).
X
Feature Extractor
Classifier
AV
2: :- 2, 2
(4)
(b) Local model (ours).
Figure 2: The illustration of the local model. We emphasize the parts that correspond to PRBM and MR with red and green, respectively.
We let the feature extractor output 2 instead of the original Z;, i.e., z := fæx;; 0f) and keep the trainable vector z locally. z. is specific among clients but identical for all the local data one client, it memorizes client-specifer mean. The original feature extractor is trained to capture the biased features for 2, Instead, with the personalized mean stored in z". the feature extractor turns to capture z with less biased feature information. We illustrate the difference between the original approach and our method in Figure (PRBM). Then, we define the local objective as ming, LD. (0:), where 0, = =00 1
(0)) == E( ( (<,, 0²) + 2,;; ).)).
(5)
From the view of transformation, we rewrite Eq. 5 to
LD. (O.) = E ( (((((((P(((())::; 0²); =;); 0²), 9.):).
(6)
where PRBM 3 2 a personalized translation transformation [8] parameterized by E.. Formally, PRBM(> ; x = + z V z Z. With PREM, we create an additional level of representation x besides the original level of representation 2, We call 2. and 2; as the first and second levels of representation, respectively. For the original local model (Figure ((a) we have z Zi.
4.3 Mean Regularization (MR)
Without explicit guidance, hard for the feature extractor and the trainable PRBM to distinguish between unbiased and biased information in representations automatically. Therefore, to let the feature extractor focus on the unbiased information and further separate 2 and z!, we propose an MR that explicitly guides the local feature extractor to generate z with the help of client-invariant mean, which opposite to the client-specific mean memorized z as shown in Figure 2 (MR). Specifically, we regularize the mean of 2 to the consensual global mean 29 at each feature dimension independently. We then modify Eq. (6) as
(O.) = ( [[(M(PBB([ 0 =(); v+)] + K. MR(2), E9),
N where xº = =E D. (xx 0z) We obtain the consensus E i= x during the initialization
period before FL (see Algorithm 7.. We measure the distance of æ and 2 by mean squared error (MSE) (72), and A is a hyperparameter to control the importance of MR for different tasks. Empirically,
1 n4
Co, (0.)
([(( 0²); 2();; 0²), wis) + K MR
(wiz; 01), )9),
(8)
which is also optimized by SGD following FedAvg.

---

In Eq. (8), the value of the MR term is obtained after calculating the empirical version of 2 2/ 1 Σj=ı f(z 0 over the entire local data, but the loss computing in SGD cannot see all the local data during one forward pass in one batch. In practice, inspired by the widely-used moving average 48 90 in approximating statistics over data among batches, in each batch, we obtain
(1 (4) 2 Lotd + P 24 ".eew'
where - "food and 24 Lnew are computed in the previous batch and current batch, respectively. μ is hyperparameter called momentum that controls the importance of the current batch. The feature extractor is updated continuously during local training but discontinuously between adjacent two iterations due to server aggregation. Thus, we only calculate 2 via Eq. (9) during local training and recalculate it in a new iteration without using its historical records. We consider the representative FedAvg+DBE as an example and show the entire learning process in Algorithm
Algorithm 1 The Learning Process in FedAvg+DBE
Input: N clients with their local data; initial parameters of:0 and 0 ry: local learning rate; & and μc hyperparameters; p. client joining ratio; E: local epochs: T: total communication iterations. Output: Global model parameters {0), 0h) and personalized model parameters (=!..... 2 }
> IInitialzaaionn Period
Server sends {010 0h.0) } to all clients to initialize their local models.
2: N clients train their local models without DBE for one epoch and collect client-specific mean ( } over their data domain.
3: Server generates a consensual global mean 2 through weighted averaging: 29 =>
4: Client initializes 2,.0,, Vi E [N].
b Federated Learning Period
5: for communication iteration I..... do
6: Server samples a client subset I based on p.
7: Server sends (0f.t-1, gh.c-1) to each client T
8:
for Client i E I in parallel do Initialize f and l with 0f and 0h.t-1 respectively.
9:
Obtain (0..e =p.. 0h.2) using SGD for ming. Lb, (0,) with 7,, n. and p for E epochs.
10:
Upload (ef.t. ehe1 to the server.
12: Server calculates n² = Σiez= and obtains
0f Eiez
n: 0..
n!
14:
0h.1 = Ni n. 0h1
15:
return {ØfTT, gh.T) and (2P.T
PP.
N
4.4 Improved Bi-directional Knowledge Transfer
In the FL field, prior methods draw connection from FL to domain adaptation for theoretical analysis
and consider a binary classification problem [21] 55 69 96]. The traditional FL methods, which
focus on enhancing the performance of global model, regard local domains D;, i e [N] and the virtual global domain I as the source domain and the target domain, respectively [96], which is called local-to-globa knowledge transfer in this paper. In contrast, pFL methods that focus on improving the performance of personalized models regard D and D., i E [N] as the source domain and the target domain, respectively [5 69]. We call this kind of adaptation as global-to-low knowledge transfer. The local-to-global knowledge transfer happens on the server while the global-to-local one occurs on the client. Please refer to Appendix B for details and proofs.
4.4.1 Local-To-Glohe Knowledge Transfer
Here, we consider the transfer after the server receives a client model. We guide the feature extractor to learn representations with global mean and gradually narrow the gap between the local domain and global domain at the first level of representation (i.e., 2²) to improve knowledge transfer:
Corollary 1. Consider a local data domain D and virtual global data domain 'D for client and the server, respectively, Let D, UU, c²) and D (U, E"), where c² X + y is ground-truth
5

---

labeling function. Let H. be hypothesis space of VC dimension d and h Z.y. H. When using DBE, given a feature extraction function F9 X 2 that shared between D; and D, a random labeled sample of size m generated by applying F to a random sample from Ui labeled according to c², then for every h E ",, with probability at least 6:
V
4
4 2em + log 8) (A, + dis
CD(h9) S L., +
[(dlog
m
d
where L. is the empirical loss on Di, e is the base of the natural logarithm, and (x((;:) is the H-divergence between two distributions. X minjo LD(ho) + LD. (h"), Ugf S Z, U9 S Z, and du du( 49) S dz(U,, Ue and U9 are the induced distributions of U; and U under F9, respectively. Ü, and ü are the induced distributions of and u under F, respectively. F the feature extraction function in the original FedAvg without DBE.
As shown in Figure 2 given any D, on client i, one can obtain 2; via FF in original FedAvg or obtain 2 via J in FedAvg+DBE. With dx (UL (KA S du, ( holds, we can achieve lower generalization bound in local-to- local-to-global knowledge transfer than traditional FL., thus training better global feature extractor to produce representations with higher quality over all labels. A small gap between the local domain and global domain in Z promotes the knowledge transfer from clients to the server
94
4.4.2 Global-To-Locas Knowledge Transfer
The llobal-t--local knowledge transfer focuses on the assistance role of the global model parameters for facilitating local training, i.e., the transfer ability from D to D,. After the client receives the global model and equips it with PRBM, for the second level of representation (i.e., 2,). we have
Corollary 2. Let D;, D, F9, and X; defined as in Corollary [ Given a translation transformation function PREM: : Z + Z that shared between DD and virtual D, rundom labeled sample of size m generated by applying F' to a random sample from U; labeled according to c², F = PRBM F9 X + Z, then for every h E H. with probability at least 8:
4
2em (dlog d
4 + log 5) du (Ū' (u) + Xi.
CD.(h)) S Cb(())+
V mm
where dz (Ú,, ū.) = dr ((,,))())) (i9) S du ii) = du(lii úl and ÜK are the induced distribu- tions uf and U, under F¹, respectively.
Given I; on client i, we can obtain z; via F' in FedAvg+DBE. h h1' o PRBM, so PREM does not influence the value of du(:,-) for the pair of he and h' (see Appendix B.3). then we have dz (K() dx (U/ i.)). The inequality dx (ū}) < dz( U.) shows that the information aggregated on the server can be more easily transferred to clients with our proposed DBE than FedAvg. We train PREM the local loss and preserve it locally, so the local feature extractors can generate representations suitable for clients' personalized tasks. According to Corollary [ and Corollary 2 adding DBE facilitates the bi-directional knowledge transfer in each iteration, gradually promoting global and local model learning as the number of iterations increases.
4.5 Negligible Additional Communication and Computation Overhead
DBE only modifies the local training, the downloading, uploading, and aggregation processes in FedAvg are unaffected. In DeB the communication overhead per iteration is the same as FedAvg but requires fewer iterations to converge (sce Appendix D). Moreover, PRBM only introduces K additional trainable parameters, and the MSE value in the parameterless MR computed for two representations of K dimension. K is the representation space dimension, typically smaller value than the dimension of data inputs or model parameters g 83 Thus, DBE introduces no additional communication overhead and negligible computation overhead for local training in any iteration.
4.6 Priivcc-PPreervvve Discussion
Compared to FedAvg, using DBE requires client to upload one client-specific mean zạ (one K- dimensional vector) to the server only once before FL, which solely captures the magnitude of the

---

mean value for each feature dimension within the context of the given datasets and models. Thanks to this particular characteristic, as shown in Section B.1.4] the performance of FedAvg+DBE can be minimally affected while enhancing its Gaussian noise with a zero mean to z during the initialization phase.
capabilities by introducing proper
Experiments
Datasets and models. Following prior FL. approaches 14 20 45
50
56 we use four public
datasets for classification problems in FI., including three CV datasets: Fashion-MNIST (FM- NIST) 177. Ci [42). and Tiny-ImageNet (100K images with 200 labels) [1). as well as one NLP dataset: AG News [91]. For three CV datasets, we adopt the popular 4-layer CNN by default following FedAvg, which contains two convolution layers (denoted by CONVI and CONV2) and two fully connected layers (denoted by FCI and FC2). Besides, we also use a larger model ResNet- 18 22 on Imageem. For AG News, we use the famous text classification model fastTexx 33).
Statistically heterogeneous scenarios. There are two widely used approaches to construct statistically heterogeneous scenarios on public datasets: the pathological setting [56] 66) and practical setting 45 [0]_ For the pathological setting, disjoint data with 2/10/20 labels for each client sampled from 10/100/200 labels on FMNIST/Cifar CitCCifaaOOOOTnn---AAa O0/Tiny-lmageNet with different data amounts. For the practical setting, we sample data from FMNIST, Cifar Tiny-ImageNNt, and AG News based on the Dirichlet distribution (50) (denoted by Dir(3)). Specifically, we allocate a qe.i (ge, ~ Dir(3)) proportion of samples with label client i, and we set 3 0.1/3 by default for CV/LLP tasks following previous FL approaches 50 75).
Following PFedMe and FedRoD, have 20 clients and set client participating ratio = 1 by default unless otherwise stated. We measure the generic representation quality across clients and evaluate the MDL 665 74] of representations over all class labels. To simulate the common FI. scenario where data only exists on clients, we split the data among each client into two parts: a training
Implementation Details.
(75% data) and a test set (25% data). Following pFedMe,
evaluate pFL methods by averaging the results of personalized models on the test set of each client and evaluate traditional FL methods by averaging the results of the global model on each client. Following FedAvg, we set the batch size to 10 and the number of local epochs to 1, so the number of local SGD steps is LTo_ for client We run three trials for all methods until empirical convergence on each task and report the mean value. For more details and results (e.g.. fine-tuning FedAvg on new participants and real-world application). please refer to Appendix D
5.1 Experimental Study for Adding DBE
5.1.1 How to Split the Model?
A model is split into a feature extractor and a classifier, but there are various ways for splitting, as each layer in deep neural network (DNN) outputs feature representation and feeds it into the next layer g 4m1 g [44]. We focus on inserting DBE between the feature extractor and the classifier, but which splitting way is the best for DBE? Here answer this question by comparing the results regarding MDL and accuracy when the model is split at each layer in the popular 4-layer CNN. We show the MDL of the representation z outputted by the prepositive layer of DBE (with underline here) and show MDL of z; for other layers. Low MDL and high accuracy indicate superior generalization ability and superior personalization ability, respectively.
In Table | the generic representation quality is improved at each layer for all splitting ways, which shows that matter how the model is split, DBE can enhance the generalization ability of the global feature extractor. Among these splitting ways, assigning all FC layers to the classifier, i.e CONV2-DBBE FCCI achieves almost the lowest MDL and highest accuracy. Meanwhile, FCI->DBE FCC2 can also achieve excellent performance with only 4.75 4.73% trainable parameters for DBE.
Since FedRep. FedRoD. and FedBABU choose the last FC layer as the classifier by default, we follow them for fair comparison and insert DBE before the last FC layer (e.g.. FC1-DBE FFC2). In T our FCI FC2 outperforms FedPer, FedRep, FedRoD, FedBABU, and FedAvg with lower MDL and higher accuracy. Since feature extractors FedPer and FedRep are locally trained to cater to personalized classifiers, they extract representations with low quality.
7

---

Table I: The MDL (bits. L) of layer-wise representations, test accuracy (%, 1), and the number of trainable parameters (4) in PRBM when adding DEE to FedAvg on Imay-em using 4-layer CNN the practical setting. We also show corresponding results for the close pFL. methods. For FedBABU, "[36.82]" indicates the test accuracy after post-Fl. fine-tuning for 10 local epochs.
MDL
Metries
Accuracy
Param.
CONVI-CONV2
CONV2-FCI
FCI-FC2
Logits
FedPer 3
5143
4574
3885
4169
FedRop 80
5102
4237
3922
4244
37.27
FedRoD 14
5063
4264
3783
3820
36.43
FedBABU 61
5083
4181
3948
3849
16.86 [36.82]
Original (FedAvg)
5081
4151
3844
3895
19,46
0
CONVI -COONV
4650 (-8.4 (88.88%)
4105 (-1.11%)
3679 (-4.29%)
3756 (-3.57%)
21.81 (+2.35)
28800
CONV2->DEE -FCI
4348 (-14.43%)
3716 (-10.48%)
3463 (-9.91 %)
3602 (-7.52%)
47.03 (+27.57)
10816
FCI DDBE -FC2
4608 ( (.9.31%)
3689 (-11.13%)
3625 (-5.70%)
3688 (-5.31%)
43.32 (#23.86)
512
5.1.2
Representation Bias Eliminated for the First Level of Representation
(a) FedAvg (B). (b) FedAvg (A). (c) +DBE (z B). (d) +DBE (z A). (e) +DBE (z,, B). (1) +DBE (z., A).
Figure 3: t-SNE visualization for representations on Tiny-ImageNet (200 labels). "B" and "A" denote "before local training" and "after local training". respectively. We use color and shape to distinguish labels and clients, respectively. Best viewed in color and zoom-in.
We visualize the feature representations using t-SNE 73 in Figure Compared to the representations outputted by the feature extractor in FedAvg, 2 in FedAvg+DBE is no longer biased the local data domain of each client after local training. With the personalized translation transformation PRBM, z; can fit the local domain of each client either before or after local training. According to Figure (bb) Figure 3(e) and Figure 3(f) 2; in FedAvg+DBE can fit the local domain better than FedAvg.
5.1.3 Ablation Study for DBE
Table 2: The MDL (bits, 1) and test accuracy (%, 1) when adding DBE to FedAvg Tiny-Imageh using 4-layer CNN and ResNet- 18 in the practical setting.
Models
4-layer CNN
R8
Components
FedAvg
+MR
+PRBM
+DBE
FedAvg
+MR
+PRBM
+DBE
MDL
3844
3643
3699
3625
3560
3460
3471
3454
Accuracy 19.46
22.21
26.70
43.32
19.45
20.85
38.27
42.98
We further study the contribution of MR and PRBM in terms of generalization and personalization abilities by applying only one of them to FedAvg. From Table we find that for 4-layer CNN and ResNet-18, +DBE gives larger improvement in both MDL and accuracy than just using MR or PRBM, which suggests that MR and PRBM can boost each other in bi-directional knowledge transfer. The contribution of MR greater than that of PREM in improving the generalization ability in MDL, while +PRBM gains more accuracy improvement for personalization ability than MR.
5.1.4 Privacy-Presevvngg Ability
Following FedPAC [70], we add Gaussian noise client-specifit means zy...., Eg with a scale parameter (s) for the noise distribution and perturbation coefficient (q) for the noise. Adding the unbiased noise sampled from one distribution is beneficial for representation bias elimination and can further improve the performance of DBE to some extent, as shown in Table 3 Besides, adding too
much noise can also bring an accuracy decrease. However, setting & = 0.05 and q = 0.2 is sufficient to ensure privacy protection according to FedPCL.
8

---

Table 3: The test accuracy (%. 1) using DedAvg+DBE on TINY in the practical setting with noise.
0.2
0.05
Original
0.05
0.5
0.1
q = 0.5
q = 0.8
q -= 0.9
43.32
44.10
44.15
43.78
36.27
43.81
44.45
43.30
41.75
5.1.5
DBE Improves Other Traditional Federated Learning Methods
Table
The MDL (bits, 4) and test accuracy (%, f) before and after adding DBE to traditional FL methods on C00 Tiny-ImageNet, and AG News in the practical setting. TINY and TINY* represent using 4-layer CNN and ResNet-18 on Tiny-ImageNet, respectively.
Metries
MDL
Accuracy
Datasets
Ci
TINY
TINY
AG News
Cifar100
TINY
TINY
AG News
SCAFFOLD
38
1499
3661
3394
33.08
23.26
24.90
88.13
FedProx
46
1523
3701
3570
2092
31.99
19.37
19.27
87.21
MOON
45
1516
3696
3536
1836
32.37
19.68
19.02
84. 14
FedGen
96
1506
3675
3551
1414
30.96
19.39
18.53
89.86
SCAFFOLD+DBE
1434
3549
3370
1743
63.61
45.55
45.09
96.73
FedProx+DBE
1439
3587
3490
1689
63.22
42.28
41.45
96.62
MOON+DBE
1432
3580
3461
1683
63.26
43.43
41.10
96.68
FedGen+DBE 1426
3563
3488
1098
63.26
42.54
41.87
97.16
large number of FL methods design algorithms based on the famous FedAvg we describe DBE based FedAvg for example, DBE can also be applied to other traditional FL methods to improve their generalization and personalization abilities. Here, we apply DBE to another four representative traditional FL methods: SCAFFOLD (38), FedProx [46]. MOON [45], and FedGen [96]. They belong to four categories: update-correction based FL, regularization-base mooel--plit-based FL, and
69
Although
FL, FL, respectively. In Table 4 DBE promotes
traditional FL methods by at most -22.35 % in MDL (bits) and +32.30 accuracy (%), respectively. Based the results of Table 2 and Table on Tiny-Iman Imneeem,,NNe,, FedAvg+DBE achieves lower MDL and higher accuracy than close methods MOON and FedGen.
5.2 Comparison with Personalized Federated Learning Methods
5.2.1
Personalization Ability on Various Datasets
Table 5:
The test accuracy (%, 1) of pFL. methods in two statistically heterogeneous settings. Cifar1001 represents the experiment with 100 clients and joining ratio p = 0.5 on Cifar100.
Settings
Pathological setting
Practical setting
FMNIST
Cifar100
TINY
FMNIST
Cifar
Cifar100
TINY
News
Per-FedAAg 22
99.18
56,80
28.06
95.10
44.28
38.28
25.07
21.81
87.08
pFedMe 66
99.35
58. 20
27.71
97.25
47.34
31.13
26.93
33.44
87.08
Ditto
99.44
67.23
39,90
97.47
52.87
39.01
32.15
35.92
FedPer 3
99.47
63.53
39.80
97.44
49.63
41.21
33.84
38.45
91.85
FedRep
99.56
67.56
40.85
97.56
52.39
41.51
37.27
39,95
92.25
FedRoD 1
99.52
62.30
37.95
97.52
50.94
48.56
36.43
37.99
92.16
FedBABU 66
99.41
66.85
40.72
97.46
55.02
52.07
36.82
34.50
95.86
APEL 21
99.41
64.26
36.47
97.25
46.74
39.47
34.86
35.81
89.37
FedFomo (89
99.46
62.49
36.55
97.21
45.39
37.59
26.33
26.84
91.20
APPLE
99.30
36.22
53,22
35.04
39.93
84.10
FodAvg
80.41
25.98
14.20
85.85
31.89
28.81
19.46
19.45
87.12
FoBAvg+DBE
99.74
73.38
42.89
97.69
64.39
63.43
43.32
42.98
96.87
To further show the superiority of the DBE-equipped traditional FL methods to existing PFL methods, we compare the representative FedAvg+DBE with ten SOTA pFL methods, shown in Table Note that APPLE designed for cross-silo scenarios and assumes = For Per-FedAvg and FedBABU, show the test accuracy after post-FL fine-tuning, FedAvg+DBE improves FedAvg most +47.40

---

on Cifar 100 in the pathological setting and outperforms the best SOTA pFL methods by up to +11.36 on Cifar100 including the fie-tuning-biat based methods that require additional post-FL effort.
5.2.2 Personalization Ability Under Various Heterogeneous Degrees
Following prior methods 45, 50]. we also evaluate FedAvg+DBE with different 3 on Tiny-ImageNet using 4-layer CNN to study the influence heterogeneity, as shown Table Most pFL methods specifically designed for extremely heterogeneous scenarios and achieve high accuracy at 3 = 0.01, but some of them cannot maintain the advantage compared to FedAvg in moderate scenarios. However, DBB automatically adapt to all these scenarios without tuning.
Table 6: The test accuracy (%, 1) and computation overhead (4) of pFL. methods.
Items
Heterogeneity
pFL+XR
Overhead
0.01
0.5
3
Accuracy
Improvement
Total time
Time/iteration
Per-FedAvg 22
39.39
21.14
12.08
121 min
3.56 min
PFedMe 667
41.45
17.48
4.03
1157 min
10.24 min
Ditto 47
50.62
18.98
21.79
42.82
10.67
318 min
11.78 min
FedPer
51.83
17.31
9.61
41.78
7.94
83 min
1.92
FedRep
55.43
16.74
8.04
41.28
4.01
min
4.09 min
FedRoD [4
49.17
23.23
16.71
42.74
6.31
87 min
1.74 min
FudBABU 61
53.97
23.08
15.42
38.17
1.35
811 min
1.58 min
APFI. 21
49.96
23.31
16.12
39.22
4,36
156 min
2.74 min
FedFomo
89
46.36
11.59
14.86
29.51
3.18
193 min
2.72 min
APPLE 52
47.89
24.24
17.79
132 min
2.93 min
FedAvg
15.70
21.14
21.71
365 min
1.59 min
DedAvg+DBE
57.52
32.61
25.55
171 min
1.60 min
5.2.3 MR Improves Personalized Federated Learning Methods
Since pFL methods already create personalized models modules in their specific ways, applying personalized PREM to the local model might against their philosophy. To prevent this, we only apply the MR to pFL. methods. Besides, the local training schemes (e.g., meta-learning) in Per-FedAvg, PFedMe, and APPLE are different from the simple SGD in FedAvg, which requires modification of the calculation in MR, the local-to-global knowledge transfer between server and client. Therefore, pFL methods can benefit more from better global model achieving higher accuracy on Tiny-Imagen with the 4-layer CNN. as shown in Table 6 However, their MR-equipped variants perform worse than FedAvg+DBE (Table 5 TINY) since the representation bias still exists without using PRBM.
do not apply MR to them. According Corollary [
MR can promote
5.2.4 Computation Overhead
We evaluate FedAvg+DBE in total time and time per iteration on Tiny-ImageN's using ResNet- 18, as shown in Table The evaluation task for one method monopolizes one identical machine. FedAvg, FedBABU, and FedAvg+DBE cost almost the same and have the lowest time per iteration among these methods, but FeBAvg+DBE requires less total time than FedAvg and FedBABU, Note that the fine-tuning time for FedBABU is not included in Table Since pFedMe and Ditto train additional personalized model on each client, they cost plenty of time per itcration.
Conclusion
Due the naturally existing statistical heterogeneity and the biased local data domains on each client, FL suffers from representation bias and representation degeneration problems. To improve the generalization and personalization abilities for FL, we propose general framework DBE including two modules PRBM and MR, with theoretical guarantee. Our DBE can promote the bi-directional knowledge transfer each iteration, thus improving both generalization and personalization abilities. Besides, we conduct extensive experiments to show the general applicability of DBE to existing FL methods and the superiority of the representative FedAvg+DBE to ten SOTA pFL. methods in various scenarios.
10

---

Acknowledgments and Disclosure of Funding
This work was partially supported by the Program of Technology Innovation of the Science and Technology Commission of Shanghai Municipality (Granted No. 21511 104700 and 22DZI 100103). This work was also supported in part by the Shanghai Key Laboratory of Scalable Computing and Systems, National Key R&D Program of China (202YFB44001002) Internet of Things special subject program, China Institute of IoT (Wuxi), Wuxi IoT Innovation Promotion Center (2022SP-T13-CH Industry-university- reeseacc Cooperation Funding Project from the Eighth Research Institute in China Aerospace Science and Technology Corporation (Shanghai) (USCAST2022-17) Intel Corporation (UFunding 12679), and the cooperation project from Ant Group ("Wasm-enabled Managed language in security restricted scenarios").
References
[1] Durmus Alp Emre Acar, Yue Zhao, Ramon Matas, Matthew Mattina, Paul Whatmough, and Venkatesh Saligrama. Federated Learning Based on Dynamic Regularization. In International Conference on Learning Representations (ICLR), 2021.
[2] Davide Anguita, Alessandro Ghio, Luca Oneto, Xavier Parra, and Jorge L Reyes-Ortiz. Human activity recognition smartphones using multiclass pardware-friend]] support vector machine. In Ambient Assisted Living and Home Care: 4th International Workshop, IWAAL 2012, Vitoria- Spain, December 3-5, 2012. Proceedings 4, pages 216 223. Springer, 2012.
[3] Manoj Ghuhan Arivazhagan, Vinay Aggarwal, Aaditya Kumar Singh, and Sunav Choudhary. Federated Learning with Personalization Layers. arXiv preprint arXiv: 1912.0088, 2019.
[4] Suresh Balakrishnama and Aravind Ganapathiraju. Linear discriminant analysis-a brief tutorial. Institute for Signal and information Processing, 18( 18(1998):1-8, 1998.
[5] Shai Ben-David, Nadav Eiron, and Philip M Long. On the difficulty of approximately maximizing agreements. Journal of Computer and System Sciences, 66(3):49--514, 2003.
[6] Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Pereira. Analysis of Representations for Domain Adaptation. In International Conference on Advances in Neural Information Processing Systems (NeurlPS), 2006.
[7] Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. A theory of learning from different domains. Machine learning, 79:151-175, 2010.
[8] Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation Learning: A Review and New Perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(8):1998-1828, 2013.
[9] John Blitzer. Koby Crammer, Alex Kulesza, Fernando Pereira. and Jennifer Wortman. Learning bounds for domain adaptation. Advances in neural information procesxing systems, 20, 2007.
[10] John Blitzer, Mark Dredze, and Fernando Pereira. Biographies, bollywood, boom-boo boxe- and blenders: Domain adaptation for sentiment classification. In Proceedings of the 45th annual meeting of the association of computational linguistics, pages 440-447, 2007.
[1I] nosé-Ramón Cano. Analysis of data complexity measures for classification, Expert systems with applica- tions, 40(44)0)))))))) 40(12):480-44811 2013.
[12] Xiaoyu Cao, Jinyuan Jia, and Neil Zhenqiang Gong. Provably secure federated learning against malicious clients. In AAAÍ Conference on Artificial Intelligence (AAAI), 2021.
[13] Fei Chen, Mi Luo, Zhenhua Dong, Zhenguo Li, and Xiugiang He. Federated Meta-Leamnn With Fast Convergence and Efficient Communication. arXiv preprint arXiv: 1802.07876 2018
[14] Hong- You Chen and Wei-Lan Chao. On Bridging Generic and Personalized Fedcrated Learning for Image Classification. In International Conference on Learning Representations (ICLR), 2021.
[15] Yiqiang Chen, Xin Qin, Jindong Wang, Chaohui Yu, and Wen Gao. Fedhealth: A federated transfer learning framework for wearable healthcare. IEEE Intelligent Systems, 35(4):83-93, 2020.
[16] Yiqiang Chen, Wang Lu, Xin Qin, Jindong Wang. and Xing Xie. MetaFed: Federated Leaming Among Fed- erations With Cyclic Knowledge Distillation for Personalized Healtheare. arXiv preprint 28876.2206.0887 2022.
11

---

[17] Anda Cheng, Peisong Wang, Xi Sheryl Zhang, and Jian Cheng. Differentially Private Federated Learning with Local Regularization and Sparsification. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2022.
[18] Lenaic Chizat and Francis Bach, On the global convergence of gradient descent for over-parameterized models using optimal transport. Advances in neural information processing systems, 31, 2018.
[19] Patryk Chrabaszez, Ilya Loshchilov, and Frank Hutter. A Downsampled Variant of Imagenet as an Alternative the Cifar Datasets. arXiv preprint arXiv: 1xXiv:/700 1707,08819, 2017.
[20] Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai. Exploiting Shared Representations for Personalized Federated Learning. In International Conference on Machine Learning (ICML), 2021.
[21] Yuyang Deng, Mohammad Mahdi Kamani, and Mehrdad Mahdavi, Adaptive Personalized Federated Learning. arXiv preprint arXiv:2003. 13461, 2020.
[22] Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Personalized Federated Learning with Theoretical Guarantees: A Model-Agnostic Meta- Meta-Learning Approach. In International Conference on Advances in Neural Information Processing Systems (NeurIPS), 2020.
[23] Minghong Fang, Xiaoyu Cao. Jinyuan Jia, and Neil Gong, Local Model Poisoning Attacks to Byzantine- Robust Federated Learning. In USENIX Security, 2020.
[24] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks. In International Conference on Machine Learning (ICML), 2017.
[25] Liang Gao, Huazhu Fu, Li Yingwen Chen, Ming Xu, and Cheng-Zhong Xu. FedDC: Federated Learning With Non-IID Data Via Local Drift Decoupling and Correction. In IEFE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[26] Bimal Ghimire and Danda B Rawat. Recent Advances on Federated Learning for Cybersecurity and Cybersecurity for Federated Learning for Internet of Things. IEEE Internet of Things Journal, 09(11): 8229 8249, 2022.
[27] Xuan Gong, Abhishek Sharma, Srikrishna Karanam, Ziyan Wu, Terrence Chen, David Doermann, and Arun Innanje. Preserving Privacy in Federated Learning With Ensemble Cross-Domain Knowledge Distillation. In AAAI Conference on Artificial Intelligence (AAAI), 2022.
[28] Meng Hao, Hongwei Li, Xizhao Luo, Guowen Xu, Haomiao Yang, and Sen Liu. Efficient and Privacy- Enhanced Federated Learning for Industrial Artificial Intelligence. IEEE Transactions on Industrial Informatics, 16(10):6532 6542, 2019.
[29] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[30] Tin Kam Ho and Mitra Basu. Complexity measures of supervised classification problems. IEEE transuc- tions on pattern analysis and muchine intelligence, 24(3):289-300, 2002.
[31] Rui Hu, Yuanxiong Guo, Hongning Li, Qingqi Pei, and Yanmin Gong. Personalized federated learning with differential privacy. IEEE Internet of Things Journal, 7(10)9550-9539, 2020.
[32] Tiansheng Huang, Weiwei Lin, Wentai Wu, Ligang He, Keqin Li, and Albert Y Zomaya. An efficiency- boosting client selection scheme for federated learning with fairness guarantee. IEFF Transactions on Parallel and Distributed Systems, 32((): 1552-1564, 2020.
[33] Wenke Huang, Mang Ye, and Bo Du. Learn From Others and Be Yourself in Heterogeneous Federated Learning. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[34] Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kemel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
[35] Mcirui Jiang, Zirui Wang, and Qi Dou. Harmoft: Harmonizing Local and Global Drifts in Federated Learning on Heterogeneous Medical Images. In AAAI Conference on Artificial Intelligence (AAAI), 2022.
[36] Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. Bag of Tricks for Efficient Text Classification. In Conference of the European Chapter of the Association for Computational Linguistics (EACL), 2017.
12

---

[37] Peter Kairouz, H Brendan McMahan, Brendan Avent. Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji. Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, al. Advances and Open Problems Federated Learning. arXiv preprint urXiv:1912.04991 2019.
[38] Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Schastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic Controlled Averaging for Federated Learning. In In- ternational Conference on Machine Learning (ICML), 2020.
[39] Daniel Kifer, Shai Ben-David, and Johannes Gehrke. Detecting change in data streams. In VLDB, pages 180 191, 2004.
[40] Jinkyu Kim, Geeho Kim, and Bohyung Han. Multi-Level Branched Regularization for Federated Learning. In International Conference Machine Learning (ICML), 2022.
[41] Jakub Koncõný, H Brendan McMahan, Felix X Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. Federated Learning: Strategies for Improving Communication Efficiency. arXiv preprint arXiv:161000549, 2016.
[42] Alex Krizhevsky and Hinton Geoffrey. Learning Multiple Layers of Features From Tiny Images. Technical Report, 2009.
[43] Max Kuhn, Kjell Johnson, Max Kuhn, and Kjell Johnson. Discriminant analysis and other linear classifica- tion models. Applied predictive modeling, pages 275-328, 2013.
[44] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep Leaminng Nature, 521(7553):436 444, 2015.
[45] Qinhin Li, Bingsheng He, and Dawn Song. Monel-Contrastive Federated Learning. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[46] Tian Li. Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated Optimization Heterogeneous Networks. Conference on Machine Learning and Systems (MLSys). 2020.
[47] Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. Ditto: Fair and Robust Federated Learning Through Personalization. In International Conference on Machine Learning (ICML), 2021.
[48] Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. On the Convergence of FodAvg on Non-IID Data. In International Conference on Learning Representations (ICLR), 2020.
[49] Zengpeng Li, Vishal Sharma. and Saraju P Mohanty. Preserving Data Privacy via Federated Learning: Challenges and Solutions. IEEE Consumer Electronics Magazine, 9(3):8 16, 2020.
[50] Tao Lin, Lingjing Kong, Sebastian U Stich, and Martin Jaggi. Ensemble Distillation for Robust Model Fusion in Federated Learning. In International Conference on Advances in Neural Information Processing Systems (NeurlPS), 2020.
[51] Yang Liu, Yan Kang, Chaoping Xing. Tianjian Chen, and Qiang Yang. A secure federated transfer lcarning framework. IEEE Intelligent Systems, 35(4):70-88 82, 2020.
[52] Jun Luo and Shandong Wu, Adapt to Adaptation: Learning Personalization for Cross-Silo Federated Learning. In International Joint Conference on Artificial Intelligence (IJCAI), 2022.
[53] Xinjian Luo, Yuncheng Wu, Xiaokui Xiao, and Beng Chin Ooi. Feature inference attack on model predictions in vertical federated learning. In 202/ IEEE 37th International Conference on Data Engineering (ICDE), pages 181-192. IEFF, 2021.
[54] WANG Luping, WANG Wei, and LI Bo. Cmfl: Mitigating communication overhead for federated lcarning. In 2019 IEEE 39th international conference on distributed computing systems (ICDCS), pages 954-964. IEEE, 2019.
[55] Yishay Mansour, Mehryar Mohri, Jae Ro, and Ananda Theertha Suresh. Three Approaches for Personal- ization with Applications to Federated Learning. arXiv preprint arXiv:2002. 10679, 2020.
[56] Brendan McMahan, Fider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-EFfa Efffcccin learning of Deep Networks from Decentralized Data. In International Confer- ence Artificcal Intelligence and Statistics (AISTATS), 2017.
[57] Sebastian Mika, Gunnar Ratsch, Jason Weston, Bernhard Scholkopf. and Robus-Robert Mullers. Fisher discriminant analysis with kernels. In Neural networks for signal processing IX: Proceedings of the 1999 IEEE signal processing society workshop (cat. no. 98th8468), pages 41-48, Ieee, 1999.
13

---

[58] Viraaji Mothukuri, Reza M Parizi, Seyedamin Pouriyeh, Yan Huang, Ali Dehghantanha, and Gautam Srivastava. A Survey Security and Privacy of Federated Leamning Future Generation Computer Systems, 15:6619644,, 2021.
[59] Dinh C Nguyen, Ming Ding, Pubudu Pathirana, Aruna Seneviratne, Jun Li, and Vincent Poor. Federated Learning for Internet Things: Comprehensive Survey, IEFE Communications Surveys Tutorials, 23(3):1622-1658, 2021.
[60] Yifan Niu and Weihong Deng. Federated Learning for Face Recognition With Gradient Correction. In AAAI Conference on Artificial Intelligence (AAAI), 2022.
[61] Jaehoon Oh, Sang Kim, and Se- Young Yun. FedBABU: Toward Enhanced Representation for Federated Image Classification, In International Conference on Learning Representations (ICI.R), 2022.
[62] Ethan Perez, Douwe Kiela, and Kyunghyun Cho. True Few-Shot Learning with Language Models. In International Conference on Advances Neural Information Processing Systems (NeurIPS), 2021.
[63] Tao Qi. Fangzhao Wu, Chuhan Wu, Lingjuan Lyu. Tong Xu, Hao Liao, Zhongliang Yang. Yongfeng Huang, and Xing Xie. Fairvil: fair vertical federated learning framework with contrastive adversarial learning. Advances in Neural Information Processing Systems, 35:7852-7865, 2022.
[64] Jiawei Ren, Cunjun Yu, Xiao Ma, Haiyu Zhao, Shuai Yi, et al. Balanced Meta-Softmax for Long-Tailed Tailed Visual Recognition. In International Conference on Advances in Neural Information Processing Systems (NeurIPS), 2020.
[65] Anna Rogers, Olga Kovaleva, and Anna Rumshisky. A Primer in BERTology: What We Know about How BERT Works. Transactions of the Association for Computational Linguistics, 8:842-866, 2021.
[66] Aviv Shamsian, Aviv Navon, Ethan Fetaya, and Gal Chechik. Personalized Federated Learning using Hypernetworss In International Conference on Machine Learning (ICMI.), 2021.
[67] Canh T Dinh, Nguyen Tran, and Tuan Dung Nguyen. Personalized Federated Learning with Moreau Envelopes. International Conference on Advances in Neural Information Procexsing Systems (NeurIPS). 2020.
[68] Shauhin A Talesh. Data breach, privacy, and cyber insurance: How insurance companies act as "compliance managers" for businesses. Law & Social Inquiry, 43(2):417-440, 2018.
[69] Alysa Ziying Tan, Han Yu, Lizhen Cui, and Qiang Yang. Towards Personalized Federated Learning. IEEE Transactions on Neural Networks and Learning Systems, 2022. Early Access.
[70] Yue Tan, Guodong Long, Jie Ma, Lu Liu, Tianyi Zhou, and Jing Jiang. Federated learning from pre- trained models: A contrastive learning approach. Advances in Neural Information Processing Systems, 35: 19332-19344, 2022.
[71] Welderufael B Tesfay, Peter Hofmann, Toru Nakamura, Shinsaku Kiyomoto, and Jetzabel Sema. I read but don't i agree: Privacy policy benchmarking using machine learning and the eu gdpr. Companion Proceedings of the The Web Conference 2018, pages 163-166, 2018.
[72] Michael Tuchler, Andrew C Singer, and Ralf Koetter. Minimum Mean Squared Error Equalization using A Priori Information. IEEE Transactions on Signal Processing, 50(3):673-683 2002.
[73] Laurens Van der Maaten and Geoffrey Hinton. Visualizing Data Using T-SNE, Journal of Machine Learning Research, 911)), 2008
[74] Elena Voita and Ivan Titov. Information- - Probing with Minimum Description Length. In Conference on Empirical Methods Natural Language Processing (EMNLP), 2020.
[75] Jianyu Wang. Qinghua Liu, Hao Liang, Gauri Joshi, and H. Vincent Poor. Tackling the Objective Inconsistency Problem Heterogeneous Federated Optimization. In International Conference Advances in Neural Information Processing Systems (NeurIPS), 2020.
[76] Suge Wang. Deyu Li, Xiaolei Song. Yingjie Wei, and Hongxia Li. feature selection method based on improved fisher's discriminant ratio for text sentiment classification. Expert Systems with Applications, 38 (7):8696-8702, 2011.
[77] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-MNIST. A Novel Image Dataset for Benchmarking Machine Learning Algorithms. arXiv preprint 0rXiv:1708.07747 2017.
14

---

[78] Xiaoming Xue. Kai Zhang, Kay Chen Tan, Liang Feng. Jian Wang, Guodong Chen, Xinggang Zhao, Liming Zhang, and Jun Yao. Affine Transformation-nhaanee Multifactorial Optimization for Heterogeneous Problems. IEEE Transactions on Cybernetics, 52(7):6217-623) 2020.
[79] Qiang Yang. Yang Liu, Tianjian Chen, and Yongxin Tong. Federated Machine Learning: Concept and Applications. ACM Transactions on Intelligent Systems and Technology. 10(2):1-19. 2019.
[80] Han Yu, Zelei Liu, Yang Liu, Tianjian Chen, Mingshu Cong, Xi Weng, Dusit Niyato, and Qiang Yang. A lairness-aware aware incentive scheme for federated learning. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, pages 393-399, 2020.
[81] Ming Zeng, Le T Nguyen, Bo Yu, Ole Mengshoel, Jiang Zhu, Pang Wu, and Joy Zhang. Convolutional neural networks for human activity recognition using mobile sensors. In 6th international conference on mobile computing, applications and services, pages 197-205. IEEE, 2014.
[82] Chao Zhang, Lei Zhang, and Jieping Ye. Generalization Bounds for Domain Adaptation. In International Conference on Advances in Neural Information Processing Systems (NewrIPS), 2012.
[83] Daokun Zhang, Jie Yin, Xingquan Zhu, and Chengqi Zhang. Network Representation Learning; Survey. IEEE Trunsactions on Big Duta, 6(1):3-28, 2018.
[84] Jianqing Zhang, Yang Hua, Hao Wang, Tao Song, Zhengui Xue, Ruhui Ma, Jian Cao, and Haihing Guan. Gpfl: Simultaneously learning global and personalized feature information for personalized federated learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision. pages 5041-5051, 2023.
[85] Jianqing Zhang, Yang Hua, Hao Wang. Tao Song. Zhengui Xue, Ruhui Ma, and Haibing Guan. Fedep: Separating feature information for personalized federated learning via conditional policy. In Proceedings of the 29th ACM SIGKDD Conference Knowledge Discovery and Data Mining, 2023.
[86] Jianqing Zhang, Yang Hua, Hao Wang. Tao Song. Zhengui Xue, Ruhui Ma, and Haibing Guan. FedALA: Adaptive Local Aggregation for Personalized Federated Learning. In AAAI Conference on Artificial Intelligence (AAAI), 2023.
[87] Jianqing Zhang, Yang Hua, Hao Wang. Tao Song, Zhengui Xue, Ruhui Ma, and Haibing Guan. Fedala: Adaptive local aggregation for personalized federated leamnng. In Proceedings of the AAAI Conference on Artificial Intelligence. 2023.
[88] Lin Zhang. Li Shen, Liang Ding, Dacheng Tao, and Ling- Yu Duan. Fine-Tuning Global Model Via Data-Fre Knowledge Distillation for Non-lID Federated Learning. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[89] Michael Zhang, Karan Sapra, Sanja Fidler, Serena Yeung, and Jose M Alvarez. Personalized Federated Learning with First Order Model Optimization. In International Conference on Learning Representations (ICLR), 2020.
[90] Sixin Zhang, Anna E Choromanska, and Yann LeCun. Deep Learning with Elastic Averaging SGD, International Conference on Advances in Neural Information Processing Systems (NeurIPS), 2015.
[91] Xiang Zhang, Junbo Zhao, and Yann LeCun. Character- Level Convolutional Networks for Text Classifi- cation. In International Conference on Advances Neural Information Processing Systems (NeurIPS). 2015.
[92] Yuchen Zhang, Tianle Liu, Mingsheng Long, and Michael Jordan. Bridging Theory and Algorithm for Domain Adaptation. In International Conference on Machine Learning (ICML), 2019.
[93] Zaixi Zhang, Xiaoyu Cao, Jinyuan Jia, and Neil Zhenqiang Gong. Fldetector: Defending federated lcarning against model poisoning attacks via detecting malicious clients. In Proceedings of ACM SIGKDD International Conference on Knowledge Discovery &amp; Data Mining, 2022.
[94] Han Zhao, Shanghang Zhang, Guanhang Wu, José MF Moura. Joao P Costeira, and Geoffrey J Gordon. Adversarial Multiple Source Domain Adaptation. In International Conference Advances in Neural Information Processing Systems (NeurIPS), 2018.
[95] Wennan Zhu, Peter Kairouz, Brendan McMahan, Haicheng Sun, and Wei Li. Federated Heavy Hitters Discovery with Differential Privacy. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2020.
[96] Zhuangdi Zhu, Junyuan Hong, and Jiayu Zhou. Data-Free Knowledge Distillation for Heterogeneous Federated Learning. In International Conference on Machine Learning (ICML), 2021.
15

---

We provide more details and results about our work in the appendices. Here are the contents:
Appendix
A
The extended version of the Related Work section in the main body.
Appendix
B
Proof's of Corollary and Corollary 2.
Appendix
C
More details about experimental settings.
Appendix
Additional experiments (e.g., a real-wwoll weal-world application).
Appendix
Broader impacts of our proposed method.
Appendix
Limitations of proposed method.
Appendix
Data distribution visualizations for different scenarios in our experiments.
A
Related Work
As the number of users and sensors rapidly increases with massive growing services on the Internet, the privacy
concerns about private data also draw increasing attention rescarchers [37 68 71. Then a new distributed
machine learning paradigm, federated learning (FL), comes along with the preneceiee and collaborative
learning abilities [37] [79]. Although there are horizontal FL [46] (56] (7),, vertical FL [53 63 (7), federated transfer learning 15 51 etc., we focus on the popular horizontal FL. and call it FI. for short in this paper.
Traditional FL. methods concentrate on learning single global model among a server and clients, but it suffers an accuracy decrease under statistically heterogeneous scenarios, which are common scenarios in practice [47 [56 |7]66) [86). Then, many FL methods propose learning personalized models (or modules) for each client besides learning the global model. These FL methods are specifically called personalized FL (pFL)
methods 20 22 69]
A.1 Traditional Federated Learning
FL methods perform machine learning through iterative communication and computation on server and clients. To begin with, we describe the FL procedure in one iteration based on FedAvg (56). which is famous FL method and a basic framework for later FL methods. The FL procedure includes five stages: (1) A server selects group of clients join FL. in this iteration and sends the current global model to them; (2) these clients receive the global model and initialize their local model by overwriting their local model with the parameters in the global model; (3) these clients train their local models on their own private local data, respectively; (4) these clients send the trained local models to the server; (5) the server receives client models and aggregates them through weighted averaging on model parameters to obtain new global model.
Then, massive traditional FL methods are proposed in the literature to improve FedAvg regarding privacy-
preserving 4g 58 [3]. accuracy [3 45 96]. faimess [32 80) overhead 28 4 etc. Here, focus
on the representative traditional FL methods that handle the heterogeneity issues four categories: update-
correction-basse FL
[25] 38 60 regularization based FL [1 40 46). model-split-bases FL [35 45 and
nhowledgediistlllatoon based FL.
27 33 88 (96)
Among apdate-correetion-baaee FL methods, SCAFFOLD 38 witnesses the client-drift phenomenon of FedAvg under statistically heterogeneous scenarios due to local training and proposes correcting local update through control variates for each model parameter. Among regularization-basel FL methods, FedProx (46) modifies the local objective on each client by adding a regularization term to keep local model parameters close to the global model during local training in an element-wise manner. Among model-split-base MOON (45) observes that local training degenerates representation quality, so it adds a contrastive learning term to let the representations outputted by the local feature extractor be close to the ones outputted by the received global feature extractor given each input during local training. However, input-wise contrastive lcarning relies on biased local data domains, so MOON still suffers from representation bias. Among nnwwlege-disttllationn based FL methods, FedGen [96] learns a generator on the server to produce additional representations, shares the generator among clients, and locally trains the classifier with the combination of the representations outputted by the local feature extractor and the additionally generated representations. In this way, FedGen can reduce the heterogeneity among clients with the augmented representations from the shared generator via knowledge distillation. However, it only considers the local-to-globa, knowledge transfer for the single global model learning and additionally brings communication and computation overhead for learning and transmitting the generator.
FL methods,
A.2 Personalized Federated Learning
Different from traditional FL, pFL additionally learns personalized models (or modules) besides the global model. In this paper, we consider pFL methods in four categories: meta- meta-learning baaannng-bssed pFL [13] (2). regularization- based pFL (7). htwplliieeddggggggtiionnieeeeeee pFL
21 52 89 and model-split-basee pFL |4 20 61].
16

---

Meta-learning--aasee pFL. Meta-learning is technique that trains deep neural networks (DNNs) on given dataset for quickly adapting other datasets with only few steps of fine-tuning, e.g., MAML 24 By integrating MAML into FL. Per-FedAvg [22] updates the local models like MAML to capture the learning trends of cach elient and then aggregates the learning trends by averaging on the server. It obtains personalized models by mee-tnnng the global model for cach client. Similar to Per-FedAvg, FedMeta [13 also introduces MAML on each client during training and fine-tuning the global model for evaluation. However, it is hard for these meta-leaming- based pFL methods find a consensus learning trend through averaging under statistically heterogeneous scenarios.
Regularization-bun based pFI.. Like FedProx, PFedMe 67 and Ditto also utilize the regularization technique, but they modify the objective for additional personalized model training rather than the one for local model training. In pFedMe and Ditto, each client owns two models: the local model that is trained for global model aggregation and the personalized model that is trained for personalization. Specifically, pFedMe regularizes the model parameters between the personalized model and the local model during training while Ditto regularizes the model parameters between the personalized model and the received global model. Besides, Ditto simply trains the local model similar to FedAvg while PFedMc trains the local model based on the personalized model. A Although the local model is initialized by the global model, but the initialized local model gradually loses global information during local training. Thus, the personalized model in Ditto be aware of more global information than the one in pFedMe. Both PFedMe and Ditto require additional memory space to store the personalized model and double the computation resources at least to train both the local model and the personalized model.
Personalized Personaaizze-aaggeeattion baasd pFL. These pFL methods adaptively aggregate the global model and local model according to the local data on each client, e.g., APFL (21. or directly generate the personalized model using other client models through personalized aggregation Specifically, APFL. aggregates the parameters in the global model and the local model with weighted averaging and adaptively updates the scalar weight based on the gradients. On each client, FedFomo generates the client-specific aggregating weights for the received client models through first-order approximation while APPLE adaptively learns these weights based on the local data. Both FedFomo and APPLE require multiple communication overhead than other FL methods, but FedFomo costs less computation overhead than APPLE attributed to approximation.
each client, e.g., FedFomo [89] and APPLE [52]
Model-split-base pFL. These pFL methods split given model into a feature extractor and a classifier. They treat the feature extractor and the classifier differently. Concretely, FedPer (3) and FedRep (20) keep the classifier locally on each client. FedPer trains the feature extractor and the classifier together while FedRep first fine-tunes the classifier and then trains the feature extractor in each iteration. For FedPer and FedRep, the feature extractor intends to extract representations to cater these personalized classifiers, thus reducing the generic representation quality. FedRoD [14) trains the local model with the balanced softmax (BSM) loss function 64] and simultaneously learns an additional personalized classifier for each client. However, the BSM loss is useless for missing labels on cach client while label missing is a common situation in statistically heterogeneous scenarios [50 86 89). Moreover, the uniform label distribution modified by the BSM cannot reflect the original distribution. The above pFL methods learn personalized models (or modules) in FL. but FedBABU [6] firstly trains the global feature extractor with the frozen classifier during the Fl. process, then it fine-tunes the global model on each client after FI. to obtain personalized models. However, this post-Fl. fine-tuning is heyond the scope of FL., Almost all the FL methods have multiple fine-tuning variants, e-g., fine-tuning the whole model or only part of the model. Furthermore, training the feature extractor with the naive and randomly initialized classifier FL has an uncontrollable risk due to randomness.
B Theoretical Derivations
B.1 Notations and Preliminaries
Following prior arts 21 55
(9), we consider binary classification problem in FL here. Recall that X' C R²
an input space, Z C & a representation space, and y C {0, is a label space. F X + be representation function that maps from the input space to the representation space. We denote D = (U data domain where the distribution i/ C X and c² Xx + y is ground-truth labeling function. E7 is the induced distribution of U/ over the representation space Z under F ( i.e. u C Z, that satisfies
E P [B (z)) = Fu-u u [3 (F (m))),
(10)
where B is a probability event. Given tixed but unknown 24 and c², the learning task on one domain is to choose a representation function F and hypothesis class 7 C {h: 2 + }} to approximate the function c².
Then, we provide the definition and theorem from Ben-David et al. 6 7 Blitzer et al. [9] Kifer et al. 39
under
their assumptions:
17

---

Definition 1. If a space 2 with it and and distributions over 2, let H. be hypothesis class on Z and Zn S Z be the subset with characteristic function h, the H-divergence between Ut and ld is
du (ir ((P,.)) = sup [Priz [2,] Prga [2n]]
NER
where Zn: = E = 1},h H.
Definition implies that du ((-,2()) dr (ū. (UP.U)).
Theorem 1. Consider source domain Ds and target domain Dr. Let Ds Mss c²)) and Dr (Ur, c²), where Us C Ur S X. and c² X + ) a ground-truth labeling function. Let H be a hypothesis space H. Given a feature extraction function : + Z that shared between Dy and Dr, random labeled sample of size m generated by applying F to a random sample from lss labeled according to ct, then for every h H. with probability at least 1
of VC dimension d and =2 H V.A
Cor (1)) Lbg (h) +1 (dlog 20m + log 2 dı (tr) 2,
where L.bg is the empirical loss Ds, is the base of the natural logarithu, and dre (c,)) is the H-divergence between two distributions. ús and Utz are the induced distributions of Uss and Ur under respectively, s.f. E_ Eg- [B [z) = En-its [B (F (a=)] given probability event B. and for lr. Us C Z and Ur S Z. À == ming, Log (h) + LDr (h) denotes an oracle performance.
The traditional FL methods, which focus on enhancing the performance of a global model, regard local domains D E [N] and the virtual global domain DD as the source domain and the target domain, respectively (96), which is called seal-to-glloe knowledge transfer this paper. In contrast, pFL methods that focus improving the performance of personalized models regard 'D and DD, E [N] as the source domain and the target domain, respectively [21 [55 69] We call this kind of adaptation global-to-local knowledge transfer. The local-to-global knowledge transfer happens on the server while the global-to-local one occurs on the client.
B.2 Derivations of Corollary
As we focus on the local-to-global knowledge transfer on the server side, in the FL scenario, we can rewrite Thcorem[]jo
Theorem Consider a local data domain D. and a virtual global data domain D. Let D; = (U, c²) and D = (",,"'), where U. C X and u S X. Given a feature extraction function F X + Z that shared between D, and D, a random labeled sample of size m. generated by applying F to random sample from li, labeled according to ct, then for every h E H. with probability at least 1 -5:
2em
1) + log 2) + du (u + de,
LD (h) S Lo. (1)) 1
dllog
where Uh and i are the induced distributions uf X; = mina LD. (h) + Lo (h) denotes an oracle performance.
U, and u under F, respectively. Us s z and S z.
Corollary Consider a local data domain D. and virtual global data domain D for client and the server respectively. Let D. = (W,, c²) and D = (U1 c²), where c¹ XX + y is a ground trunh labeling function. Let H be a hypothesis space of VC dimension d and h Z >> Y,VE h E H. When using DBE, given a feature extraction function F X +> Z that shared between D; and D, random labeled sample of size generated by applying F to random sample from Ud, labeled accorting
then for every h2 H. with probability least 1 8:
Lo (h²) S Co, (h²)
4
(dlog
ddlog
2em log
4.
+ (ūp,üP) dis
where is the empirical loss on D., the base of the natural logarithum, and du (---) the H-divergence between two distributions. A minss Lo (11) Lo, (1e). Ue 2. U Z. and du (Ue, (ū0 du (... U. and Ú are the induced distributions of U4 and U under F9, respectively. Uk and ú are the induced distributions Us and 14 under F. respectively. J is the feature extraction function in the original FedAvg without DBE.
Proof. Computing du (1.-) identical learning a classifier achieve a minimum error discriminating
between points sampled from il and U² i.e.. binary domain classification problem 6 7 The difficult
18

---

the domain classification problem is, the smaller du ( is. Unfortunately, computing the error of the optimal
hyperplane classifier for arbitrary distributions is well-known NP-hard problem [ 6 Thus, researchers approximate error by learning a linear classifier the binary domain classification
Inspired by
previous approaches 4 (7] we consider using Linear Discriminant Analysis (LDA) for the binary domain classification. The discrimination ability of LDA is measured by the Fisher discriminant ratio (F1) [ 30 76
(Hir 1tje)
F1 (ū)
max
(o)) (ot)) )² +0(00) 216
where u4 and (o))) are the mean variance of values in the kith dimension over Ut The smaller the Fisher discriminant ratio is, the less discriminative the two domains are. Theorem hholds with every le E H. so we omit PRBM here. MR (z, 2²) forces the local domain to be close to the global domain in terms of the mean value each feature dimension in the feature representation independently, therefore, V [K].
P S Pu:
As the feature extractors share the same structure with identical parameter initialization and the feature represen- tations are extracted from the data domain D; (D) [88 (3], we assume that ozz oū2 and dūj dū- Thus, l [K].
(nip k
(
p
(o
As this inequality is satisfied in dimensions including dimension where the maximum value exists, so for the Fisher discriminant ratio, we have
(1 14500 )²
(1
14 2
F1 (ūe. max
F1
(uh.
( + )
(a
The smaller the Fisher discriminant ratio is, the less discriminative the domains are. The less discriminative the two domains are, the smaller dx (:--) is. Thus, finally. we have
du (ee, - dn (u,.ū))
B.3 Derivations of Corollary 2
When we focus the global-to-local knowledge transfer the client side, in the FL scenario, rewrite Thcorem Mas
Theorem 3. Consider a virtual global data domain D and a local data domain D.. Let D (U,c²) and D. (U, ).), where U S X and Us C X. Given a feature extraction function F X - Z that shared between D and D., random labeled sample of size m generated by applying F to random sample from U labeled according to ci, then for every h H. with probability at least 1 -::
(1))+)+++ LD. (h) Lo (h) + / (dlog 2em + log 2) + du (u.u.) dos
2))
where U, and it are the induced distributions of Ut X; = ming Co (h) + LD, (h) denotes an oracle performance.
under F, respectively. ús S and ú C
Corollary Let Di, F and defined in Corollary 7 Given a translation transformation function that shared between D and virtual D, a random labeled sample of size m generated by applying F' to a random sample from Ut, labeled according to c", F' PREM o F + Z, then for every h E H. with probability at least 8:
PREM:
for
Lo. (h)) S Co (N') + + ddoo
2em + log 1)
du (U, + Ao,
where du (i ui) du (ūP, s dn (i. = du (i). . il and iti are the induced distributions U and U4 under F', respectively.
19

---

Proef. PREM is translation transformation with parameters 2 s.l. V æs U1 z + zf, where Ui. Therefore, we have Prap [[= E 2 = Pra. [[ E 2}] and the same applies to the pair of it and út, ie., Prize H2 2}} =
2; = FF (m:) E UK and z = FF (m:) E Uf. In other words, V o E U2 E1 2; E
Pra [ E 2|). Then the subtraction of the probability on each side is also equal, i.e.
Prac [= 2 Prez x 2H Pray I[ 2H Pre 1< E 27)-
VV E H = N' PREM E H. so x" E 2 if 120 (z*) = 1. then 1/' (z*) = 1. where x² Therefore, we have
Prop [Zno] Prizu [Zho] = Pru; [2 Pra [2w]]
where Zno = (z E 2 : 1 (2) = 11,18 1t0 E H and Znr- {= E 2 W (z) 1), E H. According to Definition have
du (it .i)) = 2 sup NER
PER [Zw]] - Pe [Zn-]
2 sup MeRR
Prazy [Zno]
Prize [2x0]
= du (uP (if)
< dH (ú,
Detailed Settings
C.1
Implementation Details
We create the datasets for each client using six public datasets: Fashion-MNIST (FMNIST 1od Tiny- ImageNef[(100K images with 200 labels) and AG New (a news classification dataset with four labels, more than 30K samples per label). The MDL is calculated through the public code" We run all experiments machine with two Intel Xcon Gold 6140 CPUs (36 cores), 128G memory, eight NVIDIA 2080 Ti GPUs, and CentOS
C.2 Hyperparameters of DBE
For hyperparameter tuning. we use grid search to find optimal hyperparameters, including x and Specifically, grid search is performed in the following search space:
<: 0.001, 0.01, 0.1, 10. 20, 50, 100, 200, 500
ju: O. 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0
In this paper, we set x = 50, 1.0 for the 4-layer CNN, k = 1, 0.1 for the ResNet-18, and 0. = 1.0 for the fastText We only set different values for the hyperparameters and μ on different model architectures but identical settings for one architecture all datasets. Different models exhibit diverse capabilities both feature extraction and classification. Given that our proposed DBE operates by integrating itself into a specific model, it is crucial to tune the parameters k and j to adapt to the feature extraction and classification abilities of different models.
As for the criteria for hyperparameter tuning, K and require different tunning methods according to their functions. Specifically, is momentum introduced along with the widely-used moving average technology in approximating statistics, for the model architectures that originally contain statistics collection operations (e.g.. the batch normalization layers in ResNet- 18) one can set relatively small value by tuning a from I with a reasonable step size. For other model architectures, one can set relatively large value for by tuning it from 1 to O. The parameter k is utilized to regulate the magnitude of the MSE loss in MR. However, different architectures generate feature representations with varying magnitudes, leading to differences in the magnitude of the MSE loss. Thus, we tune by aligning the magnitude the MSE loss with the other loss term.
httpp//ggeechhooo////sssssssssss httmlleettttsmttttmtemmmmmmmmm html #fmnist
hhgp:///..aabbbbeaaaaaaacttaaaaa news
20

---

D
Additional Experiments
D.1
Convergence
1.0
2.0
Training Loss
Training Loss Test Accuracy
0.8
Test Accuracy
1.5
0.6
1.0
0.4
0.2
0.5
100
200
300
400
500
600
Iterations
Figure 4: The training loss and test accuracy curve of FedAvg+DBE on FMNIST dataset using the 4-lyyer CNN in the practical setting.
Recall that objective
min Eie (NLD, 01....9x
(0.)), (0.)),
(1I)
and its empirical version is mine........ 0x > 2i Lp, L, (0.). Here, we visualize the value of - Th L.p (0)
and the corresponding test accuracy during the FL. process. Figure the convergence of Fedlvv+Deas and its stable training procedure. Besides, Imageet using ResNet- 18 in Table Based on the findings from Table we observe that the utilization of DBE can yield substantial reduction from 230 to 107 (more than 50%) in the total number of communication iterations needed for convergence, as compared to the original requirements of FedAvg.
also report the total iterations required for convergence on
D.2 Model-Splitting in ResNet-18
main body, we have shown that DEE improves the per-layer MDI. and accuracy of FedAvg no matter how we split the 4-layer CNN. In Table we report the per-layer MDL and accuracy when we consider model splitting in ResNet-18, 18, model deeper than the 4-layer CNN. No matter at which layer, we split ResNet- 18 to form feature extractor and classifier, DBE also reduce MDL and improve accuracy, showing its general applicability.
Table The MDL (bits, 4) layer-wise representations, test accuracy (%, 1), and the number of trainable parameters (L) in PRBM when adding DBE to FedAvg on Tiny-ImageNet using ResNet- 18 in the practical setting. The "B", "CONV", "POOL", and "FC" means the "block", "convolution block". "average pool layer", and "fully connected layer" in ResNet-18 [29], respectively.
MDL
Metries
Accuracy
Param.
CONV-BI
B1-B3
B2-BB3
B3 BB4
B4-POOI.
POOL,-FC
Logits
Original (FedAvg)
4198
3598
3501
3445
3560
3679
19.45
CONV DBB -BB
4332
4050
3528
3407
3292
3347
3493
19.96
16384
B1-DBB -B2
4527
4072
3568
3456
3361
3451
3560
19.50
16384
B2-+DBE
4091
3575
3326
3520
19.55
B3--DBE BB4
4447
4073
3511
3414
3259
3346
3467
20.72
4096
B4-+DBE -POOL
4424
4030
3391
3304
3284
3511
3612
39.99
2048
POOL--0BE -FFC
4432
4035
3359
3298
3209
3454
3594
42.98
512
D.3
Distinguishable Representations
As our primary goal is to demonstrate the elimination of representation bias rather than improving discrimination in Figure 3 (main body), we present the t-SNE visualization for our largest dataset in experiments, Imageeete (200 labels). Given that the 200 labels are distributed around the chromatic circle, adjacent labels are assigned
21

---

similar colors, resulting in Figure 3 (main body) being indistinguishable by the label. Using dataset AG News with only four labels for t-SNE visualization clearly show that the representations extracted the global feature extractor are distinguishable Figure
Figure 5: t-SNE visualization for the representations extracted by the global feature extractor on AG News (four labels) in FedAvg+DBE. We use color and shape to distinguish labels and clients, respectively.
D.4 A Practical Scenario with New Participants
simulate a practical scenario new clients joining future FL., we perform method-specific local training for 10 epachs on new participants for warming up after their local models are initialized by the learned global model (or client models in FedFomo). Since FedAvg, Per-FedAve and FedBABU do not generate personalized models during the FL process, accuracy. Specifically, using Cifar100 and 4-layer CNN, we conduct FL on clients for them to obtain test and evaluate accuracy on 20 new joining clients after warming up. We utilize the data distribution depicted in Figure According to Table [ FedAvg shows excellent generalization ability with fine-tuning. However, DBE can still improve FedAvg by up to +6.68 with more stable performance for different
fine-tur the entire global model
clients (p
p 0.1)
Table 8: The total iterations for convergence and the averaged test accuracy (%, t) of pFL methods.
Items
Iterations
New Participants
Local Epochs
p
0.5
p 0.1
5
10
Per-FedAvg 22
34
48.66
48.36
95.10
93.92
93.91
pFedMe 667
113
41.20
38.39
97.25
97.44
97.32
Ditto 47
27
36.57
45.06
97.47
97.67
97.64
FedPer
3
39.86
42.39
97.44
97.50
97.54
FedRep
20
115
38.75
35.09
97.56
97.55
97.55
FedRoD 14
50.10
51.73
97.52
97.49
97.35
FedBABU 61
513
48.60
42.29
97.46
97.57
97.65
APFL 21
57
38.19
45.16
97.25
97.31
97.34
FedFomo (89
71
27.50
27.47
97.21
97.17
97.22
APPLE [52
45
97.06
97.07
97.01
FedAvg
230
52.52
49. 49.44
85.85
85.96
85.53
FedAAg+DBE
107
57.62
56.12
97.69
97.75
97.78
22

---

D.5 Large Local Epochs
We also conduct experiments with more local epochs in each iteration FMNIST using the 4-layer CNN, shown in Table All the pFL methods perform similarly with the results for local epoch, except for Per- Per-FedAgg which degenerates around 1.18 accuracy (%).
Table 9: The test accuracy (%) on the HAR dataset.
Methods
Accuracy
FedAvg
87.20:10.27
SCAFFOLD
91.34:10.4
FedProx
88.34:0.24
MOON
89.86=0.18
FedGen
90.82:10.21
Per-FedAvg
77.12:0.17
pFedMe
91.57=0.12
Ditto
91.53=#0.09
FedPer
73.58=0.13
FedRep
80.4410.42
FedRoD
$9.91-0.22
FedBABU
87.12+0.31
APFL
92.1810.51
FedFomo
63.3910.48
APPLE
86.46110335
FedAvg+DBE
94.5310.26
D.6 Real- World Application
We also evaluate the performance of our DBE in real-world application. Specifically, apply DBE the Internet- of-Things (IoT) scenario on popular Human Activity Recognition (HAR) dataset with the HAR-CN [8] model. HAR contains the sensor signal data collected from 30 users who perform six activitics (WALKING, WALKING JUNSTASSSS WALKING DOWNSTAIRS, SITTING, STANDING, LAYING) wearing a smartphone the waist. We show the results in Table
where FedAvg+DBE still achieves superior performance.
E Broader Impacts
The representation bias and representation degeneration naturally exist in FL under statistically heterogeneous scenarios, which are derived from the inherently separated local data domains individual clients. In the main body, show the general applicability of our proposed DBE can also applied other practical fields, such as Internet Things (IoT) [26] [1] and digital health [15 16). Furthermore, introducing the view of knowledge transfer into FL sheds light on this field.
to reprosentative FL methods. More than that,
F
Limitations
Although FL comes along for privacy-preservee and collaborative learning, still suffers from privacy leakage issues with malicious clients 12 99 or under attacks 23 53) We design DBE based on FL to improve generalization and personalization abilities, and we only modify the local training procedure without affecting the downloading, uploading, and aggregation processes. Thus, the DBE-equipped FL methods still suffer from originally existing privacy issues like the original version requires future work to devise specific methods for presecrinn enhancement.
FL methods when attacks happen.
G
Data Distribution Visualization
illustrate data distributions (including training and test data) our experiments here.
23

---

200
80
150
Claee Ibs
800
Clee (Ds
1.0
80
20
50
9110112131 181
Ciient
(a) FMNIST
(b) C0
(c) ImageNet
Figure 6: The data distributions of all clients on FMNIST, C00, and Tiny-ImageNet, respectively, in the pathological settings. The size of a circle represents the number of samples.
Dloss 160
Cass
0 2 3 5 910111 41511777111
2.3 7 8.8 10111251141111777
0.1 2
Client
Chard IDs
Clini IDs
(a) FMNIST
(b) Cifar100
(c) TmageNn
Figure 7: The data distributions of all clients on FMNIST, C0far100, and Imay-ImageNet, respectively, the practical settings (3 0.1). The size of circle represents the number of samples.
200
200
150
150
150
6
89
100
Clnss
100
80
50
911011219141001171199
101112
(a) 3 0.01
(b) = 0.5
(c) 3= 5
Figure 8: The data distribution all clients Tiny-ImageNet in three additional practical settings. The size of a circle represents the number of samples. The degree of heterogeneity decreases as B3 in Dir(3) increases.
100
80
Class IDs
40
20
Client IDs
Figure 9: The data distributions of all clients on C0far100 in the practical setting (3 = 0.1) with 100 clients, respectively. The size of circle represents the number of samples.
24


Reviews:
[{"review_id": "MIFmXOxpoT", "reviewer": "Reviewer_dEaC", "summary": "The paper introduces Domain Bias Eliminator (DBE) to address the representation bias and representation degeneration issues commonly observed in federated learning under statistically heterogeneous scenarios. DBE utilizes a Personalized Representation Bias Memory (PRBM) to preserve representation bias and mean regularization to guide local feature extractors toward producing representations with a consensual global mean. The authors provided a thorough theoretical analysis and empirical studies to validate their method. The results indicated that DBE can enhance the performance of existing FL methods.", "questions": "Is there any criteria or empirical methods to choose hyperparameters in this work?", "limitations": "The authors have addressed the limitations.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper has solid theoretical analysis.\n2. The proposed method is well-motivated.\n3. Comparison with existing methods is sufficient.\n", "weaknesses": "1. The explanation on how DBE reduces the domain discrepancy between server and client in the representation space could be more detailed, as it forms a core part of the technique.\n\n2. The authors could also consider exploring DBE's scalability to larger models, to understand the full potential of the method. The models used for experiments are all small, with only million-level parameters.\n\n3. There are two main hyperparameters in the proposed method and it seems that the recommended choice is quite different for different datasets and models. It is difficult for practical scenarios to tune these hyperparameters.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback and insightful comments. We respond to your concerns as follows in the form of \"**[A weakness or question]** Our responses\". \n\n\n**[More detailed explanation on how $\\texttt{DBE}$ reduces the domain discrepancy]** As mentioned in *Section 4.4 Improved Bi-directional Knowledge Transfer*, our $\\texttt{DBE}$ reduces $\\mathcal{H}$-divergence (which measures domain discrepancy[1]) for both levels of representations. Here we show how $\\texttt{DBE}$ reduces the domain discrepancy based on the widely used maximum mean discrepancy (MMD) metric[2]. $MMD[\\Phi, P, Q] := \\sup _{\\phi \\in \\Phi}(\\mathbb{E} _{p\\sim P}[\\phi(p)] - \\mathbb{E} _{q\\sim Q}[\\phi(q)])$, where domains $P$ and $Q$ belong to a space $\\mathcal{R}$ and $\\Phi$ is a class of given functions $\\phi: \\mathcal{R} \\rightarrow \\mathbb{R}$. Our $\\texttt{MR}$ is originally designed to reduce the mean discrepancy while our proposed translation $\\texttt{PRBM}$ can keep the reduced mean discrepancy. Consider $P$ and $Q$ as the local representation domain and virtual global representation domain, respectively, and regard $\\Phi$ as a class of identical summation function $\\phi$, the MMD value can be reduced with the reduced mean discrepancy of $P$ and $Q$. \nWe will continue to refine our paper in the revised version, aiming to enhance its impact and improve its clarity for better understanding. \n\n[1] Ben-David S, Blitzer J, Crammer K, et al. A theory of learning from different domains. Machine learning, 2010, 79: 151-175.\n\n[2] Gretton A, Borgwardt K, Rasch M, et al. A kernel method for the two-sample-problem. Advances in neural information processing systems, 2006.\n\n\n**[Small models with million-level parameters]** We follow prior approaches to adopt models for a fair comparison. The majority of existing FL approaches employ models with million (M)-level parameters. Specifically, shallow CNNs (around 5.6M parameters, used in FedAvg, FedGen, MOON, Ditto, FedRep, FedRoD, APFL, FedFomo, and APPLE) and ResNet-18 (around 11.2M parameters, used in FedBABU, FedALA[3], TCT[4], and ProgFed[5]) are popular models in the FL field. \n\n[3] Zhang J, Hua Y, Wang H, et al. Fedala: Adaptive local aggregation for personalized federated learning. Proceedings of the AAAI Conference on Artificial Intelligence. 2023.\n\n[4] Yu Y, Wei A, Karimireddy S P, et al. TCT: Convexifying federated learning using bootstrapped neural tangent kernels. Advances in Neural Information Processing Systems, 2022.\n\n[5] Wang H P, Stich S, He Y, et al. Progfed: effective, communication, and computation efficient federated learning by progressive training. International Conference on Machine Learning. 2022.\n\n\n**[Hyperparameter settings]** Please note that we **only** set different values for the hyperparameters $\\kappa$ and $\\mu$ on different model architectures but use identical settings for one architecture on all datasets. Different models exhibit diverse capabilities in both feature extraction and classification. Given that our proposed $\\texttt{DBE}$ operates by integrating itself into a specific model, it is crucial to tune the parameters $\\kappa$ and $\\mu$ to adapt to the feature extraction and classification abilities of different models. As for the **criteria or empirical methods for hyperparameter tunning**, $\\kappa$ and $\\mu$ require different tunning methods according to their functions. Specifically, *$\\mu$ is a momentum* introduced along with the widely-used moving average technology in approximating statistics, so for the model architectures that originally contain statistics collection operations (e.g., the batch normalization layers in ResNet-18) one can set a relatively small value by tuning $\\mu$ from 0 to 1 with a reasonable step size. For other model architectures, one can set a relatively large value for $\\mu$ by tuning it from 1 to 0. The parameter *$\\kappa$ is utilized to regulate the magnitude of the MSE loss in our $\\texttt{MR}$ (Equation (8))*. However, different architectures generate feature representations with varying magnitudes, leading to differences in the magnitude of the MSE loss. Thus, we tune $\\kappa$ by aligning the magnitude of the MSE loss with the other loss term, i.e., $\\frac{1}{n _i} \\sum^{n _i} _{j=1} \\ell(h(\\texttt{PRBM}(f({x} _{ij}; {\\theta}^f); \\bar{z}^p _i); {\\theta}^h), y _{ij})$. "}, {"review_id": "VyvYGBUXQb", "reviewer": "Reviewer_8bi2", "summary": "To address the performance drop via heterogeneous data distribution, the proposed method -  Domain Bias Estimator (DBE) - decomposes image features into unbiased (global) and biased (personalized) representations.\n\nTo guide the feature extractor to obtain the unbiased representation, Mean Regularization (MR) is proposed which is a regularization term for local objective functions that enforce to reduce the gap between mean of representations of local data and a mean of representation of all data across clients. Meanwhile, each client has learnable biased representation which is added to unbiased representation from the feature extractor, and then feed to the local classifier. \n\nUnlike the previous works, by considering both unbiased and biased representation, the proposed method could improve bi-directional (local and global) knowledge transfer and could mitigate the performance drop via heterogeneity.", "questions": "- Collecting averaged features extracted from all local data seems to be exposed to privacy attacks. It would be helpful to give more evidence (previous works or experiments) that the ‘client-specific mean’ collection is privacy-preserving.\n- Figure 3 shows the role of client-variant (biased) representations and client-invariant (unbiased) representations. However, the representation is not separated according to labels. Are they distinguishable by label for representations extracted by the global feature extractor?", "limitations": "Yes, the authors well describe the limitation of the proposed framework. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed method handles an important problem of non-IID FL.\n- The paper is clearly written.\n- The paper provides theoretical guarantees and comprehensive experimental results that show the efficiency of the proposed method for the given problem.\n- The proposed method is novel in terms of decomposing representation generated by feature extractor into unbiased and biased representations, and efficient since it requires not much additional computational cost.", "weaknesses": "- The paper does not discuss about privacy issues that could be emerged by collection of client-specific mean over local data (line 2,3 in Algorithm 1, Supple). Since the client-specific mean contains representations from all local data, it is potentially exposed to reconstruct identifiable information of local data or even data itself.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback and insightful comments. We respond to your concerns as follows in the form of \"**[A weakness or question]** Our responses\". \n\n\n**[Privacy issues for client-specific mean collection (*line 2,3* in Algorithm 1, Supplementary)]** Please note that the client-specific mean $\\{\\bar{{z}}^g _1, \\ldots, \\bar{{z}}^g _N\\}$ we collect during initialization period before FL is a $K$ dimensional vector for each client, respectively, instead of a set \"containing representations from all local data\". Hence, the client-specific mean solely captures the magnitude of the mean value for each feature dimension within the context of the given datasets and models. Sharing this kind of information is recently *popular* in the federated learning domain. FedPAC[1], FedProto[2], and FedPCL[3] share client-specific and class-wise mean (multiple $K$ dimensional vectors per client). Compared to them, the privacy-preserving ability of FedAvg+$\\texttt{DBE}$ is superior, as they share both client-level and class-level information in each iteration while no class-level information is shared in FedAvg+$\\texttt{DBE}$ and *we share the client-specific mean only once*. Furthermore, it is convenient to add privacy-preserving techniques (e.g., adding noise) to the client-specific mean during the initialization period without influencing the performance of our $\\texttt{DBE}$, since the magnitude of the client-specific mean hardly changes with noise. Following FedPCL, we add Gaussian noise to the client-specific mean with controllable parameters scale ($s$) and perturbation coefficient ($p$). As shown in R-8bi2-Table 1, using the privacy-preserving technique can also bring slight improvement for our $\\texttt{DBE}$. \nIn the revised version of our paper, we will provide a more comprehensive privacy analysis to further strengthen it. \n\nR-8bi2-Table 1: The test accuracy (\\%) on TINY in the practical setting. \n|  | Original | $s=0.05$, $p=0.1$ | $s=0.05$, $p=0.2$ |\n|:-|:-:|:-:|:-:|\n| FedAvg+$\\texttt{DBE}$ | 43.32±0.12 | 43.81±0.15 | **44.10±0.10** | \n\n[1] Xu J, Tong X, Huang S L. Personalized Federated Learning with Feature Alignment and Classifier Collaboration. The Eleventh International Conference on Learning Representations. 2022.\n\n[2] Tan Y, Long G, Liu L, et al. Fedproto: Federated prototype learning across heterogeneous clients. Proceedings of the AAAI Conference on Artificial Intelligence. 2022.\n\n[3] Tan Y, Long G, Liu L, et al. Fedproto: Federated prototype learning across heterogeneous clients. Proceedings of the AAAI Conference on Artificial Intelligence. 2022.\n\n\n**[Are the representations extracted by the global feature extractor distinguishable by the label?]** Yes. As our primary goal is to demonstrate the elimination of representation bias rather than improving discrimination in Figure 3, we present the t-SNE visualization for our largest dataset in experiments, Tiny-ImageNet (200 labels). Given that the 200 labels are distributed around the chromatic circle, adjacent labels are assigned similar colors, resulting in Figure 3 being indistinguishable by the label. Using a dataset AG News with only four labels for t-SNE visualization can clearly show that the representations extracted by the global feature extractor are distinguishable in R-8bi2-Figure 1 (***please refer to the PDF in the global response field***). "}, {"review_id": "99TBsoxNer", "reviewer": "Reviewer_FLKZ", "summary": "This paper tackles the problem of representation bias phenomenon lead by biased domains and proposes Domain Bias Eliminator (DBE), a framework that reduces domain discrepancy between server and client in representation space by Personalized Representation Bias Memory (PRBM) and Mean Regularization (MR). Experimental results show that DBE outperforming ten state-of-the-art methods in both generalization and personalization abilities.", "questions": "1.\tI'm confused about the form of $\\bar{z}_i^p$, is it a trainable embedding vector (since it’s the personalized representation of client) or is it a model parameter (as Algorithm 1 claims)? Please elaborate on how $\\bar{z}_i^p$ is derived. \n2.\t$\\bar{z}^g$ is a consensus obtained during the initialization period, why is $\\bar{z}^g$ not updated during the training process? \n3.\tWhat’s the dimension of the global $z_i^g$ and personalized $\\bar{z}_i^p$ implemented in experiments?\n4.\tAs mentioned in Line 249 “We run three trials for all methods until empirical convergence on each task”, does this mean that all models are trained until they converge and the communication rounds are different for each model (which is rare in federal learning settings)? And what are the specific conditions for convergence?\n5.\tFigure 1 in Appendix only contains the results of FedAvg+DBE, how about other methods? It would be better to compare the results with other methods to judge the convergence rate.\n6.\tThe experimental results of the local model are missing in the paper\n7.\tIn Section E.3, the author evaluated the accuracy on 20 new clients. How is the data of these clients divided, and does the class of these data appear in the 80 old clients?\n", "limitations": "The limitation of the proposed method has been discussed before. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-\tIntroduce the PRBM and MR to address the representation bias issue, and the effectiveness of these modules is validated by experimental results.\n-\tProvide theoretical analysis on generalization bounds of the global and local feature extractors.\n-\tThis method has good compatibility and can be combined with other models.\n", "weaknesses": "-\tIt would be nice to evaluate these methods on real-world distribution shifts.\n-\tThe introduction of MR increases the computational burden.\n-\tSome parts of the writing are not clear.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and insightful feedback. We respond to your concerns as follows in the form of \"**[A weakness or question]** Our responses\". \n\n\n**[Evaluate methods on real-world distribution shifts]** Our $\\texttt{DBE}$ is also effective in real-world scenarios. Here, we evaluate FedAvg+$\\texttt{DBE}$ on the popular Human Activity Recognition (HAR) dataset[1] that contains the sensor signal data collected from 30 users who perform six activities wearing a smartphone on the waist. On HAR, following previous work[2], we use HAR-CNN as the model. We show the results in R-FLKZ-Table 1, where FedAvg+$\\texttt{DBE}$ still achieves superior performance. \n\nR-FLKZ-Table 1: The test accuracy (\\%) on the HAR dataset. \n|  | Accuracy |\n|:-|:-:| \n| FedAvg |87.20±0.27 |\n| SCAFFOLD |91.34±0.43 |\n| FedProx |88.34±0.24 |\n| MOON |89.86±0.18 |\n| FedGen |90.82±0.21 |\n| Per-FedAvg |77.12±0.17 |\n| pFedMe |91.57±0.12 |\n| Ditto |91.53±0.09 |\n| FedPer |75.58±0.13 |\n| FedRep |80.44±0.42 |\n| FedRoD |89.91±0.23 |\n| FedBABU |87.12±0.31 |\n| APFL |92.18±0.51 |\n| FedFomo |63.39±0.48 |\n| APPLE |86.46±0.35 |\n| FedAvg+$\\texttt{DBE}$ |**94.53±0.26** |\n\n[1] Anguita D, Ghio A, Oneto L, et al. Human activity recognition on smartphones using a multiclass hardware-friendly support vector machine. Ambient Assisted Living and Home Care: 4th International Workshop, IWAAL 2012.\n\n[2] Zeng M, Nguyen L T, Yu B, et al. Convolutional neural networks for human activity recognition using mobile sensors. 6th international conference on mobile computing, applications and services. IEEE, 2014: 197-205.\n\n\n**[Computational burden brought by $\\texttt{MR}$]** We discuss the additional computation overhead of our proposed $\\texttt{MR}$ in *Section 4.5 Negligible Additional Communication and Computation Overhead*. The parameterless $\\texttt{MR}$ computes the MSE loss for two representations of the $K$ dimension whose computational burden is negligible compared to model inference or backpropagation. \n\n\n**[Is the $\\bar{z}^p _i$ a model parameter?]** Yes, as Algorithm 1 claims, $\\bar{z}^p _i$ is a trainable model parameter in the client model when using $\\texttt{DBE}$. As mentioned in *line 154*, $\\bar{z}^p _i$ is the parameter of our personalized module $\\texttt{PRBM}$, which is updated simultaneously with other parts of the client model. According to Figure 2, Equation (4), *line 155*, and Equation (8), we obtain representation ${z} _i$ via ${z} _i = {z}^g _i + \\bar{z}^p _i$ during forward pass. For the backward pass, one can easily obtain the gradients of ${z} _i$, then the gradient of $\\bar{z}^p _i$ can be derived by chain rule $\\frac{\\partial \\mathcal{L} _{\\hat{\\mathcal{D}} _i}}{\\partial \\bar{z}^p _i} = \\frac{\\partial \\mathcal{L} _{\\hat{\\mathcal{D}} _i}}{\\partial {z} _i} \\frac{\\partial {z} _i}{\\partial \\bar{z}^p _i} = \\frac{\\partial \\mathcal{L} _{\\hat{\\mathcal{D}} _i}}{\\partial {z} _i}$. \n\n\n**[Why is ${z}^g$ not updated during the training process?]** We utilize ${z}^g$ to ensure consistent guidance for extracting client-invariant representation information in our $\\texttt{MR}$. The $\\texttt{MR}$ and $\\texttt{PRBM}$ form a complementary pair. The updating of ${z}^g$ during the training process introduces dynamics to the guidance in $\\texttt{MR}$, which can lead to a mismatch between the previously learned translation $\\texttt{PRBM}$ and the current state of $\\texttt{MR}$. Updating ${z}^g$ causes the test accuracy (\\%) of FedAvg+$\\texttt{DBE}$ to drop from 43.32 (Table 4, TINY, practical setting) to 41.55. \n\n\n**[The dimension of ${z}^g$ and $\\bar{z}^p _i$ implemented in experiments]** The dimension of ${z}^g$ and $\\bar{z}^p _i$ equals the dimension of feature representation space $\\mathcal{Z}$, which depends on the model architectures and model splitting methods. \nPer *line 265-266*, we choose the last FC layer as the classifier, following existing methods, for a fair comparison. The dimensions $K$ are set to 512, 512, and 32 for the 4-layer CNN, ResNet-18, and fastText models, respectively, as indicated in our code. \n\n\n**[Specific conditions for convergence?]** If an FL method (algorithm) converges at the 100th round, running 100 rounds is equivalent to running 1000 rounds in terms of performance. Running all methods until empirical convergence is equivalent to running each method for a maximum required number of rounds, such as 1000 rounds, to ensure the empirical convergence of all methods. This approach is commonly used in FL experiments. \n\n\n**[Convergence rate of other methods]** We demonstrate Figure 1 in Appendix to show the convergence of FedAvg+$\\texttt{DBE}$ rather than showing its superiority in convergence. Please note that we focus on improving the MDL and accuracy rather than emphasizing the superiority of the convergence rate. To compare the convergence rate, please refer to Table 5 (Overhead) to calculate the minimum communication iterations for convergence by dividing the values in the \"Total time\" column by the values in the \"Time/iteration\" column, i.e., Per-FedAvg: 34, pFedMe: 113, Ditto: 27, FedPer: 43, FedRep: 115, FedRoD: 50, FedBABU: 513, APFL: 57, FedFomo: 71, APPLE: 45, FedAvg: 230, FedAvg+$\\texttt{DBE}$: 107. Using our $\\texttt{DBE}$ reduces communication iterations for FedAvg. We will further improve this in the revised version. \n\n\n**[Results of the local model]** As stated in *line 244*, we adhere to the renowned pFedMe approach to present the results of the global model and personalized models for traditional FL and pFL methods, respectively. In pFL, the local model corresponds to the personalized model. \n\n\n**[The data distribution for *Section E.3* in Appendix]** We utilize the data distribution established in Table 4 (Cifar100†) consisting of 100 clients, as depicted in Figure 5 in the Appendix. "}, {"review_id": "xk6dPeISfQ", "reviewer": "Reviewer_xd4W", "summary": "This paper addresses the significant challenge of statistical heterogeneity among clients in federated learning. To tackle this issue, the authors propose personalized federated learning, where each client can learn and apply its specific model parameters. Specifically, the authors introduce DBE (Domain Bias Embedding), which comprises two crucial modules. The first module is a client-specific embedding designed to model the domain bias, while the second module is a regularization term incorporated into the objective function to encourage consensus in global representation learning. The effectiveness of these proposed modules is analyzed and demonstrated through theoretical results from domain generalization/adaptation. Moreover, extensive experiments are conducted to verify DBE's efficacy in reducing domain bias and its complementarity with other personalized federated learning methods.", "questions": "1. What's the difference between the numbers of layers that have memorized domain bias?\n2. Is it possible to show that, in the iid setting, the memory embedding is zero?", "limitations": "None.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written, effectively conveying the core idea in a clear and understandable manner.\n2. The authors address the persistent issue of non-iidness in federated learning, which remains a significant challenge. The visualization of embeddings pre- versus post-local updates provides valuable motivation for studying and understanding this problem.\n3. The proposed DBE (Domain Bias Embedding) approach is both simple and effective, demonstrating strong performance from both theoretical and empirical perspectives.\n4. The paper's exploration of the complementary nature of DBE with existing personalized federated learning methods is particularly interesting. This aspect enhances the overall impact and relevance of the paper.", "weaknesses": "1. The introduction of the memory module requires improvement. While the module itself is relatively simple, the presentation of its implementation and functioning could be made clearer. It is challenging to understand the process and how to implement it based on the initial explanation.\n2. The analysis of generalization risk, although provided, may have limited practical utility when it comes to the practical application of DBE. Further clarification or practical implications would enhance the relevance of this analysis.\n3. The experimental evaluation appears to lack diversity in the considered tasks. Including evaluations on tasks such as graph learning would provide additional evidence and make the results more compelling.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and constructive comments. We respond to your concerns as follows in the form of \"**[A weakness or question]** Our responses\". \n\n\n**[Improve the presentation of the memory module]** Thank you once again for your valuable suggestions. We will further refine the presentation of the memory module to enhance clarity. \n\n\n**[Practical application of $\\texttt{DBE}$]** For the practical application, we apply our proposed $\\texttt{DBE}$ to the IoT scenario on a popular Human Activity Recognition (HAR) dataset[1] with the HAR-CNN[2] model. HAR contains the sensor signal data collected from 30 users who perform six activities (WALKING, WALKING\\_UPSTAIRS, WALKING\\_DOWNSTAIRS, SITTING, STANDING, LAYING) wearing a smartphone on the waist. We show the results in R-xd4W-Table 1, where FedAvg+$\\texttt{DBE}$ still achieves superior performance. \n\nR-xd4W-Table 1: The test accuracy (\\%) on the HAR dataset. \n|  | Accuracy |\n|:-|:-:| \n| FedAvg |87.20±0.27 |\n| SCAFFOLD |91.34±0.43 |\n| FedProx |88.34±0.24 |\n| MOON |89.86±0.18 |\n| FedGen |90.82±0.21 |\n| Per-FedAvg |77.12±0.17 |\n| pFedMe |91.57±0.12 |\n| Ditto |91.53±0.09 |\n| FedPer |75.58±0.13 |\n| FedRep |80.44±0.42 |\n| FedRoD |89.91±0.23 |\n| FedBABU |87.12±0.31 |\n| APFL |92.18±0.51 |\n| FedFomo |63.39±0.48 |\n| APPLE |86.46±0.35 |\n| FedAvg+$\\texttt{DBE}$ |**94.53±0.26** |\n\n[1] Anguita D, Ghio A, Oneto L, et al. Human activity recognition on smartphones using a multiclass hardware-friendly support vector machine. Ambient Assisted Living and Home Care: 4th International Workshop, IWAAL 2012.\n\n[2] Zeng M, Nguyen L T, Yu B, et al. Convolutional neural networks for human activity recognition using mobile sensors. 6th international conference on mobile computing, applications and services. IEEE, 2014: 197-205.\n\n\n**[Experimental evaluation lacks diversity]** In the FL field, most of the existing methods (e.g., SCAFFOLD, MOON, FedGen, Per-FedAvg, pFedMe, FedPer, FedRoD, FedBABU, APFL, FedFomo, and APPLE) primarily consider classification tasks within CV field, while we consider both CV and NLP fields. Graph learning is not a widely explored task in the traditional FL and pFL methods we have considered and compared. Nevertheless, our $\\texttt{DBE}$ can eliminate domain bias in representation space for heterogeneous graphs on clients in theory, as the concept of representation is pervasive in deep learning tasks[3]. \n\n[3] Bengio Y, Courville A, Vincent P. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 2013, 35(8): 1798-1828.\n\n\n**[What's the difference between the number of layers that have memorized domain bias?]** Our Personalized Representation Bias Memory ($\\texttt{PRBM}$) module only contains a trainable vector $\\bar{z}^p _i$ without any additional layers. If you are inquiring about the influence of the number of layers in the feature extractors under different model splitting methods, as discussed in *Section 5.1.1 How to split the model*, it is important to note that feature extractors with different numbers of layers possess varying feature extraction capabilities. According to previous studies [4,5], it has been observed that representations extracted by higher layers, which are farther away from the input, tend to exhibit a higher degree of bias compared to the ones extracted by lower layers. \n\n[4] Yosinski J, Clune J, Bengio Y, et al. How transferable are features in deep neural networks?. Advances in neural information processing systems, 2014.\n\n[5] Luo M, Chen F, Hu D, et al. No fear of heterogeneity: Classifier calibration for federated learning with non-iid data. Advances in Neural Information Processing Systems, 2021.\n\n\n**[Is it possible to show that, in the iid setting, the memory embedding is zero?]** While the impact of our $\\texttt{PRBM}$ may be diminished in IID settings, its parameter vector $\\bar{z}^p _i$ cannot be zero as long as there exist variations among clients in the representation space. Even in ideal IID settings, where each client trains models with the same dataset, the extracted representations of the same input on different clients often differ. This discrepancy arises from the stochastic optimization process employed by deep neural networks, such as stochastic gradient descent (SGD). Moreover, the use of different random seeds on different machines can further contribute to the variation in the extracted representations during each communication iteration. In practical FL scenarios, the data distribution among clients is often non-IID and statistically heterogeneous [6]. \n\n[6] Li Q, Diao Y, Chen Q, et al. Federated learning on non-iid data silos: An experimental study. 2022 IEEE 38th International Conference on Data Engineering (ICDE). 2022."}]

Author rebuttal:
We sincerely thank the reviewers for their valuable feedback on our manuscript. We have provided detailed responses in the rebuttal field following each reviewer’s comments. In this global response field, we provide a PDF file that includes a figure named R-8bi2-Figure 1.

Meta-review: Initially, the reviewers were generally positive about the problem setting and the proposed method. They also appreciated the theoretical analysis and the thorough experimental evaluation. The author’s response clarified some aspects, and afterwards the reviewers did not see the need for further discussion. All reviewers recommend acceptance.
Decision: Accept (poster)