FedDebias: Reducing the Local Learning Bias Improves Federated Learning on Heterogeneous Data
Authors: Yongxin Guo
Xiaoying Tang
Tao Lin
Venue: Submitted to ICLR 2023 (2023)
Keywords: Federated Learning

Abstract: Federated Learning (FL) is a machine learning paradigm that learns from data kept locally to safeguard the privacy of clients, whereas local SGD is typically employed on the clients' devices to improve communication efficiency. However, such a scheme is currently constrained by the slow and unstable convergence induced by clients' heterogeneous data. 
In this work, we identify three under-explored phenomena of the biased local learning that may explain these challenges caused by local updates in supervised FL.
As a remedy, we propose FedDebias, a novel unified algorithm that reduces the local learning bias on features and classifiers to tackle these challenges. 
FedDebias consists of two components:
The first component alleviates the bias in the local classifiers by balancing the output distribution of models. 
The second component learns client invariant features that are close to global features but considerably distinct from those learned from other input distributions. 
In a series of experiments, we show that FedDebias consistently outperforms other SOTA FL and domain generalization (DG) baselines, in which both two components have individual performance gains.

Under review as a conference paper at ICLR 2023
FEDDEBIAS: REDUCING THE LOCAL LEARNING BIAS IMPROVES FEDERATED LEARNING ON HETEROGE- NEOUS DATA
Anonymous authors Paper under double-blind review
ABSTRACT
Federated Learning (FL) is machine learning paradigm that learns from data kept locally to safeguard the privacy of clients, whereas local SGD is typically employed the clients' devices to improve communication efficiency. However, such a scheme is currently constrained by the slow and unstable convergence induced by clients' heterogeneous data. In this work, we identify three under-explored phenomena of the biased local learning that may explain these challenges caused by local updates in supervised FL. As a remedy, we propose FedDebias, novel unified algorithm that reduces the local learning bias on features and classifiers tackle these challenges. FedDebias consists of two components: The first component alleviates the bias in the local classifiers by balancing the output distribution of models. The second component learns client invariant features that are close to global features but considerably distinct from those learned from other input distributions. In a series of experiments, we show that FedDebias consistently outperforms other SOTA FL and domain generalization (DG) baselines, in which both two components have individual performance gains.
INTRODUCTION
Federated Learning (FL.) is an emerging privacy-preservii distributed machine learning paradigm. The model is transmitted to the clients by the server, and when the clients have completed local training, the parameter updates are sent back to the server for integration. Clients are not required to provide local raw data during this procedure, maintaining their privacy. As the workhorse algorithm in FL, FedAvg (McMahan et al. (016)) proposes local SGD improve communication efficiency. However, the considerable heterogeneity between local client datasets leads to inconsistent local updates and hinders convergence.
Several studies propose variance reduction methods (Karimireddy et al. 2019 Das et al. 2020), or suggest regularizing local updates towards global models (Li et al., [201 2021) to tackle this issue. Almost all these existing works directly regularize models by utilizing the global model collected from previous rounds to reduce the variance or minimize the distance between global and local models LLi al. 2018b 202T). However, it hard to balance the trade-offs between optimization and regularization to perform well, and data heterogeneity remains an open question in the community, as justified by the limited performance gain, c.g. in our Table [
To this end, we begin by revisiting and reinterpreting the issues caused by data heterogeneity and local updates. We identify three pitfalls of FL, termed local learning bias, from the perspective of representation l 1) Biased local classifiers are unable to effectively classify unseen data (in Figure T(a),, due the shifted decision boundaries dominated by local class distributions; 2) Local features (extracted by local model) differ significantly from global features (similarly extracted by a centralized global model). even for the same input data. (c.f. Figure I(b):: and 3) Local features. even for data from different classes, are close to each other and cannot be accurately distinguished (c.f. Figure ((b)).
As remedy, we propose FedDebias, a unified method that leverages a globally shared pseudo-data and two key algorithmic components to simultancously address the three difficulties outlined above. The first component of FedDebias alleviates the first difficulty by forcing the output distribution of
Please refer section for more justification about the existence of our observations.

---

Under review as a conference paper at ICLR 2023
cara LOCAt
dartd lurie
Charc
(a) Bias in local classifier.
(b) Bias in local feature.
Figure I: Observation for learning bias: three pitfalls of FL on heterogeneous data with local updates. There are two clients in the figure (denoted by two colors), and cach has two classes of data (red and blue points). Figure Client I's dccision boundary cannot accurately classify data samples from client 2. Figure [ The difference between features extracted by client I's local feature extractor and global feature extractor sustainable large. However, client 2's local feature close enough to client I's, even for input data from different data distributions/cleents..
the pscudo-data to be close to the global prior distribution. The second component of FedDebias is designed for the second and third difficulties. In order to tackle the last two difficulties simultancously, develop min-max contrastive learning method to learn client invariant local features. More pre- cisely, instead of directly minimizing the distance between global and local features, we design a two- stage algorithm. The first stage learns a projection space- an operation that can maximize the differ- ence between global and local features but minimize local features of different inputs-to distinguish the features of two types. The second stage then debiases the features by leveraging the trained projec- tion space to enforce learned features that are farther from local features and closer to global features.
We examine the performance of FedDebias and compare it with other FL and domain generalization baselines on RotatedMNIST, CIFAR1O, and CIFAR100. Numerical results show that FedDebias consistently outperforms other algorithms by a large margin on mean accuracy and convergence speed. Furthermore, both components have individual performance gains, and the combined approach FedDebias yields the best results.
Contributions
We propose FedDebias, a unified algorithm that leverages pseudo-data to reduce the learning bias on local features and classifiers. We design two orthogonal key components of FedDebias to complement each other to improve the learning quality of clients with heterogeneous data. FedDebias considerably outperforms other FL and domain generalization (DG) baselines, as justified by extensive numerical evaluation.
RELATED WORKS
Federated Learning (FL). As the de facto FL algorithm, McMahan et al. (2016) Lin et al. (2020b)
propose to use local SGD steps to alleviate the communication bottleneck. However, the objective
inconsistency caused by the local data heterogeneity considerably hinders the convergence of FL
algorithms (Li et al. 2018b Wang et al. 2020 Karimireddy et al.) [2019] 2020; Guo el al., 2021). To address the issue heterogeneity in FL. series of projects has been proposed. FedProx (Li et al. 2018b) incorporates a proximal term into local objective functions to reduce the gap between the local and global models. SCAFFOLD (Karimireddy et al. 2019) adopts the variance reduction method on local updates, and Mime (Karimireddy et al. 2020 increases convergence speed by adding global momentum to global updates.
Data Augmentation in FL. To reduce data heterogeneity, some data-based approaches suggest sharing a global dataset among clients and combining global datasets with local datasets (Tuor et al. 2021 Yoshida et al. (211)). Some knowledge distillation-based methods also require global dataset (Lin al.) 2020al Li & Wang 2019), which is used to transfer knowledge from local models (teachers) to global models (students). Considering the impractical of sharing the global datasets in
FL settings, some recent research use proxy datasets with augmentation techniques. Astraea (Duan et al. 2019) uses local augmentation to create globally balanced distribution. XorMixFL (Shin ct al. 2020) encodes a couple of local data and decodes on the server using the XOR operator. FedMix (Yoon et al. 2021b) creates the protecee augmentation data by averaging local
2

---

Under review as a conference paper at ICLR 2023
batches and then applying Mixup in local iterations. VHL Tang et al. 2022) relies the created virtual data with labels and forces the local features to be close to the features of same-class virtual data. Our framework significantly outperforms VHL; unlike VHL, our solution has no label constraint and uses much less pseudo-data than VHI..
Distribution Robust FL. Domain generalization is a well-studied field, aiming to learn domain- robust models that perform well on unknown distributions. Some methods apply domain robust optimization methods (Sagawa et al., 2019; Hu & Hong 2013; Michel et al. (02T]) to minimize the
worst-case empirical error, and others propose learn domain invariant features (Ganin et al.) 2015)
Li. 2018c a Sun & Saenko 2016) by minimizing the distance of features from different domains.
By treating each client as a domain, some existing works tackle the FL problem as domain generalization problem. Several methods include optimizing the weights of different clients to lower the worst empirical error among all clients (Mohri et al.) [eng et al.) (02I). Huuan et al. (202T) assumes each client has two local datasets with different distribution, and the robustness is obtained by balancing the two local datasets. Xie et al. (222)) proposes collecting gradients from one segment of clients first, then combining them as a global gradient to reduce variance in the other segments. Reisizadeh et al. (2020) assumes the local distribution is perturbed by affine function. i.c., from to Az + 6. There are also some methods that aim to learn client invariant features (Peng et al. 2019) 2022 Shen et al. 2021: Sun al. 2022 Gan al. 2021). However, these methods are designed to learn a model that can perform well on unseen deployment distributions that differ from the (seen) clients' local distributions, which is beyond the scope of this paper.
Wang et al.
Recently, Moon (Li et al., 202T) has proposed employ contrastive loss reduce the distance between global and local features. However, their projection layer is only used as part of the feature extractor, and cannot contribute to distinguish the local and global features- crucial step identified by our investigation for a better model performance.
THE PITFALLS OF FL ON HETEROGENEOUS DATA DISTRIBUTIONS
FL and local SGD. FL is an emerging learning paradigm that supposes learning on various clients while clients can not exchange data to protect users' privacy. Learning occurs locally the clients, while the server collects and aggregates gradient updates from the clients. The standard FL considers the following problem:
= minwerd 4 [f()) -m
(1)
where fi(w) is the local objective function of client i, and p. is the weight for f((w). In practice, we set p. = |D.l/12) by default, where D; is the local dataset of client and D is the combination of all local datasets. The global objective function f(w) aims to find w that can perform well on all clients. In the training process of FL, the communication cost between client and server has become an essential factor affecting the training efficiency. Therefore, local SGD (McMahan et al.) (2066) has been proposed to reduce the communication round. In local SGD, clients perform multiple local steps before synchronizing to the server in each communication round.
The negative impact of local update steps. Despite the success of local SGD, the non-iid nature of clients' local data leads to local gradient inconsistency, which will slow down the convergence (Li al. 2018b Karimireddy et al.) 2019). A series of studies have proposed several methods for client heterogeneity to address this issue. One natural idea considers using the global gradient/model of previous rounds during the local updates to reduce variance or minimize the distance between
the global and local model (K (Karimireddy et al.) 2019 2020 Li et al. 2018b 2021). However, the performance of such algorithms limited in our challenging scenarios (as we shown in Table [T]. Using FedProx (Li et al., 2018b) 8b) as example, setting larger weights for proximal terms will hinder the further optimization steps of the local model, while setting a small weight will result in a marginal improvement of FedProx over FedAvg.
Bias caused by local updates. To mitigate the negative impact of local updates, we first identify the pitfalls of FL on heterogeneous data with a sufficient number of local updates and then design the algorithms to address the issues caused by the local updates.
The pitfalls can be justified by a toy experiment. More precisely, divide the MNIST dataset into two sets. The first dataset, denoted by X,, contains the 5 classes 0-4. The other dataset, denoted by X2, contains the remaining five classes. Then we train CNN model on X, for 10 epochs and store
3

---

Under review as a conference paper at ICLR 2023
03
10
03
(a) Global feature of X1, (X1) (b) Local feature of X1, Fi().)
(c) Local feature of X2, (X2)))
Figure 2: Observation for biased local features on a shared t-SNE projection space. Local updates will cause:
Large difference in local and global features for the same input data. Colored points in sub-figures (a) & (b) denote the global and local features of data from X1, and the same color indicates data from the same class. Notice that even for data from the same class (same color). the global and local features are clustered into distinct groups, implying considerable distance between global and local features even for the same input data distribution. e High similarity of local features for different imputs. Notice from sub-figure (b) & (c) that X, and X2 are two disjoint datasets (no data from the same class). However, the local features of X1 and X2 are clustered into the same group by t-SNE, indicating the relatively small distance between local features of different classes.
0.-
04-
04
99
(a) Sample from class
(b) All samples in X₂
Figure 3: Observation for biased local classifiers: the outpur distribution of the local classifiers will be dominated the local class distribution. The model is trained data X, and tested on data X2. The sub-figure (a) illustrates the model output distribution of sample belonging to Class The sub-figure (b) shows the total prediction distribution of all samples in X2. Results show that the biased local model will classify all samples into classes that are only present in the X1.
the feature extractor of the local model as F1. We also train a global model on the mixture of X1 and X2 (equal to centralized training) and store the feature extractor as Fg. We use Fg as the ground truth model because it works well over the entire dataset and does not suffer from local updates.
and [ show that local features differ from global features for the same input, and Figures ( and 2(c) show that local features are similar even for different input distributions. We define this observation as the "biased local feature". In detail, we calculate Fi (X,), Fi22)). 1X1). and (X2), and use I-SNE to project all the features to the same 2D space. P We can observe that the local features of data in X2 are so close to local features of data in X1, and it is non-trivial to tell which category the current input belongs to by merely looking at the local features.
Example 3.1 (Observation for biased local features). Figures ((a) and
Example 3.2 (Observation for biased local classifiers). Figure sshows the output of local model on data X2, where all data in X2 are incorrectly categorized into classes O to 4 of X1. The observation, i.e, data from classes that are absent from local datasets cannot be correctly classified by the local classifiers, refers the "biased local classifiers", More precisely, Figure [[a] shows the prediction result of one sample (class 8) and Figure 3(b) shows the predicted distribution of all samples in X2-
Based on Example 3.1 and Example 3.2 we summarize our observations and introduce the formal definition of "local learning bias" caused by local updates:
Definition 3.3 (Local Learning Bias). We define the local learning bias below:
Biased local feature: For local feature extractor Fel.), and centralized trained global feature extractor F.()), we have: 1) Given the data input X, Fx(X) could deviate largely from FX(X). 2) Given the input from different data distributions X1 and X2, F,(X;) could be very similar or almost identical to F(())..
Biased local classifier. After a sufficient number of iterations, local models classify all samples into only the classes that appeared in the local datasets.
We provide the results after using FedDehias in Appendix C33
4

---

Under review as a conference paper at ICLR 2023
It is worth to note that some related works also discussed the learning bias(Karimirede et al. 2019
et al. 2018b 2021]. However, we have an inherent difference compared with previous works. I) FedProx defines the local drifts as the difference between model weights ||w, - w/: ||. and SCAFFOLD
considers gradient difference as client drifts. Despite the theoretical success of these methods, these
Iwo methods usually have minor improvements on deep models (Tang al. 2022 Li al. 202T
Yoon et al., 2021a (Chen & Chao 2021] Luo et al.] [202I). 2) Though MOON is a crucial first step that minimizes the distance between global and local features, its performance gain is still limited due to the improper methodology design (cf. Table
FEDDEBIAS: REDUCING LEARNING BIAS IN FL BY PSEUDO-DATA
Addressing the local learning bias is crucial to improving FL on heterogeneous data, due to the bias discussed in Definition 3.3 To this end, we propose FedDebias shown in Figure novel framework that leverages the globally shared pseudo-data with two key components to reduce the local training bias, namely 1) reducing the local classifier's ' bias by balancing the output distribution of classifiers (component 1), and 2) an adversary contrastive scheme to learn unbiased local features (component 2).
4.1
OVERVIEW OF THE FEDDEBIAS
The learning procedure of FedDebias on each client i involves the construction of a global pscudo-data (c.f. Section 4.2), followed by applying two key debias steps in min-max approach to jointly form two components (c.f. Section 4.3 and 4.4 to reduce the bias in the classifier and feature, respectively. The min-max procedure FedDebias can be interpreted as first projecting features onto spaces that can distinguish global and local feature best, then 1) minimizing the distance between the global and local features of pseudo-data and maximizing distance between local features of pseudo-date and local data; 2) minimize classification loss of both local data and pseudo-data: Max Step:
max Ladu (Dp, Di) = Rx DD. [Feon (xp: Φg &., 2.:
(2)
Min Step:
min Cgnn(Dp D.) ((x,)) (x, y, R. w]] + XBx, ~D [Cdl (Xp: yp. Φ w)l 2:w0
+xlExp~Dpxxe DpiND, [Coon (Xp, 2 Ø 0)
Lets and Lcon represent the cross-entropy loss and a contrastive loss (will be detailed in Section respectively. D, denotes the distribution of local dataset at client Dp is that of shared pseudo- dataset, where y, is the pseudo-label of pseudo-data. The model is composed of feature extractor & and a classifier w, where the omitted subscript i and g correspond to the local client and global parameters, respectively (e.g., & denotes the feature extractors received from the server at the beginning of each communication round). We additionally use a projection layer 0 for the max step to project features onto spaces where global and local features have the largest dissimilarity. Apart from the standard classification loss of local data in Equation
(3). the second term aims to overcome the biased local classifier while the local feature is debiased by the third term.
The proposed FedDebias is summarized in Algorithm [ The global communication part is the same as FedAvg, and the choice of synchronizing the new pseudo-data to clients in each round is optional"
4.2 CONSTRUCTION OF THE PSEUDO-DATA
The choice of the aseudo-data in our FedDebias framework is arbitrary. For ease of presentation and taking the communication cost into account, we showcase two construction approaches below and detail their performance gain over all other existing baselines in Section
Random Sample Mean (RSM). Similar to the treatment in FedMix (Yoon et al. 2021b), one RSM sample of the pseudo-data is estimated through a weighted combination of a random subset of local samples, and the pseudo-label is ser to yp ) 1. Details can be found in Algorithm of Appendix B
3 shown in Figure 5(b). the communicatinn-effcicnt variant of FedDehias-iae only transferring pscudo- data at the beginning of the FI. training-is iss on par with the choice of frequent pscudo-data synchronization. We assume an uniform distribution for label and pseudo-data does not helong any particular classes.
5

---

Under review as a conference paper at ICLR 2023
Require: Local datasets Di..... Dy. pseudo dataset Dp where [pp = B. and B is the batch size, number of local iterations K, number of communication rounds T', number of clients chosen in each round M, weights used in designed loss X, μ. local learning rate n). Ensure: Trained model Wr, 0r, Φr-
Cas du.)
Cross-Entropy Lose
Contrastive Loss
y
Up
1: Initialize wu; 0u. Po-
2: for t 0,...,, 7
do
Nedchaa
3: Send wi, Q,, Lt. D2 (optional) to all clients.
Local Classilier
Projection Layer
4: for chosen client 1,.. 1...., M do
5:
wP w. 0. O., Φ. Φ. Φ Φ. for k ,,..., K do
6:
wix.)
0utSpp
7:
# Max Step
Local Feature Extractor
8:
0 = 0.-1
n1 Voaade
Extractor
9:
# Min Step
10:
w
VuCn.
11:
Φ
V Cgen.
12:
Sond w 0k
,
server.
Figure 4: Overview of FedDebias. We illustrate how calculate the three terms in Equation ( and (3) We calculate the cross-entropy loss of local data (x,y), and pseudo-data (xp. and use the local feature Φ,, (x), db, (xp). and global feature Φ5,xx,) for contrastive loss.
13:
W1+1 Σ w
4.11 M
DM 0%*
15:
ΦL+1 M7
1 EE Φ,
Algorithm Algorithm Framework of FedDebias
Mixture of local samples and the RSM of a proxy dataset (Mixture). This strategy relies on applying the procedure of RSM to an irrelevant and globally shared proxy data (refer to Algorithm 33. To guard the distribution distance between the pseudo-data and local data, one sample of the pseudo-data at each client is constructed by
X
K+I Xp + 1
- E6.1+ERATY
(4)
where Xp is one RSM sample of the global proxy dataset, and x2 and Yı correspond to the data and label of one local sample (vary depending on the client). K is a constant that controls the closeness between the distribution of pscudo-data and local data. As we will show in Section S setting K data efficient yet sufficient achieve good results.
4.3 COMPONENT 1: REDUCING BIAS IN LOCAL CLASSIFIERS
Due to the issue of label distribution skew the absence of some samples for the majority/minority classes, the trained local model classifier tends to overfit the locally presented classes, and may
further hinder the quality of feature extractor (as justified in Figure 3 and Definition (33))
As a remedy, here we implicitly mimic the global data distribution- by using the pseudo-data constructed in Section 4.2 to regularize the outputs and thus debias the classifier:
XE DD [Lets (Xpx Np: &.,w))
Note that the Component is appeared to be the second term of Equation
4.4 COMPONENT 2: REDUCING BIAS IN LOCAL FEATURES
In addition to alleviating the biased local classifier in Section here we introduce a crucial adversary strategy to learn unbiased local features.
Intuition of constructing an adversarial problem. As discussed in Definition 3.3 effective federated learning on heterogeneous data requires learning debiased local feature extractors that 1) can extract local features that are close to global features of the same input data; 2) can extract different local features for input samples from different distributions. However, existing methods that directly minimize the distance between global features and local features (Li et al. 2018b 2021) have limited performance gain (c.f. Table (1) due to the diminishing optimization objective caused by the indistingushabiiity between the global and local features of the same input. To this end, we propose to extend the idea of adversarial training
FL. scenarios:

---

Under review as a conference paper at ICLR 2023
1. We construct a projection layer as the critical step to distinguish features extracted by the global and local feature extractor: such layer ensures that the projected features extracted by local feature extractor will be close to each other (even for distinct local data distributions), but the difference between features extracted by the global and local feature extractor after projection will be considerable (even for the same input samples).
2. We can find that constructing such a projection layer can be achieved by maximizing the local feature bias discussed in Definition 3.3 More precisely, it can be achieved by maximizing the distance between global and local features of pseudo-data and simultaneously minimizing the distance between local features of pseudo-data and local data.
3. We then minimize the local feature biases (discussed in Definition 3.3) under the trained projection space, so as to enforce the learned local features of pseudo-data to be closer to the global features of pseudo-data but far away from the local features of real local data.
On the importance of utilizing the projection layer to construct the adversary problem. To construct the aforementioned adversarial training strategy, we consider using an additional projection layer to map features onto the projection space In contrast to the existing works that similarly add a projection layer (Li et al. 202T), we show that 1) simply adding a projection layer as part of the feature extractor has trivial performance gain (c.f. Fi)) 2) our design is the key step to reducing the feature bias and boosting the federated learning on heterogeneous data (c.f. Table 3).
Objective function design. We extend the idea of Li et al. (002)) and improve the contrastive loss initially proposed in simCLR (Chen et al. (222)) our challenging scenario. Different from previous works, we use the projected features (global and local) on pseudo-date as the positive pairs and rely on the projected local feature of both pseudo-data and local data as the negative pairs:
sin( (PI; (xp)), ((pp))))))))
Leon Xp, x, Øg B.. 0) = -log
(5)
sim (P(d) (P(() (ttpp.)PPP,,.))).
sm (x)
where the projection layer parameterized by 0, ri and 72 are temperature parameters, and sim is the cos-similarity function. Our implementation uses tied value for Tı and 72 for the sake of simplicity, but an improved performance may be observed by tuning these two.
5 EXPERIMENTS
5.1 EXPERIMENT SETTING
We elaborate the detailed experiment settings in Appendix
Baseline algorithms. We compare FedDebias with both FL baselines and commonly used domain generalization (DG) baselines that can be adapted to FL scenarios. Note that we do not consider domain generalization scenarios and include DG baselines to check DG methods can benefit FL on non-iid clients. For FL baselines, we choose FedAvg (McMahan et al. (216)), Moon (Li et al.) 202I), FedProx (Li et al. 2018b), VHL (Tang et al.) ).22), and FedMix (Yoon et al. 2021b). which most relevant to proposed algorithms. For DG baselines, choose GroupDRO (Sagawa) ct al. 2019). Mixup Yaan et al. 2020), and DANN (Ganin et al. 2015). Unless specially mentioned, all algorithms use FedAvg the backbone algorithm.
Models and datasets. We examine all algorithms on RotatedMNIST, CIFAR1O. and CIFAR100 datasets. We use a four-layer CNN for RotatedMNIST, VGG11 for CIFARIO, and Compact Convolutional Transformer (CCT (Hassani et al.] 202T) for CIFARI00. We split the datasets following the idea introduced in Yurochkin et al. (2019) Hsu et al. (019 ; Reddi et al. (2021), where we leverage the Latent Dirichlet Allocation (LDA) to control the distribution drift with parameter a. The pseudo-data is chosen as RSM by default, and we also provide results on other types of pseudo-data (c.f. Figure 5(c)). By default, we generate one batch of pseudo-data (64 for MNIST and 32 for other datasets) in each round, and we also investigate only generating one batch of pseudo-data at the beginning of training to reduce the communication cost (c.f. Figure (5b) Figure 5(c).. We use SGD optimizer (with m0 for CCT), and set the learning rate to 0.001 for RotatedMNIST. and 0.01 for other datasets. The local batch size is set to 64 for RotatedMNISTT and 32 for other datasets (following the default setting in DomainBed (Gulrajani & Lopez-Pazl Paz 2020 Additional
Such a projection layer is not part of the feature extractor or used for classification, shown in Figure

---

Under review as a conference paper at ICLR 2023
Table I: Performance of algorithms. We split RotatedMNIST. CIFARIO, and CIFARIO0 to clients with & = 0.1, and ran 1000 communication rounds on RotatedMNIST and CIFARIO for each algorithm, 800
communication rounds CIFARI00, We report the mean of maximum (over rounds) 5 test accuracies and the number of communication rounds to reach the threshold accuracy.
RotATedMNIST (CNN)
CIFARIO (VGGIT)
CIFAR100 (CCT)
Algorithm
Ace (%)
Rounds for 80%
Acc (%)
Rounds for 55%
Acc (%)
Rounds for 43%
FodAvg
82.47
828 (1.0X)
58.99
736 (1.X)
44.00
550 (1.0X)
FedProx
82.32
824 (1,0X)
59.14
738 (1.0X)
43.09
756 (0.7X)
Moon
82,68
864 (0,9X)
58.23
820 (0.9X)
42.87
766 (0.7X)
DANN
84.83
743 (1.1X)
58.29
782 (0.9X)
41.83
GroupDRO FedDebias (Ours)
80.23
910 (0.9X)
56.57
835 (0.9X)
44.34
444 (1.2X)
86.58
628 (1.3X)
64.65
496 (1.5X)
45.14
352 (1.5X)
FedAvg Mixup
82.56
840 (1.0X)
58.57
(0.9X)
46.37
(1.6X)
FedMix
81.33
902 (0,9X)
57.37
872 (0.8X)
42.69
FodDebias Mixup (Ours)
83.42
736 (1.IX)
65.32
392 (1.9X)
47.75
294 (1.9X)
Table 2: Comparison with VHI.. We split CIFARTO and CIFAR 100 10 clients with a 1, and run 1000 communication rounds on CIFARIO for each algorithm and communication rounds on CIFARI00. We report the mean of maximum (over rounds) 5 test accuracies and the number of communication rounds to reach the threshold accuracy. We set different numbers of virtual data to check the performance of VHIL, and pseudo-data only transfer once in FedDebias (32 pscudo-data), For CIFARIOM
choose Mixup the backbone.
(IARIO
CIFAR100 (CCT)
Algorithm
Acc (%)
Rounds for 60%
Acc (%)
Rounds for 46%
VHL (2000 virtual data)
61.23
886 (1.0X)
46,80
630 (1.0X)
VHL (20000 virtual data)
59.65
998 (0.9X)
46.51
714 (0.9X)
FedDebias (32 preudo data)
64.61
530 (1.8X)
554 (1.1X)
Table 3: Ablation studies of FedDebias on the effects of two components. We show the performance of two components, and remove the max step (Line 8 in Algorithm [T) of component 2. We split RotatedMNIST. CIFAR1O, and CIFARIOO to 10 clients with 0.1. We run 1000 communication rounds RotatedMNIST CIFARIO for each algorithm and 800 communication rounds on CIFAR100. We report the mean of maximum (over rounds) test accuracies and the number of communication rounds to reach the target accuracy.
RotatedMNIST (CNN)
CIFARTO (VGGIT)
CIFAR100 (CCT)
Algorithm
Ace (%)
Rounds for 80%
Acc (%)
Rounds for 55%
Acc (%)
Rounds for 43%
FedAvg
82.47
828 (1.0X)
58.99
736 (1.0X)
46.37
358 (1.0X)
Component 1
84.40
770 (1.1X)
64.32
442 (1.7X)
47.22
330 (1.1X)
Component 2
86,25
648 (1.3X)
63.44
483 (1.5X)
47.78
308 (1.2X)
+ w/o max step
81.24
926 (0.9X)
58.84
584 (1.3X)
43.50
512 (0.7X)
FedDehias
86.58
628 (1.3X)
64.65
496 (1.5X)
47.75
294 (1.2X)
results regarding the impact of typer-parameter choices and performance gain of FedDebias other batasett//eetinnsslvaaaaattto metrics can be found in Appendix C
5.2 NUMERICAL. RESULTS
The superior performance of FedDebias over existing FL and DG algorithms 0
Table | we
show the results of baseline methods as well as our proposed FedDebias algorithm. When comparing different FL and DG algorithms, we discovered that: 1) FedDebias performs best in all settings; 2) DG baselines only slightly outperform ERM, and some even worse; 3) Regularizing local models to global models from prior rounds, such as Moon and Fedprox, does not result in positive outcomes.
Comparison with VHI.. We vary the size of virtual data in VHL and compare it with our FedDebias in 2 our communication-effiiieee FedDebias only uses 32 pseudo-data and transfers pseudo-data once, while the communication- inteessiieeeeeeeeeeeeee VHL (Tang et al. (0222) requires the size of virtual data to be proportional to the number of classes and uses at least 2,000 virtual data (the authors suggest 2,000 for CIFAR 10 and 20,000 for CIFAR respectively in the released official code, and we use the default value of syper-parameters and implementation provided by the authors). We can find that 1) FedDebias always outperforms VHL. 2) FedDebias overcomes several shortcomings of VHL, e.g., the need for labeled virtual data and the large size of the virtual dataset.
WW give the results of CIFAR10 with ResNet18 in Tf Appendix
8

---

Under review as a conference paper at ICLR 2023
75-
42
93
Padligg Fllup
CruppRo
EIPARID
CIPARIDE
CINAROO
(a) w/ and w/o projection layer.
(b) communication of pscudo-data.
(c) choices of pscudo-data.
Figure 5: Ablation studies of FedDebias, regarding the impact of projection layer, the communication strategy of pseudo-data, and the choices of pseudo-data. In Figure 5[a) show the performance algorithms with/without the additional projection layer on CIFAR10 10 dataset with the VGG1 model. Figure 5(b) we show the performance of FedDebias on RotatedMNIST, CIFAR 10, and CIFAR 100 datasets when only transferring aseudo-data once (at the beginning of training) generating new pseudo-data each round. In Figure we
show the performance of FedDebias using different types of pseudo-data (all transfer once the beginning of training). We split each dataset into clients with 0.1 and used CNN for RotatedMMNST dataset, VGGII for CIFAR1O, and CCT for CIFAR 100. run 1000 communication rounds on RotatedMNISTT and CIFAR for each algorithm and 800 communication rounds on CIFAR 100. We report the mean of maximum 5 test accuracies.
Table Performance of FedDebias on CIFARIO with different number of clients. We split CIFARIO dataset into 10, 30, and 100 clients with = 0.1. run 1000 communication rounds for each algorithm on the VGG1 model, and report the mean of maximum 5 accuracies (over rounds) during training on test datasets.
Methods
Acc (%) with 10 clients
Acc (%) with 30 clients
Acc (%) with 100 clients
FedAvg
58.99 64.65
44.83
38.20
FedDebias
50.28
41.59
5.3 ABLATION STUDIES
Effectiveness the different components FedDebias. In Table 3, we show the improvements brought by different components of FedDebias. In order to highlight the importance of our two components, especially the max-step (c.f. Line in Algorithm[] [T] in component 2, we first consider two components of FedDebias individually, followed by removing the max-step. We find that: 1) Two components of FedDebias have individual improvements compared with FedAvg, but the combined solution FedDebias consistently achieves the best performance. 2) The projection layer is crucial. After removing projection layers, the component of FedDebias performs even worse than FedAvg; such insights may also explain the limitations of Moon (Li et al. 2021).
Performance of FedDebias CIFAR10 with different number of clients. In Table 4 vary the number of clients among (10,30 30, 100). For each setting, 10 clients are randomly chosen each communication round. FedDebias outperforms FedAvg by significant margin in all settings.
Reducing the communication cost of FedDebias. To reduce the communication overhead, reduce the size of pseudo-data, and only transmit one mini-batch of pseudo-data (64 for MNIST and 32 for others) once the beginning of training. In Figure 5(b) show the performance of FedDebias when aseudo-data only transfer to clients at the beginning of the training (64 pseudo-data for RotatedMNIST, and 32 for CIFARI0 and CIFAR 100). Results show that only transferring pseudo- data once can achieve comparable performance gain compared with transferring pseudo-data in each round. This indicates that the performance of FedDebias will not drop even we give small number of pseudo-data.
Regarding privacy issues caused by RSM. Because RSM may have some privacy issues, we consider using Mixture to protect privacy. In Figure 5(c) we show the performance of FedDebias with different types of pseudo-data (pseudo-data only transfer once at the beginning of training as in Figure 5(b)). Results show that: 1) FedDebias consistently outperforms FedAvg on all types of
pseudo-data data. 2) When using Mixture pseudo-date data and setting K 0 (Equation (4), FedDebias still have a performance gain compared with FedAvg, and a more significant performance gain can be observed by setting K =

---

Under review as a conference paper at ICLR 2023
REFERENCES
Martin Arjovsky, Léon Bottou, Ishaan Guirajani. and David Lopez-Paz. Invariant risk minimization, 2019. /atpivi./aivvvvv/a www. 197. 02893
Hong- You Chen and Wei-Lun Chao. On bridging generic and personalized federated learning for image classification. International Conference on Learning Representations, 2021.
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Rudrajit Das, Anish Acharya, Abolfazl Hashemi, Sujay Sanghavi, Inderjit S Dhillon, and Ufuk Topcu. Faster non-convex federated learning via global and local momentum. arXiv preprint MrXiv:2012.04061, 2020.
Yuyang Deng, Mohammad Mahdi Kamani, and Mehrdad Mahdavi. Distributionally robust federated averaging, 2021.
Moming Duan, Duo Liu, Xianzhang Chen, Yujuan Tan, Jinting Ren, Lei Qiao, and Liang Liang. Astraca: Self-balancing federated learning for improving classification accuracy of mobile deep learning applications. 2019 IEEE 37th International Conference on Computer Design (ICCD). Nov 2019. doi: 10.1 (009///cc445242219990000 ICCD46524 2019. 00038
URL http://da www.dooo..o..ooooo 1109/
Shaoduo Gan, Akhil Mathur, Anton Isopoussu, Fahim Kawsar, Nadia Berthouze, and Nicholas Lane. Fruda: Framework for distributed adversarial domain adaptation. IEEE Transactions on Parallel and Distributed Systems, 2021.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversariil training of neural networks. 2015. doi: 10.45500ARRXI..55500777.
URL hRRL///ppss/aaaaa...rrr/a////// 07818
Ishaan Gulrajani and David Lopez-Paz. In search lost domain generalization, 2020. URL
https: org/
Yongxin Guo, Tao Lin, and Xiaoying Tang. Towards federated learning on time-evolving heteroge- neous data. arXiv preprint 22.i::2//2.1222 2021.
Ali Hassani, Steven Walton, Nikhil Shah, Abulikemu Abuduweili, Jiachen Li, and Humphrey Shi. Escaping the big data paradigm with compact transformers. 2021. URL /ttts:://axxt /arxiv org/abs/21 2104. 05704
Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint 100.03590, 2019.
Zhaolin Hu and L Jeff Hong. Kullback-leible divergence constrained distributionally robust opti- mization. Available at Optimization Online, pp. 1695-1724, 2013.
Feihu Huang, Junyi Li, and Heng Huang. Compositional federated learning: Applications in distributionally robust averaging and meta learning, 2021.
Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank J. Reddi, Sebastian U. Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning, 2019.
URL httts:://aaatt:t org. abs/1910. 06377
Sai Praneeth Karimireddy, Martin Jaggi. Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian U Stich, and Ananda Theertha Suresh. Mime: Mimicking centralized stochastic algorithms in federated learning. arXiv preprint arXiv:2008.0:36 2020.
David Krueger, Ethan Caballero, Joem--Henii Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville, Out-of-diittibbttion generalization via risk extrapola- tion htt.://ttps///arra....oo///////a
0068
10

---

Under review as a conference paper at ICLR 2023
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M. Hospedales. Learning to generalize: Meta- learning for domain generalization, 2017. URL https:////// www.v. http//bb//1111 /tt.g////AaL...oorr//////1.....0 0334/
Daliang Li and Junpu Wang. Fedmd: Heterogenous federated learning via model distillation. arXiv preprint arXiv:1910.03581, 2019.
Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C. Kot. Domain generalization with adversarial feature learning. In 2018 IEEE Conference on Computer Vision and Puttern
Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22. 2018, pp. 5400-5409. Computer Vision Foundation / IEEE Computer Society, 2018a. doi: 20.1109//V 00566.
thecvf. www/contentttttteeooooooooo cvpr 1eerr_201ahhttmmmem Domain Generalization Bttpiithh___PPPP_ BBAVVPR_2211___PPPe html
Qinbin Li, Bingsheng He, and Dawn Song, Model-contrastive federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks, 2018b. URL htt.tp::/aaxxi.oog//// 1812. 06127
Ya Li, Xinmei Tian, Mingming Gong, Yajing Liu, Tongliang Liu, Kun Zhang, and Dacheng Tao. Deep domain generalization via conditional invariant adversarial networks. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 624-639, 2018c.
Tao Lin, Lingjing Kong, Sehastian U Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 2351-2363, Curran As- sociates, Inc., 2020a. URL httptttpppppptteetteeeeeeeeeeeee
neurips..p.... CC/ http://ppprr/22//11/1//// htt.:eeeee818229994444444440 [caafhisccccaa11eee9900fCCeeeeee pdf
Tao Lin, Sebastian U. Stich, Kumar Kshitij Patel, and Martin Jaggi. Don't use large mini-batches. use local sgd. In International Conference on Learning Representations, 2020b. URL https
//openreview wtt.://eeeooommm11BBeeeo00001100
Mi Luo. Fei Chen, Dapeng Hu, Yifan Zhang, Jian Liang, and Jiashi Feng. No fear of heterogeneity: Classifier calibration for federated learning with non-iid data. Advances in Neural Information Processing Systems, 34:5972-5984, 2021.
H. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera Arcas. Communication--ffiiio. learning of deep networks from decentralized data. 2016. doi: 10.48550/ ARXIV.1602.05629 URL htt.tttss:::/artttt
602. 05629
Paul Michel, Tatsunori Hashimoto, and Graham Neubig. Modeling the second player in distribution- ally robust optimization. arXiv preprint arXiv:2103.10288, 2021.
Mehryar Mohri, Gary Sivek, and Ananda Theertha Suresh. Agnostic federated learning, 2019.
Xingchao Peng, Zijun Huang, Yizhe Zhu, and Kate Saenko. Federated adversarial domain adaptation, 2019. URL httptttts///aaxtt. wwwp:iiwwwrgraaaaaaaa abs////////////// /etp:////..1.... 991..02054
Sashank J. Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečný, Sanjiv Kumar, and Hugh Brendan McMahan. Adaptive federated optimization. In International
Conference on Learning Representations, 2021. (RRLhhtts:///oooo http:////pppeeeeeeeeeeeeeeeeeeee
[d=1kFG31B1305
Amirhossein Reisizadeh, Farzan Farnia, Ramtin Pedarsani, and Ali Jadbabaie. Robust federated learning: The case of affine distribution shifts, 2020.
Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization, 2019. URL htt.::t/t/seatt www.vvrrgggeee htwrr//aa///// /ttpa//////1111/////
11

---

Under review as a conference paper at ICLR 2023
Yan Shen, Jian Du, Han Zhao, Benyu Zhang. Zhanghexuan Ji, and Mingchen Gao. Fedmm: Saddle point optimization for federated adversarial domain adaptation. arXiv preprint arXi::2110.08477, 2021.
MyungJae Shin, Chihoon Hwang, Joongheon Kim, Jihong Park, Mehdi Bennis, and Seong-Lyun Kim. Xor mixup: Privacy-preserving data augmentation for one-shot federated learning, 2020.
Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation, 2016. URL BKKKhhtpss///axxx.orr///////
01719
Yuwei Sun, Ng Chong, and Ochiai Hideya. Multi-source domain adaptation based on federated knowledge alignment. arXiv preprint arXiv:2203. /635,, 2022.
Zhenheng Tang, Yonggang Zhang, Shaohuai Shi, Xin He, Bo Han, and Xiaowen Chu. Virtual homogeneity learning: Defending against data heterogeneity in federated learning, arXiv preprint arXiv:2206.022 4645,, 2022.
Tiffany Tuor, Shiqiang Wang, Bong Jun Ko, Changchang Liu, and Kin Leung. Overcoming noisy and irrelevant data in federated learning. In 2020 25th International Conference Pattern Recognition (ICPR), pp. 5020-5027. IEFE, 2021.
Bin Wang, Gang Li, Chao Wu, WeiShan Zhang, Jiehan Zhou, and Ye Wei. A framework for self- supervised federated domain adaptation. EURASIP Journal Wireless Communications and Networking, 2022(1):1-17, 2022.
Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H Vincent Poor. Tackling the objective inconsistency problem in heterogeneous federated optimization. arXiv preprint 200:20007.048.... 2020.
Jiahao Xie, Chao Zhang, Zebang Shen, Weijie Liu, and Hui Qian. Efficient cross-device federated learning algorithms for minimax problems, 2021.
Shen Yan, Huan Song, Nanxiang Li, Lincan Zou, and Liu Ren. Improve unsupervised domain adaptation with mixup training. 2020. URL /ttctpss///rxx..oorr/b//////////
Jaehong Yoon, Wonyong Jeong, Giwoong Lee, Eunho Yang, and Sung Ju Hwang. Federated continual learning with weighted inter-client transfer. In International Conference on Machine Learning, pp. 12073-12086. PMLR, 2021a.
Tehrim Yoon, Sumin Shin, Sung Ju Hwang, and Eunho Yang. Fedmix: Approximation of mixup under mean augmented federated Icarning, 2021b.
Naoya Yoshida, Takayuki Nishio, Masahiro Morikura, Koji Yamamoto. and Ryo Yonetani. Hybrid- fl: Cooperative learning mechanism using non-iid data in wireless networks. arXiv preprint 2019.
Mikhail Yurochkin, Mayank Agarwal, Soumya Ghosh, Kristjan Greenewald, Nghia Hoang, and Yasaman Khazaeni. Bayesian nonparametric federated learning of neural networks. In International Conference on Machine Learning, pp. 7252-7261. PMLR, 2019.
12

---

Under review as a conference paper at ICLR 2023
A EXPERIMENT DETAILS
Framework and baseline algorithms. In addition to traditional FL methods, we aim to see if domain generalization (DG) methods can help increase model performance during FL training. Thus, we use the DomainBed benchmark (Gulrajani & Lopez-Par Paz 2020). which contains a series of regularly used DG algorithms and datasets. The algorithms in DomainBed can be divided into three categories:
Some algorithms can't be applied in FL scenarios due the privacy concerns, for example, MLDG (Li et al. 2017), MMD (Li et al. 2018a), CORAL (Sun & Saenko (2016).
Infeasible methods:
VREx (Krueger al. 2020) that need features data from each domain in each iteration.
Feasible methods (with limitations): Some algorithms can be applied in FL. scenarios with some
limitations. For example, DANN (Ganin et al. 2015). CDANN (Li et al., 2018c) require knowing the number of domains/clients, which is impractical in the cross-device setting.
Feasible methods ( without limitations): Some algorithms can be directly applied in FL settings.
For example, ERM, GroupDRO (Sagawa et al. 2019), Mixup (Yan et al. 2020). and IRM
(Arjovsky et al. 2019).
We choose several common used DG algorithms that can easily be applied in Fl scenarios, including ERM, GroupDRO (Sagawa et al. 2019), Mixup (Yan al. 2020). and DANN (Ganin et al.
2015
For FL baselines, we choose FedAAg (McMahan et 20.1 (equal to ERM), Moon (Li et al.
2021
FedProx (Li et al. 2018b), SCAFFOLD (Karimiredyy et al. 2019) and FedMix (Yoon et al.. which are most related to our proposed algorithms.
202lb
Notice that some existing works consider combining FL. and domain generalization. For example, combining DRO with FL (Mohri et al. 2019 Deng et al. (202I) and combine MMD or DANN with FL (Peng et al. 2019: Wang al. [2022] Shen et al. (021). The natural idea of the former two DRO-based approaches is the same as our GRO implementations, with some minor weight updates differences; the target of the later series of works that combine MMD or DANN is to train models to work well on unseen distributions, which is orthogonal with our consideration (overcome the local heterogeneity). To check the performance of this series of works, we choose to integrate FL and DANN into our environments.
Notice that we carefully tune all the baseline methods. The implementation detail of each algorithm is listed below:
GroupDRO: The weight of each client is updated by wf+1 = w exp(0.017)), where i is the loss value of client i round t.
Mixup: Local data is mixed by x = Ax; + (1 - -)x,,, and À is sampled by Beta Beta(0.2, 0.2).
DANN: Use a three-layer MLP as domain discriminator, where the width of MLP is 256. The
weight of domain discriminate loss is tuned in {0.01. 0.1,
FedProx: The weight of proximal term is tuned in {0.001, 0.01, 0.1).
Moon: The projection layer is a two-layer MLP, the MLP width is setting to 256, and the output dimension 128. We tuned the weight of contrastive loss in {0.01, 0.1, 103.
FedMix: The mixup weight À used in FedMix is tuned in {0.01, 0.1, 0.2]. we construct 64 augmentation data in each local step for RotatedMNIST, and 32 samples for CIFARIO and CIFAR100..
VHL: We use the same setting as in the original paper, with the weight of augmentation classifica- tion loss a 1.0, and use the "proxy_alige aligg_losss" loss" provided by the authors for feature alignment. Virtual data is generated by untrained style-GAN-v2, and we sample 2000 virtual data for CI- FARI0 and RotatedMNIST; 20000 virtual data for CIFARIOM follow the default setting of the original work. To make a fair comparison, we sample 32 virtual samples in each local step for CIFARIO and CIFAR100.
FedDebias: We use three-layer MI.P as the projection layer, the MLP width is set to 256, and the output dimension is 128. By default, we set 71 = 72 = 2.0, the weight of contrastive loss μ = 0.5, and the weight of AugMean x= = 1.0 on MNIST and CIFAR 100, X 0.1 on CIFAR 10 and PACS. We sample pseudo-data in each local step for RotatedMNIST and 32 samples for CIFARIO and CIFAR100.
Feature correction when using proxy datasets to construct pseudo-data. When using proxy datasets to construct the pseudo-data, we additionally mix up local data with pseudo-data to make the pseudo-data not too far from the local distribution. However, the pseudo-data will have large
overlap with local data after the mixup. Then the exp simm
term in Equation (
13

---

Under review as a conference paper at ICLR 2023
which is used to maximize the distance between local features of local data and pseudo-data, will be meaningless. To address this issue, we change this term to
sim (xp) - pr y) ((xx),
(6)
where di, is the pseudo-labed of Ty, and y is the one-hot label of local data æ. Then we can minimize the relationship between x and up caused by the mixup with local data.
Datasets and Models. For datasets, choose RotatedMNIST, CIFARIO, CIFAR100. and PACS. For RotatedMNIST, CIFAR1O, 10, and CIFARI00, we split the datasets following the idea introduced in Yurochkin et al. (2019); Hss al. (2019); Reddi et al. (202T), where we leverage the Latent Dirichlet Allocation (LDA) to control the distribution drift with parameter a. Larger a indicates smaller non-üidness. We divided each environment into two clients for PACS, with the first client containing data from classes 0-3, and the second client containing data from classes 4-6.
Unless specially mentioned, we split RotatedMNIST. CIFARIO, and CIFAR 100 to 10 clients and set = 0.1. For PACS, we have clients instead. Notice that for each client of CIFAR 10, we utilize special transformation, i.e., rotation to the local data, to simulate the natural shift. In detail:
RotatedMNIST: We first split MNIST by LDA using parameter
0.1 10 clients, then for each client, we rotate the local data by {0, 15, 30, 45, 60, 75, 90, 105, 120, 135].
CIFARIO: We first split CIFARIO by LDA using parameter a 0.1 N clients. Then for each client, we sample E IR 10 from Dir(1.0). For each image in local data, we sample an angle in {0, 15, 30, 45, 60, 75, 90, 105, 120, 135) by probability q. and rotate the image by the angle. Clean CIFARIO: Unlike the previous setting, we do not rotate the samples in CIFARIO (no inner-class non-iideess). 100: We split the CIFARI00 by LDA using parameter a 0.1, and transform the train data using RandomCrop, RandomHorizzotallFiip,, and normalization.
Each communication round includes 50 local iterations, with 1000 communication rounds for RotatedMNIST and CIFARIO, 800 communication rounds for CIFAR100, and 400 communication rounds for PACS. Notice that the number of communication rounds is carefully chosen, and the accuracy of all algorithms does significantly improve after the given communication rounds.
The public data is chosen as RSM (Yoon et al. (02Tb) by default, and we also provide results on other proxy datasets. We utilize four-layer CNN for MNIST, VGG1 1 for CIFAR10 10 and PACS, and CCT (Hassani al. 2021) (Compact Convolutional Transformer, ect cct_ 3x1 32 c100) for CIFAR100
For each algorithm and dataset, employ SGD as the optimizer, and set learning rate (r 0.001 for MNIST, and Ir 0.01 for CIFARIO, CIFAR 100, and PACS. When using CCT and ResNet, we set momentum 0.9. We set the same random seeds for all algorithms. We set local batch size to 64 for RotatedMNISTT, and 32 for CIFAR1O, CIFAR100, and PACS.
B DETAILS OF AUGMENTATION DATA
We use the data augmentation framework the same as FedMix, as shown in Algorithm 2 For each local dataset, we upload the mean of each M samples to the server. The constructed augmentation data close to random noise. shown in Figure 6 we randomly choose one sample the augmentation dataset of CIFAR10 dataset.
Algorithm 2 Construct Augmentation Data
Require: local Datasets Di.... Dx. number augmentation data for each client K, number of samples to construct one augmentation sample M.
Ensure: Augmentation Dataset D,.
1: Initialize D, 0.
2: for 1, ..... N do
3:
for k = 1..... K do
4: Randomly sample 2t ......, aM from D..
f ΣM= TM-
6:
D, (#
14

---

Under review as a conference paper at ICLR 2023
Algorithm 3 Construct Augmentation Data by Proxy Data
Require: Proxy Datasets Drroz, number of augmentation data K, number of samples to construct augmen- tation sample M.
Ensure: Augmentation Dataset Dp.
I: Initialize Dr = 0.
2: for k I.....K K do
3:
Randomly sample ,.,..., J.M from Drror.
4: M Σ 2m JM-
5: D, Dp (2}
Figure show 20 augmentation data CIFAR1O dataset here. Notice that the augmentation data is close random noise and can not be classified as any class.
64
Accuracy
Accurarcy
Accuracy
May
02
Communication Rounds
Communication Rounds
Communication Rounds
(a) Convergence
Rotat-(b (b) Convergence CIFARIO (c)
Convergence
Cl-
edMNIST
FAR100
Figure Convergence curve of algorithms on different datasets.
15

---

Under review as a conference paper at ICLR 2023
C ADDITIONAL RESULTS
C.I RESULTS WITH ERROR BAR
In this section, we report the performance of our method FedAug and other baselines with an error bar to verify the performance gain of our proposed method.
Table 5: Performance of algorithms with error bar. All examined algorithms use FedAvg the backbone. We rum 1000 communication rounds on RotATedMNIST and CIFARIO for each algorithm. For cach algorithm, we run three different trials with different random seeds. each trial, we report the mean of maximum accuracies for test datasets and the number of communication rounds to reach the threshold accuracy.
RotatedMNIST
CIFARIO
Algorithm
Acc (%)
Rounds for 80%
Acc (%)
Rounds for 55%
ERM (FedAvg)
82.78 + 0.38
821 (1.0X)
58.97 0.30
742 (1.0X)
DANN
84.67 1 0.46
754 (1.1X)
58.98 + 0.61
747 (1.0X)
Mixup
82.38 + 0.07
853 (1.0X)
58.32 + 0.33
822 (0.9X)
GroupDRO
80.65 + 0.53
929 (0.9X)
56.72 = 0.26
840 (0.9X)
FedDebias (Ours)
87.05 + 0.44
637 (1.3X)
64.62 + 0.32
374 (2.0X)
Table 6: Performance of algorithms on CIFAR10. We split CIFARIO dataset to 10 clients with = 0.1, without additional rotation. For each algorithm, we run 1000 communication rounds on ResNet18 (with group normalization), and set local steps 50. Note that momentum 0.9 for ResNet18.
FedAvg
FedProx
Moon
VHL
FedDebias (ours)
Accuracy (ResNet18)
45.91
46.28
43.85
43.7
47.29
C.2 ABLATION STUDY OF FEDDEBIAS
Values of Ti and 72 in Componennt 2. In this paragraph, we investigate how the value of and T2 affect the performance of the second component of FedDebias. In table we show the results on RotATed-MNISTT dataset with different weights T₁ and 72. Results show that: I) Setting 72 0 which only minimizes the distance of global and local features, has significant performance gain compare with ERM. However, adding T2 further improve the performance. 2) The best weight on Rotated-MNNST dataset is v = 2.0 and 72 = 0.5.
Table 7: Performance Component of FedDebias under different values TI, 72. run 1000 communication rounds on RotatedMNIST dataset. For each setting, we run three different trials with different random seeds. For each trial, we report the mean of maximum 5 accuracies for test datasets and the number of communication rounds to reach the threshold accuracy.
Acc (%)
Rounds for 80%
Rounds for 85%
T1
T2
2.0
0.0
86.11 + 0.77
746
933
2.0
0.1
86.22 + 0.33
753
920
2.0
0.5
647
851
2.0
1.0
86.25
+ 0.87
705
922
2.0
2.0
86.01 + 0.33
680
932
Weights of the first component of FedDebias. In this paragraph, investigate how the weights of the first component of FedDebias affect the performance of models in table 9
Domain robustness of FL and DG algorithms. We also hope that our method can increase the model's robustness because it expects to train client invariant features. Therefore, we calculate the worst accuracy test datasets of all clients/domains and report the mean of each algorithm's
16

---

Under review as a conference paper at ICLR 2023
Table Performance of FedDebias under different values of We 1000 communication rounds the CIFARIO dataset. each setting, we run three different trials with different random seeds. For each trial, we report the mean of maximum 5 accuracies for test datasets and the number of communication rounds to reach the threshold accuracy.
Acc (%)
Rounds for 55%
Rounds for 60%
T1
T2
2.0
0.0
64.05 = 0.27
390
563
2.0
0.5
64.26 + 0.47
382
585
2.0
1.0
64.77 + 0.24
374
533
2.0
2.0
64.62 + 0.32
374
541
Table 9: Performance of component under different weights. We 1000 communication rounds the CIFARIO dataset. For each setting, we run three different trials with different random seeds. For each trial, we report the mean of maximum 5 accuracies for test datascts and the number communication rounds to reach the threshold accuracy. We use À as the weight of the first component of FedDebias.
À
Acc (%)
Rounds for 55%
Rounds for 60%
0.1
64.12 + 0.27
442
591
0.5
64.92 + 0.46
385
536
1.0
64.50 + 0.34
379
565
top 5 worst accuracies in Table show the domain robustness of algorithms. We have the following findings: 1) FedDebias significantly outperforms other approaches, and the improvements of FedDebias FedAvg become has role in learning a nomain-invariann feature and improving robustness, as evidenced by this finding. 2) Under these settings, DG baselines outperform FedAvg. This finding demonstrates that the DG algorithms help to enhance domain robustness.
significant than the mean accuracy in Table FedDebias
Table 10: Performance of algorithms. All examined algorithms use FedAvg the backhone. We 1000 communication rounds RotATedMNIST and CIFARIO for each algorithm, 800 communication rounds CIFARI00 and communication rounds for PACS. We report the mean of maximum accuracies for test datasets and the number of communication rounds to reach the final accuracy of ERM.
RotatedMNIST
CIFARIO
PACS
Algorithm
Acc (%)
Rounds (Spced up)
Acc (%)
Rounds (Speed up)
Acc (%)
Rounds (Speed up)
ERM (FedAvg)
82.47
828 (1.0X)
58.99
736 (1.0X)
64.03
168 (1.0X)
FedProx
82.32
824 (1.0X)
59.14
738 (1.0X)
65.
168 (1.0X)
SCAFFOLD
82.49
814 (1.0X)
59.00
738 (1.0X)
64.49
168 (1.0X)
FedMix
81.33
902 (0.9X)
57.37
872 (0.8X)
62.14
228 (0.7X)
Moon
82.68
864 (0.9X)
58.23
820 (0.9X)
122 (1.4X)
DANN
84.83
(1.1X)
58.29
782 (0.9X)
64.97
109 (1.5X)
Mixup
82.56
840
58.57
826 (0.9X)
64.36
210 (0.8X)
GroupDRO
80.23
910 (0.9X)
56.57
835 (0.9X)
64.40
170 (1.0X)
FedAug (Ours)
86.58
628 (1.3X)
64.65
496 (1.5X)
65.63
100 (1.7X)
C.3 T-SNE AND CLASSCIFER OUTPUT
As the setting in Figure and Figure we investigate if the two components of FedDebias will help for mitigating the proposed bias on feature and classifier. Figure show the features after the second component of FedDebias, which implies this component can significantly mitigate the proposed feature bias: 1) on the seen datasets, local features are close to global features. 2) on the unseen datasets, the local feature is far away from that of seen datasets. Figure shows the output of the local classifier after the first component of FedDebias on unseen classes. Notice that compared with Figure 3 the output is more balanced.
17

---

Under review as a conference paper at ICLR 2023
(a) Global feature X1, (X)) (b) Local feature of X1, (X)) (c) Local feature of X2. Fi (X2)
Figure Features after the second component of FedDebias.
24--
Figure 9: Classifier output after the first component of FedDehias unseen classes.
(a) Global feature X1, (X1) (b) Fine-tuned Local feature of X1,()) Fine-tuned Local feature of X2, (i())
(X()
Figure 10: Fine-tuned local features after 10 local epochs.
30
(a) Global feature of X1, F, (X1) (b) Fine-tuned Local feature of (c.()) Fine-tuned Local feature of X2. Fi(X2)
FX)))
Figure 11: Fine-tuned local features after 20 local epochs.
18

---

Under review as a conference paper at ICLR 2023
Table 11: Worst Case Performance of algorithms. All examined algorithms FedAvg the backbone. We run 1000 communication rounds on RotatedMNIST and CIFARIO for each algorithm, rounds CIFAR100. and 400 communication rounds for PACS. We calculate the worst accuracy for all clients in each round and report the mean of the top 5 worst accuracics for each method. Besides, we report the number of communication rounds to reach the final worst accuracy of FedAvg.
Algorithm
RotatedMNIST
CIFARIO
PACS
Acc (%)
Rounds (Spced up)
Acc (5)
Rounds (Speed up)
Ace (%)
Rounds (Speed up)
FRM (FedAvg)
66.60
816 (1.0X)
41.30
846 (1.0X)
42.79
170 (1.0X)
FodProx
65.88
780 (1.0))
41.84
840 (1.0X)
42.82
170 (1.0X)
SCAFFOLD
66.72
804 (1 (1.0X)
40.88
840 (1.0X)
41.63
170 (1.0X)
FedMix
60.52
(0.9X)
28.44
38,00
Moon
66.18
866 (0.9X)
40.34
908 (0.9X)
41.59
66 (2.6X)
DANN
67.85
753 (1.1X)
43.38
747 (1.1X)
40.51
59 (2.9X)
Mixup
66.25
836 (1.0X)
40.32
984 (0.9X)
41.89
252 (0.7X)
GroupDRO
68,53
568 (1.4X)
46.90
656 (1.3X)
43.18
246 (0.7X)
FedDebias (Ours)
77.13
(1.3X)
48.94
632 (1.3X)
43.99
58 (2.9X)
(a) Global feature X1, Fy (X1)
(b) Fine-tuned Local feature X1,()) Fine-tuned Local feature X2. (X2)
(X,)
Figure 12: First train global model the whole dataset for epoch, then report local features after local epochs.
In Figure 10 and Figure show the local learning bias when local model has better feature initialization. We copy the feature extractor of global model to local models, and randomly initialize local classifiers. Results show that: 1) The drifts between global and local features even has good feature initialization. 2) The local features of unseen data are less relevant to the local features of seen data compare with training from scratch. This indicates that such problem will be mitigated after enough training rounds. 3) The drifts between global and local features increase as the number of local epochs increases.
still significant
We also investigate if our observation remains for different stages of global models. In this experiment, we use CIFAR10 dataset, and train global model for 1, 3, 10 epochs on the whole dataset to obtain 29.74%, 38.65%, 49.2 49.28% global accuracy, then we directly copy global models to clients (including classifier). We fine-te fine-tune the global models for 10 local epochs, results are shown in Figure I2 Figure 13 and Figure 14 Results show that: For not well-trained global models, difference between global features on the same input and similarity between local features of different inputs are both significant.
(a) Global feature X1, F, (X1)) (b) Fine-tuned Local feature X1.(c) Fine-tuned Local feature of X2- (Xa)
(X))
Figure 13: First train global model the whole dataset for 3 epochs, then report local features after 10 local epochs.
19

---

Under review as a conference paper at ICLR 2023
(a) Global feature X1, (X1) (b) Fine-tuned Local feature (1.()) Fine-tuned Local feature of X2. (X2)
(i))))
Figure 14: First train global model on the whole dataset for 10 epochs, then report local features after 10 local epochs.
Max step: Maximize distance between global and local features (Distinguish blased and unbiased features)
Min step: Learn unbiased local features
Projected unbiased global
Projected unbiased global
Unbiased global features
Unbiaaed global features
Unblased learning
direction
Projected unbiased
local
Brased local features
Blased local features
Projection layer
Projection layer
Projected blased local features
Projected biased local festures
Figure 15: Illustration of the min-max process. In the left side of the figure, we show that the projection layer maximizes the distance between biased and unbiased features. In the right side of the figure. we show that the unbiased local feature trained by forcing projected features closer to unbiased features.
C.4 ILLUSTRATION OF MIN-MAX PROBLEM
In Figure 15 we illustrate the intuition to use the proposed min-max process. The projection layer is used distinguish biased and unbiased features that can be distinguished well the original feature space, and the min step is to learn unbiased local features that close to unbiased features on the projected spaces.
20


Reviews:
[{"review_id": "ckb6ScaNTp", "reviewer": "Reviewer_14m7", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on federated learning with heterogeneous data. Specifically, this paper identifies the negative effects of learning with heterogeneous in the federated framework from a representation learning perspective, which is termed local learning bias in its discussion and analysis. To handle with that challenge, the authors propose FedDebias, to reduce the local learning bias on features and classifiers. They conducted various experiments to verify the effectiveness of the proposed method and claimed FedDebias had outperformed other SOTA FL and domain generalization baselines.", "summary_of_the_review": "In summary, this paper proposed FedDebias to tackle learning with heterogeneous data. The presentation of the current version is encouraged to improve by making the focused problem setting clearer and adding more discussion except for the empirical effectiveness. ", "strength_and_weaknesses": "Strength:\n1. The focused general problem, i.e., federated learning with heterogeneous data, is significant for federated learning and is very practical.\n2. This paper provides further analysis of the pessimistic from learning with heterogeneous data, which are being unable to classify unseen data, local features differ from global features, and local features cannot be accurately distinguished.\n3. The proposed FedDebias shows empirical effectiveness in the experiments.\n\nWeaknesses:\n1. The motivation of the proposed methods seems to be based on the illustration (e.g., Figure 1), it lacks the empirical justification or some related reference to the direct evidence which can support the existence of the raised issues. Could the author provide more quantitative results to describe the issues, like measuring the local learning bias by some metrics?\n2. The three identified \"pitfalls of FL\" in this paper have been explored or discussed in previous FL literature [1-3], like FedProx, Scaffold, or Moon, which tackle the heterogeneity via analyzing the local client drift or feature level information and proposed corresponding methods. Except for the empirical superiority than previous methods in the experiments (e.g., Table 1), could the author discuss more about the uniqueness of the three identified pitfalls? \n3. It is unclear why to consider the domain generalization baseline here. The presentation of the current version is also a little bit confusing as the conventional work in learning with heterogeneity data assumed non-iid data partition for training while keeping the test set unchanged. It is hard to analyze since here the contains two research problems, e.g., federated learning with heterogeneity data and domain generalization, and the proposed methods seem to have superimposed effects on tackling each problem. It could be better if the authors could provide clear and rigorous problem setups in the preliminary (maybe at the beginning of Section 3 or somewhere). This may help to improve the presentation.\n4. For the experiments, considering the sensitivity of data partition, could the authors also report the std value in some parts of the experimental results? For the experiment parts, more advanced algorithms in both FL and DG are encouraged to compare, e.g., Scaffold, FedNova, IRM, and VREx, if they are applicable. Since none of the current baseline methods has considered both subproblems. \n\nOther question/comment:\n1. It seems to be a typo at the beginning of the 3rd paragraph of Appendix A: \"domain generation\" -> \"domain generalization\".\n2. The overall presentation of the current version seems to be a little confusing as the tackled problem is both conventional FL with heterogeneous data and FL with domain generalization. It may be better if the author could clarify more about the rigorous problem definition before analysis. \n3. It could be better if this work can has some further theoretical analysis about the debiasing effect since section 3 has introduced the definition of local learning bias.\n4. Could the authors further discuss the difference between the local bias analyzed in Scaffold with the proposed \"local learning bias\"?\n\n[1] Li, Tian, et al. \"Federated optimization in heterogeneous networks.\" Proceedings of Machine Learning and Systems 2 (2020): 429-450.\n[2] Karimireddy, Sai Praneeth, et al. \"Scaffold: Stochastic controlled averaging for federated learning.\" International Conference on Machine Learning. PMLR, 2020.\n[3] Li, Qinbin, Bingsheng He, and Dawn Song. \"Model-contrastive federated learning.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "_4wa0Z2DeV", "reviewer": "Reviewer_p1f4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new idea of reducing local learning bias to improve FL performance on heterogeneous data. This is achieved via the use of a validation dataset generated as either (a) random samples of local data; or (b) averaged of sampled local data. \n\nThe key idea is to solve a min-max optimization task which optimizes simultaneously for (1) a representation mapping that best distinguishes between global and local feature vectors; and (2) local representations and classifiers that produce best (local) performance and also minimize the distance to global representation of the same input. \n\nIntuitively, that means we are seeking local models that produce good local classification performance and have feature representation that cannot be \"easily distinguished\" from feature representation induced by a model trained on validation data. This means minimizing the performance of a distinguisher that learns to map features to a space on which distance between global & local features of the same input is maximized (hence, min-max optimization).\n\nThe proposed algorithm is compared with multiple FL baselines on Rotated MNIST, CIFAR-10 and CIFAR-100 datasets.", "summary_of_the_review": "This paper introduces a new idea to FL with heterogeneous client data. The idea is new and somewhat interesting but I have concerns regarding its practical feasibility, privacy compliance. I notice that there is also a lack of coverage over a very related FL literature that addresses the same issue. Certain aspects of the proposed algorithm have also not been discussed at necessary length (see my specific comments above). ", "strength_and_weaknesses": "Strengths:\n\n+ The paper is well-written. Key ideas are well-presented.\n+ The explanation of the model drift phenomenon under heterogeneous data setup is well put in three main causes all linked to the biased feature representation generated by local learning algorithm when client data are skewed.\n+ The key ideas follow quite naturally from the insightful interpretation of model drift.\n+ Overall, the motivation is clean & the idea is novel to me.\n\nWeaknesses:\n\n- The technical execution of the idea might not be practically feasible (although the formulation makes sense) -- more on this later\n- There is a complete lack of coverage on a very related suite of personalized FL algorithms specifically designed to deal with this issue\n- Not clear how much validation data is needed for this to work\n- Also, the validation (or pseudo) data obviously needs to be balanced -- how do we ensure that if the design of such validation data comes from random or aggregated sample of local data?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "SEIybJVHBpv", "reviewer": "Reviewer_UkVs", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "- This paper proposes two techniques to alleviate the biased local update due to the data heterogeneity across the clients: 1) generating pseudo data by simply averaging a subset of data and regularizing the local model not to be biased toward skewed classes, 2) employing contrastive loss that facilitates close representation between global model and local model for the same classes, while simultaneously enforcing discriminative representation for different classes.\n- Experiments on multiple image classification benchmarks validate the effectiveness of the proposed method over SOTA federated learning and domain generalization algorithms.", "summary_of_the_review": "At this point, this paper is borderline reject. While This paper is well-written and easy to follow. The objective of this work is clear, but the experiments are hard to understand and not convincing to support the main contribution.", "strength_and_weaknesses": "### Strong points\n\n- Experimental results show non-trivial accuracy gain.\n- The idea of enforcing high similarity between the global features of pseudo data and the local features of pseudo data is interesting. Also, using an adversarial projection layer to facilitate effective representation learning by the proposed contrastive loss is also straightforward.\n\n### Weak points\n\n- Toy experiments in Section 3 are not precise enough, so the results in the toy experiments may not support the claims (observations)\n    - In Figure 2, the authors plot t-SNE for two independently trained models. Even if we train two deep models with the same data, simply a different initialization can result in different representations. Since the two models are trained with different data distributions, it is not surprising that there are distinct clusters for two independent models. In my opinion, the local model should be fine-tuned from the global model with the local data and observe the representation trends.\n    - In a similar vein, the small scope of the local features for both seen and unseen classes is also not surprising since the 5-way classification problem is much easier than the 10-way classification.\n- Observation in Figure 3 is a widely known problem in existing works in continual learning.\n- For the construction of the pseudo-data, setting gt as $1/C$ may be the abuse of the prior knowledge of evaluation datasets since CIFAR10, CIFAR100, and MNIST have balanced data over classes.\n\n\n### Questions\n\n- Why the accuracy of FedDebias in Table 1 is different from the accuracy in Table 2??", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ODBMeV6_3HB", "reviewer": "Reviewer_cKx8", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes FedDebias to address the data heterogeneity issue in federated learning. The paper observes that local features and global features are different even for the same input data and the local models classify all samples into the classes that appear in the local dataset. To address the above biased local features and local classifiers, FedDebias utilizes pseudo-data to improve local training. Specifically, it maximizes the distance between the local features of pseudo-data and local data, and minimizes the distance between the global and local features of pseudo-data. Experiments show that FedDebias outperforms the other FL approaches.", "summary_of_the_review": "Overall, I think the paper proposes an interesting idea without solid analyses. The paper can be further improved to discuss privacy, effectiveness, and convergence of the proposed approach.\n", "strength_and_weaknesses": "Strength: The idea is clear. The experiments are comprehensive. The improvement of FedDebias is significant.\n\nWeaknesses:\n1. FedDebias needs to transfer pseudo-data during training, which is produced by averaging the raw data. Thus, additional privacy concerns and communication overhead are introduced in FedDebias, which lack analyses and discussion.\n\n2. The paper lacks theoretical analyses of the effectiveness and convergence of the proposed approach.\n\n3. The experiments can be further improved. What is the pseudo-data size generated by RSE? What is the subset size used to construct the samples in RSE? I think the pseudo-data is important in FedDebias. Thus, more investigation and experiments on the pseudo-data (e.g., changing the size of pseudo-data) are needed.\n\n4. I do not understand why it is necessary to maximize the distance between the local features of pseudo-data and local data first. The pseudo-data may have the same class with the local data. I think the paper should provide more insights into why such a min-max process can improve training. \n\n5. Typo: Page 5: maximizing local features -> maximizing the distance between \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject