Federated Learning under Partially Disjoint Data via Manifold Reshaping
Authors: Ziqing Fan
Jiangchao Yao
Ruipeng Zhang
Lingjuan Lyu
Yanfeng Wang
Ya Zhang
Venue: Accepted by TMLR ()

Abstract: Statistical heterogeneity severely limits the performance of federated learning (FL), motivating several explorations e.g., FedProx, MOON and FedDyn, to alleviate this problem. Despite effectiveness, their considered scenario generally requires samples from almost all classes during the local training of each client, although some covariate shifts may exist among clients. In fact, the natural case of partially class-disjoint data (PCDD), where each client contributes a few classes (instead of all classes) of samples, is practical yet underexplored. Specifically, the unique collapse and invasion characteristics of PCDD can induce the biased optimization direction in local training, which prevents the efficiency of federated learning. To address this dilemma, we propose a manifold reshaping approach called FedMR to calibrate the feature space of local training. Our FedMR adds two interplaying losses to the vanilla federated learning: one is the intra-class loss to decorrelate feature dimensions for anti-collapse; and the other one is the inter-class loss to guarantee the proper margin among categories in the feature expansion. We conduct extensive experiments on a range of datasets to demonstrate that our FedMR achieves much higher accuracy and better communication efficiency.

Published in Transactions on Machine Learning Research (10/2023)
Federated Learning under Partially Class-Disjoint Data via Manifold Reshaping
Ziqing Fan, Jiangchao Y Ruipeng Zhang
{zgfom {zqfon_knighht knight, sunarker,
Cooperative Medionet Innovation Center, Shanghai Jino Tong University Shanghwi AI Laboretory
Lingjuan Lyu
Lingjuan, Sonnjjaan.Lassonyy.coo
Sony AI
Ya Zhang, Yanfeng Wang
{yo shang, wangyonfeng) @astoo..u..uuu...ooom
Cooperalive Medianel Innovation Center, Shanghai Jiao Tong Universily Shanghwi AI Laborulory
Reviewed on OpenReview: https: // openreview. net/ forum? id= JLJTqJXAG7
Abstract
Statistical heterogencity severely limits the performance of federated learning (FL), moti- vating several explorations e.g., FedProx, MOON and FedDyn, to alleviate this problem. Despite elfectiveness, their considered scenario generally requires samples from almost all classes during the local training of each client, although some covariate shifts may exist among clients. In fact, the natural case of partially class-disjoint data (PCDD), where cach client contribules few classes (instead of all classes) of samples, is practical yet underex- plored. Specifically, the unique collapse and invasion characteristics of PCDD can induce the biased optimization direction in local training, which prevents the efficiency of federated learning. To address this dilemma, we propose manifold reshaping approach called FedMR to calibrate the feature space of local training. Our FedMR adds two interplaying losses to the vanilla federated lcarning: one is intra-class loss decorrelate fcature dimensions for anti-collaps;; and the other is inter-class loss to guaranlee the proper margin among cat- egories in the feature expansion. We conduct extensive experiments
range of datasets to demonstrate that our FedMR achieves much higher accuracy and better communication efficiency. Source code is available at: hetps://github.e. http:/oom/MMMiaaaaaaaaSStt//a/aa
1 Introduction
Federated learning (McMahan et al. (017 Li et al.
(2020a) Yang et al. (2019) has drawn considerable
attention due to the increasing requirements on data protection (Shorii & Shmatikov (2015); Zhu & Han
(2020gg Hu et al.] (2021): Li et al. (2021c); Lyu et al. (2020) in real-world applications like medical image
analysis (Guo et al. (2021): Park et (2021): Yin et al. (2022) Dou et al. (2021); Jiang et al. (2022)
and autonomous driving (Liang al. (2019): Pokhrel Choil (2020) ). Nevertheless, the resulting challenge of data heterogeneity severely limits the application of machine learning algorithms (Zhao et al. (2018) in federated learning. This motivations plenty of explorations to address the statistical heterogeneity issue
and improve the efficiency [Kairouz et al. (2021): Wang et al. (2020): Li et al. (2022) )
Existing approaches to address the statistical heterogeneity can be roughly sunnariied into two categories. One line of research to constrain the parameter update in local clients or the central server. For example, FedProx (Li et al. (2020b)), FodDyn (Acar et al.] (2020)) and FedDC (Gao et al.) (2022)) explore how reduce the variance or calibrate the optimization by adding the proximal regularization on parameters in FedAvg (McMahan et. al. (2017)). The other line of research focuses constraining the representation

---

Published in Transactions on Machine Learning Research (10/2023)
Server
Server
Client
Client 2
Client N GOOG
Client
Client
Client N
GOGG
6606-6
G600
(a) Typical FL.
(b) FL under PCDD.
(c) Global Collapsed/ / Reesaaped (by FedMR) feature space.
Figure 1: Federated learning under partially class-disjoint data (PCDD).
from the model to implicitly affect the update. FedProc (Mu et al. [)) and FedProto (Tan et al. (0022) introduce prototype learning to help local training, and MOON (Li al. (2021b)) utilizes contrastive learning to minimize the distance between representations learned by local model and global model, and maximize the distance between representations learned by local model and previous local model. However, all these methods validate their efficiency mostly under the support from all classes of samples in each client while lacking well justification natural scenario, namely partially dlass-disjoint data w.r.t. classes.
As illustrated in Figure [(a) in typical federated learning, cach client usually contains all classes of samples but under different covariate shifts, and all clients work logether to train a global model. However, in the case of PCDD (Figure [b)],, there are only a small subset of categories in each client and all clients together provide information of all classes. Such a situation is very common in wollw applications. For example, there are shared and distinet Thyroid diseases in different hospitals due to regional diversity (Gaitan et al.) (1911)- Hospitals from different regions can construct federation to learn comprehensive model for the diagnostic of Thyroid diseases but suffer from the PCDD challenge. We conduct toy study on simulated dataset (see details in the Appendix BB), and visualize the feature space under centralized training (the left panel of Figure [c]] and local training under PCDD (the middle panel of Figure T(c) by projecting on the unit sphere. As shown in Figure [cc] PCDD induces a dimensional collapse onto a narrow area due to the lack of support from all classes, and causes a space invasion to the missing classes. Previous approaches such as FedProc and MOON may implicitly constrain the space invasion by utilizing class prototypes or global features generated from the global model, and methods like FedProx, FedDyn, and FedNova could also help bit from the view of optimization. However, these methods are not oriented towards the PCDD issue, and they are inefficient to avoid the collapse and invasion characteristies of PCDD and achieve sub-optimal performance from the both view of experimental performance shown in Table D and the feature variance of all methods shown in Table 7
To address this dilemma, we propose a manifold-reshaping approach called FedMR to properly prevent the degeneration caused by the locally class missing. FedMR introduces two interplaying losses: one is intra-class loss to decorrelate feature space for anti-collapsc; and another one is the inter-class loss to guarantee the proper margin among calegories by means of global class protolypes. The right panel of Figure I(c) provides rough visualization of FedMR. Theoretically, we analyze the benefit from the interaction of the intra-class loss and the inter-class loss under PCDD, and empirically, we verify the effectiveness of FedMR compared with the current ttate-of-the-ar methods. Our contributions can be summarized as follows:
We are among the first attempts to study dimensional collapse and space invasion challenges causod by PCDD in Generic FL that degenerates embedding space and thus limits the model performance.
2

---

Published in Transactions on Machine Learning Research (10/2023)
We introduce approach termed FedMR, which decorrelates the feature space to avoid dimensional collapse and constructs proper inter-class margin to prevent space invasion. Our theoretical analysis confirms the rationality of the designed losses and their benefits to address the dilemma of PCDD.
We conduct a range of experiments on multiple benchmark datasets under PCDD and a real-world disease dataset to demonstrate the advantages of FedMR over the state-of-the-art methods. We also develop several variants of FedMR consider the communication cost and privacy concerns.
2 Related Works
2.1 Federated Learning
There are extensive works to address the statistical heterogeneity in federated learning, which induces the bias of local training due to the covariate shifts among clients (Zhao et al. (2018); Li et al. [002)] Zhang et al. (2023a)) ) A line of research handles this problem by adding constraints like normalization or regularization on model weights in the local training or in the server aggregation. FodProx (Li et al. (2020b)) utilizes
proximal term limit the local updates so as to reduce the bias, and FedNova (Wang et al. (2020))
introduces the normalization (2000g) makes the global model and local models approximately aligned in the limit by proposing dynamic regularizer for each client at each round. FedDC (Gao et al. (2022)) reduces the inconsistent optimization on the client-side by local drift decoupling and correction. Another line research focuses on constraining representations from local models and the global model. MOON (Li et al. (2021b)) corrects gradients in local training by contrasting the representation from local model and that from global model. FedProc (Mul et al. (2021)) utilizes protolypes as global information to help correct local representations. Our FedMR is also conducted representations of samples and classes but follows a totally different problem and spirit.
total gradients to eliminate the objective inconsistency. FedDyn (Acar et al.
2.2 Representation Learning
The collapse problem is also an inevitable concern in the area of representation learning. In their research lines, there are several attempts to prevent the potential collapse issues. For example, in self-supervised learning, Barlow Twins, VICReg and Shufled-DBN (Bardes et al. (202)): Zbontar el al. (2021); 5 Hua et al. (202l)) manipulate the rank of the (co-)variance matrix to prevent the potential collapse. In contrastive learning, DirectCLR (Jing et al. (2021)) directly optimizes the representation space without an explicit trainable classifier to promote larger feature diversity. In incremental learning. CwD (Shi al. (2022)) applies similar technique to prevent dimensional collapse in the initial phase. In our PCDD case, it is challenging, since we should not only avoid collapse but also avoid the space invasion in the feature space.
2.3 Federated Prototype Learning
In many vision tasks (nell et al. (2017); Yang et al. (2018) Deng et al. (2021)), prototypes are the mean values of representations of a class and contain information like feature structures and relationships of different classes. Since prototypes are population-level statistics of features instead of raw features, which are relatively safe to share, prototype learning thus has been applied in federated learning. In Generic FL, FedProc (Mu et al. (2021)) utilizes the class prototypes as global knowledge to help correct local training. In Personalized FL, FedProto (Tan et al. (2022) shares prototypes instead of local gradients to reduce communication costs. We also draw spirits from prototype learning to handle PCDD. Besides, to make fair comparison to methods without prototypes and better reduce extreme privacy concerns, we conduct range of auxiliary experiments in Section 4.4
3
The Proposed Method
3.1
Preliminary
PCDD Definition.
There are many nonnegligible real-world PCDD scenarios. ISIC2019 dataset (Codella
et al. (2018 Tschand] et al. (2018) Combalia et al. 2019) ) region-driven subset of types of Thyroid
3

---

Published in Transactions on Machine Learning Research (10/2023)
W
B
W
Figure 2:
An illustration about the shift of the optimization direction under PCDD. Here, we assume our client contains two classes ei and 22 with one missing class c3. w is the optimal classifier direction for e2 (perpendicular to the plane a) when all classes exist, and wi is the learned classifier direction when c3 is missing, which can be inferred by the docision plane 3 between ei and Cz. As can be seen, PCDD leads the angle shift 0 in the optimization.
diseases in the hospital systems, is utilized in our experiments. In landmark detection (Weyand et al. (2020)) for thousands of categories with data locally preserved, most contributors only have subset of categorics of landmark photos where they live or traveled before, which is also a scenario for the federated PCDD problem. To make it clear, we first define some notations of the partially class-disjoint data situation in federated learning. Let C denote the collection of full classes and P denote the set of all local clients. Considering the real-world constraints like privacy or environmental limitation, each local client may only own the samples of partial classes. Thus, for the k-th client Px, its corresponding local dataset Dj can be expressed a ((Ir,.. Vy,e) = Ck), where C Ç C. The number of samples of the class (c E Ck) in Pk is No. We denote local model f(; W&) all clients two parts: backbone network fil; Wx,1) and a linear classifier f2(: Wx,2). The loss of the k-th client can be formulated as
No Elwe. falzz, wx.,)) (an
(Dx
NR
where 2R, is the feature representation of the imput Ih,s and E(.,)) is the loss measure. Under PCDD, we can empricially find the dimensional collapse and the space invasion problems about representation.
FedAvg. The vanilla federated learning via FedAvg consists of four sleps (McMahan al. 2017 ): 1)
round t, the server distributes the global model w' clients that participate in the training; 2) Each local client receives the model and continues to train the model, e.g., the k-th client conducts the following,
wh w v/Vaabl:: w().
(1)
where 7 the learning rate, and bj. is mitc- of training data sampled from the local dataset D After E epochs, acquire new local model w 3) The updated models are then collected to the server as (w[, wh, w....., w 4) The server performs the following aggregation to acquire new global model w
K
P&w
(2)
where Pke is the proportion of sample number of the k-th client the sample number of all the participants, i.e., ph= Ns/ E Nv. When the maximal round T reaches, we will have the final optimized model wT.
3.2 Motivation
In Figure 2 we illustrate a low- dimensinnii example to characterize the directional shift of the local training under PCDD on the client side. In the following, the parameter aggregation of linear classification
4

---

Published in Transactions on Machine Learning Research (10/2023)
to study the directional shift of global model in the server, which further clarifies the adverse effect of PCDD. Similar to Figure 2 let. C1, and denote three classes of samples on circular face respectively centered at (1, 0), (0, v3) and (0, -v33 with radius r= under a uniform distribution. Then, if there are samples of all classes in each local client, we will get the optimal weight for all categorics as follows:
1
w"
/3
/3
Note that, we omit the bias term in linear classification for simplicity. Conversely, among total three participants, if cach participant only has the samples of two classes, c.g., (ci, 02), (c1, es) and (cz, es) respectively, then their learned weights can be inferred follows:
v3
0
0
wi W2
w3
0 1
v3
0
-1
After the server aggregation, we have the estimated weight
w
v/++2
v/++2
Then, we can find that except the difference on amplitude between w" and w", more important issue is the optimization direction for e (or cạ) shifts about 45 by computing the angle between vector (- 2)
and vector (- 6. x2²42 (or between vector (- and vector (- -=) Actually, the angle shift can be enlarged in some real-world applications, when the hard negative classes are missing. However, if we can have the statistical centroid of the locally missing class in the local client, namely C3 in the case of Figure 2 it is casy to find that the inferred optimal u² is same to w" as the decision plane can be normally characterized with the support of the single point eg. This inspires us design the subsequent method
3.3 Manifold Reshaping
As the aforementioned analysis, PCDD in federated learning leads to the directional shift of optimization both the local models and the global model. An empirical explanation is that the feature representation of the specific class that should support classification is lotally missing, inducing the feat ure representation of other observed classes arbitrarily distributes as a greedy collapsed manifold, as shown in Figure [[c] To address this problem, we explore a manifold-reshaping method from both the intra-class perspective and the inter-class perspective. In the following, we will present two interplaying losses and our framework.
3.3.1 Intra-Class Loss
The general way to prevent the representation from collapsing into a low-dimensional manifold, is to decor- relate dimensions for different patterns and expand the intrinsic dimensionality of each category. Such goal can be implemented by manipulating the rank of the covariance matrix regarding representation. Specif- ically, for each client, we first compute the class-level normalization for the representation 2 Rd as 26, - of
where Mf and of are the mean and standard deviation of features belonging to class c
and calculated as: 1 - zh, and of = me.e DEn covariance matrix based the above normalization for each observed class in the k-th client:
Sa. (=E, 2 Then, we compute an intra-class
N Σ 1=1
M
-
(BR.)))
No
Note that, would like to point that the wollworldd case, we cannot acquire such statistical centraid advance but have resort training process along with the special technique design.
5

---

Published in Transactions on Machine Learning Research (10/2023)
Since each eigenvalue of ME Rdxd characterizes the importance of feature dimension within class, we can make them distributed uniformly to prevent the dimensional collapse of each observed class. However, considering the learnable pursuit of machine learning algorithms, we actually cannot directly optimize eigen- values to reach this goal. Fortunately, it possible to use an equivalent objective as an alternative, which is clarified by the following lemma.
Lemma 1. Assuming covariance matric M E R computed from the feature of each sample with the standard normalization, and its eigenvalues
ddt, we will have the following equality that satisfied
Σ (
1 Σ 2,)²
|M d.
j=1
The complete proof is summarized in the Appendix A.2 From Lemma | we can see that pursuing the uniformity of the eigenvalues for the covariance matrix can transform into minimizing the Frobenius norm of the covariance matrix. Therefore, our intra-class loss to prevent the undesired dimensional collapse for observed classes formulated as 4
gintra
|
(3)
C
3.3.2 Inter-Clas Class Loss
Although the intra-class loss helps decorrelate the feature dimensions to prevent collapse, the resulting space invasion for the missing classes can be concomitantly exacerbated. Thus, is important to guarantee the proper space of the missing classes in the expansion as encouraged by equation 3 To address this problem, maintain series of global class prototypes and transmit them to local clients support of the missing classes in the feature space. Concretely, we first compute the class prototypes in the k-th client as the average
of feature representations (Snell et al. (0017) Yang et al. 2018 Deng et al. (201
N
gh of f
2 for E Ck
Nc
Then, all client-level prototypes submitted to the server along with local models in federated learning. In the central server, the global prototypes for all classes are updated
2h (o Σ phgh for forferee cEc].
where df the global prototype of the e-th class in round and p = N Ng. In the next round, the central server distributes the global prototypes to all clients as the references to avoid the space inva- sion. Formally, we construct the following margin loss by contrasting the distances from prototypes to the representation of the sample.
inter
Σ
Σ
Deagy
(4)
[Cal(Cal-1)
4,Exx c/C
where Deve, definod as:
N
D Deles
max - g 11 01.01.
In the following, we use theorem show how inter-class loss jointly with intra-class loss makes the representation of the missing classes approach to the optimal.

---

Published in Transactions on Machine Learning Research (10/2023)
Central Server
Local Client
000 CC Co
Final
Manifold Reshaping
Local Model
Clasatler
Intro class Loss
Inter class
Aggregation
Feature
Carnect
d
Fenture: x
Dimeasions
Space
Global Protetspes
Lacal Class Protetspes
Global Class Prototypes
Input
GGGG-G
G
Figure 3: The framework of FedMR. On the client side, except. the vanilla training with the classification loss, the manifold-reshapinnn parts, i.e., the intra-class loss and the class- loss, respectively help conduct the feature decorrelation to avoid the dimensional collapse, and leverage the global prototypes to construct the proper margin among classes to prevent the space invasion. On the server side, except the model aggregation, the global class prototypes are also the reference for missing classes participating in the local training.
Algorithm FedMR
Input: a set of K clients that participate in cach round, the initial model weights w the maximal round T, the learning rate 7), the local training epochs E.
for t= 0, ... T 1 do randomly sample K clients
Ve, + Ek pigh updates global model weights and global class protolypes (w' -== p/. E
distribute w and Gelgi, si. ge) to the K clients.
do in parallel for Vk K clients w ww. for r= E- do
sample a mini-batch from local dataset and perform updates( w + tw: - Lx Get w/)) ) to server.
end for
update local class protolypes (Ve, sf + En mr 2t) and submit w and {ot ..., "..,, end in parallel end for
ott Theorem 1. Lel the global optimal representation for class e be gt [ac,,,... dral, and zi be the represen- tation of sample x in the class of the k-thh client. Assuming that Vi, both la%,l and gtt by G, and all dimensions are disentangled, in round t, the i-th dimension of local representation 2 satisfies
2 are upper bounded
1 o-, s 2(1 D)
where Pi is the accumulation regarding the i-th dimension of the class-c prototype, I' 1-(r)) T T- (pf)* refers
to the ph raised to the power of t, and 8 is the mazimum margin of the inter-loss term.
Note that, Theorem shows five critical points: 1) The proof of the theorem requires each dimension of the representation be irrelevant each other, which is achieved by the intra-class loss. Although disentanglement of dimensions might not be totally achieved in practical, empirically, we find that the intra-class loss converges and maintains relatively low value easily, meaning that the model achieves good
7

---

Published in Transactions on Machine Learning Research (10/2023)
decorrelation. In the Appendix, show the training curve of intra-class loss during the federated training. 2) In the theorem, 8 is trade-off of training stability and theoretical results determined by the margin. The larger the margin, the larger the d, however the more stable the local training is. This is because the global prototypes are not very accurate in the carly stage of local training and directly minimizing the distance of samples to their global class protolypes can bring side effect to the feature diversity. When the margin is removed (De,ey = 1 En =1 - 9 Il) 5 will be zero. 3) Without considering 6, as t incrcases, I is smaller and representation = closer to global optimal prototype ge, showing the promise of our method. 4) When t is large enough, we can get an upper bound 2 1- - G, meaning more clients with the specific I-PÉ dimensional information participating in the training, the lighter the upper bound is. When all other clients can provide the support (Øf 1- ph), the error will be O. 5) While &% denotes the proportions of subset of clients that can provide the support information for this dimension, the theoretical result depends on the class distribution and overlap across the clients. The complete proof is sunmarized in the Appendix A.1
Ng
3.3.3 The Total Framework
After introducing the intra-class loss and the inter-class loss, we give the total framework of FedMR. On the client side, local models are trained their partially class-disjoint datasets. Through manifold-reshpping loss, dimensions of the representation are decorrelated and local class subspace is corrected to prevent the space invasion, and gradually approach the global space partition. The total local objective including the vanilla classification loss can be written as
La - pele + (pre dintra + (
manifold reshaping
where Mi and A2 are the balancing hyperparameters and will be discussed in the experimental part. In Figure 3 we illustrate the corresponding structure of FedMR and formulate the training procedure in Algo- rithm In terms of the privacy concerns about the prototype transmission and the communication cost, we will give the comprehensive analysis on FedMR about these factors.
4 Experiment
4.1 Experimental Setup
Datasets. We adopt four popular benchmark datasets SVHN Netzer et al. (2011)), FMNIST (iia et al.
(0017)) CIFAR10 and CIFAR100 (LeCun et al. (1998) in federated learning and real-world PCDD medical dataset ISTC2019 (Codella et al. (2018); Tschand] et al. 2018 Combalia et al. (2019)) to conduct experi- ments. Regarding the data setup, although Dirichelet Distribution is popular to split data in FL, it usually generales diverse imbalance data coupled with occasionally PCDD. In order to better study pure PCDD, for the former four benchmarks, we split each dataset into & clients, each with s categories, abbreviated PoC's For example, P10C10 in CIFAR100 means that we split CIFAR100 into 10 clients, cach with 10 classes. Please refer to the detailed explanations and strategies in the Appendix. ISIC2019 is a real-world federated application under the PCDD situation and the data distribution among clients is shown in the Appendix C.1.3 We follow the settings in Flamby benchmark Terrail et al. 2022)).
Implementation. We compare FedMR with FedAvg (McMahan al. (0117) and multiple state-of-the- arts including FedProx (Li et al. (2020b)), FedProc (Kariirredy al. (2220) ), FedNova. (Li et al. (2021b) )
MOON (Wang et al. (2020)) FedDyn (Acar et al. (2020) and FedDC Gao et al. (2022) To make fair and comprehensive comparison, we utilize the same model for all approaches and three model structures for different datasets: 18sNet18 (He et al. (2016)) (follow (Li et. al. (2021b 2022 )) for SVHN, FMNIST and CIFAR1O, wide ResNet (Zagoruyko Komodakis (2016) for CIFAR100 and EfficientNet (Tan & Le (019)) for ISIC2019. The optimizer SGD with a learning rate 0.01, the weight decay 10-5 and momentum 0.9. The batch size is set to 128 and the local updates are set 10 epochs for all approaches. The detailed information of the model and training parameters are given in the Appendix.
8

---

Published in Transactions on Machine Learning Research (10/2023)
Table 1: Performance of FedMR and range of state-of-the-art approaches on four datasets under PCDD
partitions. Datascts are divided into @ clients and each client has ç classes (denoted PoCs). We compute average accuracy of all partitions, highlight results of PedMR, underline results of best baseline, and show
the improvement of FedMR to FedAvg (subscript of FedMR) and to the best baseline (A).
Datasets
Split
FedAvg
FedProx FedProc FedNova MOON FedDyn FedDC
FedMR
A
P5C2
67.29
69.60
66.28
66.87
66.82
71.01
68.50
75.518.22%
+4.50
P10C2
67.33
67.76
69.08
48.44
67.93
67.16
67.36
74.977.64%
+5.89
FMNIST
P10C3
81.67
80.93
82.06
83.20
83.42
83.00
83.24
83.551.8844
+0.13
P10C5
88,53
89.22
89.23
88.86
88.98
88.38
89.22
90.041.51%5
+0.81
91.93
91.95
92.06
91.84
92.12
91.76
92.15
92.19
+0.04
avg
79.35
79.89
79.74
75.84
79.87
80.26
80.09
83.455..0%%
+3.19
P5C2
81.85
81.83
79.54
81.11
81.60
79.89
81.63
83.10 13.101.25%
+1.27
P10C2
78.92
79.60
78.75
66,86
79.83
76.24
78.96
$2.473.55%%
+2.64
SVHN
P10C3
87.70
87.40
88.13
87.50
87.83
87.27
88.05
89.13 43%1
+1.00
P10C5
91.20
91.24
91.63
92.09
91.16
90.17
91.64
92. 92.180.96 08.98%
+0.09
IID
92.74
92.89
93.57
92.62
93.12
92.26
92.90
93.04o.500 30%1
-0.53
avg
86.48
86.59
86.32
83.87
86.71
85.17
86.64
87.98 1.5%
+1.27
P5C2
67.68
68.18
69.27
67.57
66.86
69.64
69.18
74.19 6.611
+4.55
P10C2
67.27
71.09
67.02
57.79
67.61
67.74
67.64
73.32, 23%1
+2.23
CIFAR10
P10C3
77.82
77.89
77.87
77.22
78.42
77.99
77.94
82.75
+4.33
P10C5
88,22
88.34
88.19
88.20
88.00
88.35
88.14
89.06
+0.71
IID
91.88
92.14
92.62
92.37
92.56
92.29
92.85
93.06..1891
93.06
1881 82.48,
+0.21
avg
78.58
79.53
78.99
76.63
78.69
79.20
79.15
90%6
2.95
P10C10
54.31
54.79
54.69
54.45
54.98
55.94
54.73
57.27
57.272.96%1
2.96%1
+1.33
P10C20
64.81
65.37
64.98
65.79
65.75
65.02
65.21
65.811 1.00%
+0.02
CIFAR100
P10C30
69.35
69.75
69.64
69.55
69.51
69.84
69.38
70.24 0.99
+0.40
P10C50
71.28
71.35
72.13
71.25
71.54
71.25
72.11
72.17 5.5%%
+0.04
IID
72.28
72.55
73.07
72.66
73.01
73.04
72.77
72.79
-0.28
avg
66.41
66.76
66.90
66.74
66.96
67.22
66.84
67.661.22 25%%1
+0.44
Table 2: Global test accuracy of methods on CIFARIO and CIFARI00 under larger scale of clients (P10, P50 and P100) and ISIC2019. PeCs denotes that the dataset is divided into g clients and each client has ę classes of samples. We highlight results of FedMR, underline results of best baseline, and show the improvement of FedMR FedAvg (bottom right corner of results) and to the best baseline (A).
Datasets
SSplit
FedAvg FedProx FedProc FedNova MOON FedDyn FedDC
FedMR
A
P10C3
77.82
77.89
77.87
77.22
78.42
77.99
77.94
82.754.9992
+4.33
CIFARIO
P50C3
75.46
77.46
76.27
74.12
76.51
75.52
76.03
79.58..12%1
+2.1
P100C3
71.65
72.37
72.44
70.46
72.46
71.85
73.95
76.93,.28%t
+2.98
P10C10
54.31
54.79
54.69
54.45
54.98
55.94
54.73
57 57.272.96%%
+1.33
CIFAR100
P50C10
49.84
51.17
51.94
50.22
52.19
50.53
51.17
53.36 6.2%%
+1.17
P100C10
47.90
48.26
49.01
48.07
48.94
49.24
48.76
49.60, 70%6
+0.36
ISIC2019
Real
73.14
75.41
75.26
73.62
75.46
75.07
75.25
76.55
+1.09
4.2
Performance under PCDD
In this part, we compare FedMR with FedAvg and other methods on FMNIST, SVHN CIFAR10 and CI- FAR100 datascts under partially class-disjoint situation. Note that, in FodProx, MOON, FodDyn, FodProc, FedDC and our method, there are parameters that need to set. We use grid search to choose the best parameters for each method. See more concrete settings in the Appendix C.2 and Section
As shown in Table I with the decreasing class number in local clients, the performance of FedAvg and all other methods greatly drops. However, comparing with all approaches, our method FedMR achieves far better improvement to FedAvg, especially 7.64% improvement us. 1.75% FedProc for FEMNIST (P10C2) and 6.51% improvement us. 1.96% of FodDyn for CIFAR10 (P5C2). Besides, FedMR also performs better under less PCDD, and on average of all partitions listed in the table, our method outperforms the best baseline by 3. 3.19% FMNIST and 2.95% CIFAR10.

---

Published in Transactions on Machine Learning Research (10/2023)
Table 3: The communication cost of approaches with prototypes (i.e., FedProc and FedMR) and without prototypes (i.e., FodAvg, FodProx, FodNova, MOON, FodDyn and FedDC).
Method Type
FMNIST
SVHN
CIFARIO CIFARI00
ISIC2019
w/o Prototypes
11.182M 11.187M
11.14MM
11.144M
36.565.
4.875
w/ Prototypes
11.189M
11.189M
36.629M
4.880M
Additional cost
0.044% 1 0.044% 1 0.044% 1
0.175% 1 0.102% 1
Table Number of communication rounds and the speedup of communication when reaching the best accuracy of FedAvg in federated learning on CIFAR100 under three partition strategies.
Method
P10C10
P50C10
P100C10
(CIFAR100)
Commu.
Speedup
Commu.
Speedup
Commu.
Speedup
FedAvg
400
1x
400
1x
400
1x
FedProx
352
1.14x
370
1.08x
384
1.04x
FedProc
357
1.12x
381
1.05x
389
1.03x
FedNova
290
1.38x
385
1.04x
382
1.05x
MOON
332
1.20x
373
1.07
395
1.01x
FedDyn
178
2,25x
366
1.09
384
1.04x
FedDC
275
1.45x
393
1.02x
387
1.03x
FedMR
149
2.68x
293
1.37x
368
1.09x
Litet0
Local Compu Memory
0.75
Lite50
FuMME (1.x50)
PiOMIl
FedMR
test accuracy
0.73
FedDC
FedDym
0.71
RePPno
MOON
FedNowa
0.69
FeIDC
FalPro
FedPore
0.67
PedNesa
Falkig
MOON
FedProx
FedAvg
0.65
10
30
50
70
6x
time per round(s)
(a) Local burden.
(b) Computation times with global racy.
The average memory consuming, computation time of local training and performance on all datasets of all basclines, FedMR and its light versions (Lite 10 and Lite 50) for accelcrating.
Figure 4:
4.3 Scalability and Robustness
In the previous section, we validate the FedMR under 5 or 10 local clients under partially class-disjoint data situations. In order to make a comprehensive comparison, we increase the client numbers of CTFAR10 and CIFAR100 and in each round, only 10 of clients participate in the federated procedures. Besides, we also add one real federated application ISIC2019 with multiple statistical heterogeneity problems including PCDD. The exact parameters and communication rounds of all methods can be found in the Appendix C.2
In Table 2 divide CIFAR10 and CIFAR100 into 10, 50 and 100 clients and keep the PCDD degree in the same level. As can be seen, with the nunber of client increasing, the performance of all methods drops greatly. No matter in the situations of fewer or more clients, our method achieves better performance and outperforms best bascline by 2.98% in CIFAR-10 and 0.95% in CIFAR-100 on average. Besides, in Table 2 we verify FedMR with other methods under a real federated applications: ISIC2019. As shown in the last line of Table 2. our method achieves the best improvement of 3.41% relative to Fed and of 1.09% relative best baseline MOON, which means our method is robust in complicated situations more than PCDI
4.4 Further Analysis
Except performance under PCDD, we here discuss the communication cost between clients and server, local burden of clients, and privacy, and conduct the ablation study.
Note that, in Appendix, we provide two real-worlt datasets further demonstrate the efficiency of FedMR compared with baselines.
10

---

Published in Transactions on Machine Learning Research (10/2023)
Table 5: Performance of FedMR on CIFARIO and CIFAR100 when only 50%, 80% allowed to submit their class prototypes under different partitions.
and 100% of clients are
Datasets
Split
FedAvg
50%
80%
100%
P10C3
77.82
80.42
80.27
82.75
CIFAR1O
P50C3
75.46
79.14
79.43
79.58
P100C3
71.65
76.73
76.87
76.93
P10C10
54.31
56.41
56.73
57.27
CIFAR100
P50C10
50.22
51.06
52.57
53.36
P100C10
47.90
48.80
48.88
49.60
Table 6: The ablation study of FedMR. We illustrate average accuracy of FedMR on the four datasets without
the inter-class loss or the intra-class loss or both. Results of all partitions are shown in Appendix C.7
Inter
Intra
FMNIST
SVHN
CIFAR10
CIFAR100
79.35
86.48
78.58
66.41
80.18
87.46
78.86
66.62
/
81.03
87.15
80.89
67.11
83.45
87.98
82.48
67.66
Communication Concern. In terms of communication cost, our method needs to share the additional prototypes. To show how much extra communication cost will be incurred, in Table we show the number of transmission parameters in cach round to compare the communication cost of methods with prototypes (Fed- Proc and FedMR) and without protolypes (FedAvg, FedProx, FedNova, MOON, FedDyn and FedDC). From the results, the additional communication cost in single round is negligible. Except for sharing class pro- totypes averaged from class representations, we also tried to use 1-hot vectors to save computation and communication, but the performance is unsatisfactory and even worse than the FedAvg. This is because such discriminative uru might not fit the optimal statistics as each class has different hardness to learn, and it is not clear that arriving at such an optimization stationary from the given initialization can be good choice. In Table 4 also provide the communication rounds on CIFARI00 under three different partitions, where all methods require to reach the best accuracy of FedAvg within 400 rounds. From the table we can see that FedMR uses less communication rounds (best speedup) to reach the given accuracy, indicating that FedMR is a communication-effiieett approach.
Local Burden Concern. In real-world federated applications, local clients might be mobile phones or other small devices. Thus, the burden of local training can be the bottlenock for clients. In Figure 4(a) we compule the number of parameters that needs to be saved in local clients and the average local computation time per round. As can be seen, FedDC, FedDyn and MOON require triple or even quadruple storing memory than FedAvg, while FedProc and FedMR only need little space to additionally store prototypes. In terms of local computation time, FedMR requires more time to carefully reshape the feature space. To handle some computing-restricte clients, we provide light versions of FedMR, namely Lite 10 and Lite 50, where local clients randomly select only 10 or 50 samples to compute inter-class loss. From Figure 4(a) the training time of Lite 10 and Lite 50 decreases sharply, while their performance is still competitive and better than other baselines, as shown in Figure 4(b) Please refer to Appendix C.6 for more details.
Privacy Concern. Although prototypes are population-level statistics of features instead of raw features, which are relatively safe (Mu et al. (2021): Tan et al.] (2022)), it might be still hard for some clients with extreme privacy limitations. To deal with this case, one possible compromise is allowing partial local clients not to submit prototypes. In Table 5] we verify this idea for FedMR on CIFAR10 and CIFAR100, where at the beginning, even under the prototypes of 50% clients, FedMR still performs better than FedAvg, showing the clastic potential of FedMR in the privacy-restritted scenarios. In the extreme case where all prototypes of clients are disallowed to be submitted, we can remove the inter-class that depends on prototypes from FedMR and use the vanilla federated learning with the intra-class loss. As shown in Table it can achieve a promising improvement than that without the intra-class loss. Note that, we cannot counteract the privacy concerns of federated learning itself, and leave this in the future explorations.
only randomly pre-select 50% and 80% clients to require prototypes. From Table 5
11

---

Published in Transactions on Machine Learning Research (10/2023)
Table 7: Variance of top 50 egienvalues of covariance matrices of all classes within a mini-batch on CIFAR1O.
Method
P5C2
P10C2
P10C3
P10C5
IID
FedAvg
1201
1274
1055
814
640
FedProx
931
977
874
727
FedProc
1011
1074
955
739
522
FodNova
1234
1277
977
755
572
MOON
749
866
774
744
579
FodDyn
854
906
844
759
FedDC
1077
1104
784
766
572
FedMR (intra)
478
437
372
407
538
FedMR inter))
570
538
566
579
635
Table 8: The average accuracy of all methods adopted in Table | with or without the aid of prototypes on FMNIST. Since FedProc is the prototype version of FedAvg, here we don't show them.
Method
FedProx
FedNova
MOON
FodDyn
FedDC
FedMR
w/o Prototypes
79.89
75.84 78.76
79.87
80.26
80.09 80.57
81.03
w Prototypes
80.29
80.12
81.21
83.45
Decorrelating Analysis Here, empirically verify the effectiveness of FedGELA compared with all methods on constraining feature spaces from the view of the variance of eigenvalues of the covariance matrix M. Table 7 we show the variance of top-50 eigenvalues (sort from the largest to the smallest) of the covariance matrix M within a mini-batch (batchsize 128) after training 100 rounds on CIFARIO, calculated 88: 125 Eie (A, 50 Ej= Aj)2. As can be seen, when the PCDD problem eased, the variance of Fed Avg gradually drops, which means in more uniform data distribution, the variance should be relatively small. The slight and sharp decreasing variance of prior federated methods and our FedMR indicate that they indeed help but still suffer from the dimensional collapse problem caused by PCDD while FedMR successfully decorrelates the dimensions. However the variance under the intra-class loss is too small and far away from the values of FedAvg in the IID setting, meaning it may enlarge the risk of the space invasion. In order to prevent space invasion, our inter-class loss provide margin for the feature space expansion. In the table, we could see that the variance of FedMR (under the intra-class loss and the inter-class loss) is little larger compared to FedMR (the intra-class loss) and approaches to the FedAvg under the IID sctting.
Ablation Study. FedMR introduces two interplaying losses, the intra-class loss and the inter-class loss, to vanilla FL. To verify the individual elficiency, we conduct an ablation experiment in Table G As can be seen, the intra-class loss generally plays a more important role the performance improvement of FedMR, but their combination complements cach other and thus performs best than any the single loss, confirming our intuition prevent the collapse and space invasion under PCDD jointly. Besides, as FedMR and FedProc need local clients additionally share class prototypes which might not fair for other baselines, in Table properly configure all baselines with prototypes FMNIST to show the superiority of FedMR.
5 Conclusion
In this work, we study the problem of partially class-disjoint dota (PCDD) in federated learning, which is practical and challenging due to the unique collapse and invasion problems, and propose a novel approach called FedMR to address the dilemma of PCDD. Thcoretically, we show how the proposed two interplaying losses in FedMR to prevent. the collapse and guarantee the proper margin among classes. Extensive periments show that FedMR achieves significant improvements FedAvg under the PCDD situations and outperforms range of state-of-the-art methods.
Acknowledgements
The work supported by the National Key R&D Program of China (No. 20222D0160702), STCSM (No. 22511106101, No. 22511105700, No. 21DZ1100100), 111 plan (No. BP0719010) and National Natural Science Foundation of China (No. 62306178). Ziqing Fan and Ruipeng Zhang were partially supported by Wu Wen Jun Honorary Doctoral Scholarship, AT Institute, Shanghai Jiao Tong University.
12

---

Published in Transactions on Machine Learning Research (10/2023)
References
Durmus Alp Emre Acar, Yue Zhao, Ramon Matas, Matthew Mattina, Paul Whatmough, and Venkatesh Saligrama. Federated learning based on dynamic regularization. In International Conference on Learning Represenlations, 2020.
Adrien Bardes, Jean Ponce, and Yann Lecun. Vicreg: Beriance-ivvarianceccovaa
regularization for self-supervised learning. In ICLR 2022-10th International Conference on Learning Representations, 2022.
Hanna Borgli, Vajira Thanbawila, Pia H Smedsrud, Steven Hicks, Debesh Jha, Sigrun L. Eskeland, Kristin Ranheim Randel, Konstantin Pogorelov, Mathias Lux, Duc Tien Dang Nguyen, et al. Hyper- kvasir, a comprchensive multi-class image and video datasct for gastrointestinal endoscopy. Scientific data, 7(1):283, 2020.
Noel CF Codella, David Gutman, M Emre Celebi, Brian Helba, Michael A Marchetti, Stephen W Dusza, Aadi Kalloo, Konstantinos Liopyris, Nabin Mishra, Harald Kittler, et al. Skin lesion analysis toward melanoma detection: A challenge at the 2017 international symposium on biomedical imaging (isbi), hosted by the international skin imaging collaboration (isic). In 2018 IEEE 15th international symposium on biomedical imaging (ISBI 2018), pp. 168 172. IEEE, 2018.
Marc Combalia, Noel CF Codella, Veronica Rotemberg, Brian Helba, Veronica Vilaplana, Ofer Reiter, Cristina Carrera, Alicia Barreiro, Allan C Halpern, Susana Puig, et al. Ben20000: Dermoscopic lesions in the wild. arXiv preprint Xiv:1:1908. 02288, 2019.
Jiankang Deng, Jia Guo, Jing Yang. Alexandros Lattas, and Stefanos Zafeiriou. Variational prototype learning for deep face recognition. In Proceedings of the IEEE/CVP Conference on Computer Vision and Pattern Recognition, pp.
2021.
Qi Dou, Tiffany Y So, Meirui Jiang, Quande Liu, Varut Vardhanabhuti, Georgios Kaissis, Zeju Li, Weixin Si, Heather HC Lee, Kevin Yu, et al. Federated deep learning for detecting covid-19 lung abnormalities in ct: a privacy-preservigg multinational validation study. NPJ digital medicine, 4(1):1-11, 2021.
Ziqing Fan, Yanseng Wang, Jiangchao Yao, Lingjuan Lyu, Ya Zhang, and Qi Tian. Fedskip: Combatting statistical heterogeneity with federated skip aggregation. In 2022 IEEE International Conference on Data Mining (ICDM), pp. 131-140. IEEE, 2022.
Eduardo Gaitan, Norman C Nelson, and Galen V Poole. Endemic goiter and endemic Chyroid disorders. World journal of surgery, 15(2):205-28) 215, 1991.
Liang Gao, Huazhu Fu, Li Li, Yingwen Chen, Ming Xu, and Cheng- Zhonng-Zhonggg Xu. Fedde: Federated learning with non-iid data via local drift decoupling and correction. In Proceediigg of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 10112-10124 2022.
Pengfei Guo, Puyang Wang, Jinyuan Zhou, Shanshan Jiang, and Vishal M Patel. Nulti-institational col- laborations for improving deep learning-based magnetic resonance image reconstruction using federated learning. In Proceedings of the IEEE/CU /CVF Conference on Compuler Vision and Pallern Recognilion, pp. 2423 2432, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770 778, 2016.
Rui Hu, Yanmin Gong, and Yuanxiong Guo. Federated learning with sparsification-mmllis privacy and adaptive optimization. In IJCAT, 2021.
Tianyu Hua, Wenxiao Wang, Zihui Xue, Sucheng Ren, Yue Wang, and Hang Zhao. On feature decorrelation in self-supervised learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9598 9608, 2021.
13

---

Published in Transactions on Machine Learning Research (10/2023)
Mcirui Jiang, Zirui Wang, and Qi Dou. Harmoft: Harmonizing local and global drifts in federated learning heterogeneous medical images. In Proceedings of the AAAT Conference on Artificial Intelligence, volume 36, pp. 1087- 1095, 2022.
Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning. In International Conference on Learning Representations, 2021.
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mchdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, al. Advances and open problems in federated learning. Foundations and Trends@ in Machine Learning. 14(1-2):1-210 2021.
Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International Conference Machine Learning, pp. 5132-5143. PMLR, 2020.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to docu- ment recognition. Proceedings of the IEEE, (6(11)) 2324, 1998.
Ning Li, Tao Li, Chunyu Hu, Kai Wang, and Hong Kang. A benchmark of ocular disease intelligent recogni- tion: One shot for multi-discase detection. In Benchmarking, Measuring, and Optimizing: Third Bench- Council International Symposium, Bench 2020, Virtual Evenl, November 15 16, 2020, Revised Selected Papers 9, pp. 177-193. Springer, 2021a.
Qinbin Li, Bingsheng He, and Dawn Song. Model-contrastive federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 10713 10722, 2021b.
Qinbin Li, Zeyi Wen, Zhaomin Wu, Sixu Hu, Naibo Wang, Yuan Li, Xu Liu, and Bingsheng He. A survey on federated learning systems: vision, hype and reality for data privacy and protection. IEEE Transactions on Knowledge and Dala Engineering, 2021c
Qinbin Li, Yiqun Diao, Quan Chen, and Bingsheng He. Federated learning on non-iid data silos: An experimental study. In 2022 IEEE 38th International Conference on Dala Engineering (ICDE), pp. 965- 978. IEEE, 2022.
Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. Federated learning: Challenges, methods, and future directions. IEEE Signal Processing Magazine, 37(3):50 60, 2020a.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Systems, 2:429 450, 2020b.
Xinle Liang, Yang Liu, Tianjian Chen, Ming Liu, and Qiang Yang. Federated transfer reinforcement learning for autonomous driving. Xrxiv preprint XrXio:1910. 191,000010,,,,,,, 2019.
Lingjuan Lyu, Han Yu, and Qiang Yang. Threats to federated learning: A survey. arvi preprint arXiv:2003.0211 2020.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Commmmication-efiiicne learning of deep networks from decentralized data. In Artificial intelligence and statistics, pp.
PMLR, 2017.
Xutong Mu, Yulong Shen, Ke Cheng, Xueli Geng, Jiaxuan Fu, Tao Zhang, and Zhiwei Zhang. Fedproc: Prototypical contrastive federated learning on non-iid data. arXiv preprint arXim:109. 12273, 2021.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. NIPS Workshop on Deep Learning and Unsuperoised Feature Learning, 2011.
Sangjoon Park, Gwanghyun Kim, Jeongsol Kim, Boah Kim, and Jong Chul Ye. Foderated split task-agnostic vision transformer for covid-19 exr diagnosis. Advances in Neural Information Processing Systems, 34, 2021.
14

---

Published in Transactions on Machine Learning Research (10/2023)
Shiva. Raj Pokhrel and Jinho Choi. Federated learning with blockchain for autonomous vehicles: Analysis and design challenges. TEEE Transactions Communications, 68(8):473444777 4746, 2020.
William Shakespeare al. William Shakespeare: the complete works. Barnes & Noble Publishing, 1989.
Yujun Shi, Kuangqi Zhou, Jian Liang, Zihang Jiang, Jiashi Feng, Philip HS Torr, Song Bai, and Vincent YF Tan. Mimicking the oracle: An initial phase decorrelation approach for class incremental learning. Tn Proceedings of the LEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16722 16731, 2022.
Reza Shokri and Vitaly Shmatikov. Privacy-preserving deep lcarning. In Proceedings of the 22nd ACM SIGSAC conference on compuler and communications securily, pp. 1310-1321, 2015.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. Advances in neural informalion processing systems, 30, 2017.
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning, pp. 6105 6114. PMLR, 2019.
Yue Tan, Guodong Long, Lu Liu, Tianyi Zhou, Qinghua Lu, Jing Jiang, and Chengqi Zhang. Fedproto: Federated prototype learning across heterogencous clients. In AAAI Conference on Artificial Intelligence, volume 1, pp. 3, 2022.
Jean Ogier du Terrail, Samy-Safwan Ayed, Edwige Cyffers, Felix Grimberg, Chaoyang Hc, Regis Loeb, Paul Mangold, Tanguy Marchand, Othmane Marfoq, Erum Mushtaq, et al. Flamby: Datasets and benchmarks for cross-silo federated learning in realistic healthcare settings. ar preprint W:Xxu22200046200 2022.
Philipp Tschandl, Cliff Rosendahl, and Harald Kittler. The ham10000 dataset, large collection of multi- source dermatoscopic images of common pigmented skin lesions. Scientific data, 5(1):1-9, 2018.
Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and Vincent Poor. Tackling the objective inconsistency problem in heterogeneous federated optimization. arXie preprint orXiv:2007.07433 2020.
Tobias Weyand, Andre Araujo, Bingyi Cao, and Jack Sim. Google landmarks dataset v2-a large-scale benchmark for instance-lovel recognition and retricval. In Proceedings of the IEEE/CVF conference on compuler vision and pallern recognition, pp. 2575 2584, 2020.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image dataset for benchmarking machine learning algorithms, aixiv preprint arXiv: (alb:1708.0777 07747, 2017.
Hong-Ming Yang, Xu-Yao Zhang, Fei Yin, and Cheng-Lin Liu. Robust classification with convolutional
prototype learning. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 3474-3482, 2018.
Qiang Yang, Yang Liu, Tianjian Chen, and Yongxin Tong. Fedcrated machine learning: Concept and applications. ACM Transactions on Intellignn Systems and Technology (TIST), 10(2):1 19, 2019.
Jiangchao Yao, Shengyu Zhang, Yang Yao, Feng Wang, Jianxin Ma, Jianwei Zhang, Yunfei Chu, Luo Ji, Kunyang Jia, Tao Shen, et al. Edge-cloud polarization and collaboration: A comprehensive survey for ai. IEFE Transactions on Knowledge and Data Engineering, 35(7):6866 6866-6866, 2022.
Rui Ye, Zhenyang Ni, Fangzhao Wu, Siheng Chen, and Yanfeng Wang. Personalized federated learning with inferred collaboration graphs. 2023a.
Rui Ye, Mingkai Xu, Jianyu Wang, Chenxin Xu, Siheng Chen, and Yanfeng Wang. Feddisco: Federated learning with discrepancy-aware collaboration. arXiv preprint arxv: 19229, 2023b.
Youtan Yin, Hongzheng Yang, Quande Liu, Mcirui Jiang, Cheng Chen, Qi Dou, and Pheng-Am Heng. Effi- cient federated turor segmentation via normalized tensor aggregation and client pruning. In International MICCAT Brainlesion Workshop, pp. 433-443 Springer, 2022.
15

---

Published in Transactions on Machine Learning Research (10/2023)
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In British Machine Vision Conference 2016. British Machine Vision Association, 2016.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In International Conference Machine Learning, pp. 12310-12322 PMT.R,
2021.
Ruipeng Zhang, Ziqing Fan, Qinwei Xu, Jiangchao Yao, Ya Zhang, and Yanfeng Wang. Grace: generalized and personalized federated learning method for medical imaging. In International Conference on Medical Image Compuling and Cospuler-Assisled Intervention, pp. 14 24. Springer, 2023a.
Ruipeng Zhang, Qinwei Xu, Jiangehao Yao, Ya Zhang, Qi Tian, and Yanfeng Wang. Federated domain generalization with generalization adjustment. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognátion, pp. 3954-3963, 2023b.
Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Federated learning with non-iid data. arXiv preprint Xiv:11800 00582, 2018.
Ligeng Zhu and Song Han. Deep leakage from gradicnts. In Federated learning, pp. 17-31. Springer, 2020.
16


Reviews:
[{"review_id": "8pfkMo3kBr", "reviewer": "Reviewer_dGrL", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well written, except for the citation requirement, which does not seem to be understood by the authors (see below).\n\nThe approach is novel and tackles an interesting problem. Although expected, the experimental results are interesting and show their approach outperforms standard ones on this particular task.\n\n\nBeyond PDD framework, I would like to see how the author' approach perform on non-PDD framework compare to SOTA FD approaches. \nSome additional concerns are described below.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper provides a new approach for Federated Learning (FD) in classification. The paper tackle the particular case of classification FD where dataset of each local client doesn't contain the same classes from a global dataset while having the same features $\\mathbf{X}$. Since actual methods are poorly adapted to this setting, they propose a new approaches based on ‘‘manifold reshaping'', i.e. by adding an intra-class and inter-class regularization of the training loss in each local client training w.r.t. representations of local neural networks models.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "\n- What do you use as distribution and classifier for the example in Section 3.2. Claims in this section seem coming without support/details.\n\n- I am not completely familiar with specificity of federated learning but your example in Section 3.2 seems to be close to a multi class classification with one versus one setting. In this situation, it is difficult for me to see what are the links between FedMR and one vs one classification. Or at least, why your MR method would be advantageous compared to doing a one vs one ?\n\n- I am not sure to understand how you construct $\\hat{z}^c_{k,i}$. Each representation space is suppose to be multivariate. According to your notations, how do you compute $\\sigma_{k}^{c}$, that is suppose to be a matrix? Could you please specify the space where quantity you introduce belong to?\n\nMinor comments:\n\n- The format of your citations are not good. For example: Line 1: use \\citep{cite1,cite2,cite3} instead of (\\cite{cite1,cite2,cite3}). These errors are present in the entire text, See Section 4.1 of the TMLR LaTeX stylefile to see the citation guidelines.\n\n- Figure 1: I think it would be more appropriate to leave only 4 classes. in the examples (a) and (b) with the same colors than in (c). The fact that you want to put $n$ classes in (a) , (b) reduces the ease of comprehension. \n\n- Where the name ‘‘prototypes'' used in Section 3.3.2 comes from? It just represents the class average representation.", "broader_impact_concerns": "No.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "pPw4zdQlyd", "reviewer": "Reviewer_q3Gk", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The introduced scenario of partially disjointed data with respect to classes seems indeed to be more realistic, as it’s difficult to ensure that all client devices have data from all classes. \n\n- The paper includes measurement of the performance of various existing FL methods under this PDD setup (e.g., Table 1) and demonstrates the improvements made by FedMR. The paper also provides a comprehensive analysis of different aspects of FL methods, including accuracy, computation, communication, and memory usage.\n\nWeaknesses:\n\n- While showing promising results, the paper does not clearly show whether space invasion is the only reason prior works achieve sub-optimal performance under the PDD setups.\n\n- The cost analysis of the manifold reshaping terms, such as inter- and intra-class loss calculation, shows that FedMR has a much higher computation overhead than other FL methods, being about 4.5 times more expensive than FedAvg. This could be a major limitation for choosing FedMR, as FL also runs on mobile and edge devices where energy consumption is crucial. A lite version has been provided to reduce the computation overhead of FedMR, but the comparison is only made against FedAvg, not more advanced baselines like FedProx and MOON. \n\n- Only one real dataset, ISIC2019, is used for evaluation, while the rest are toy datasets like FMNIST and CIFAR10/100. \n\n- There is also a contradiction in the evaluation methodology. While the authors claim to use a fixed set of hyperparameters for all methods, such as a learning rate of 0.01 as described in Section 4.1, Table 10 in the appendix states that they tune hyperparameters like the learning rate from 0.000001 to 1.\n\n- The paper seems to have some writing issues that affect readability. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces FedMR, a new federated learning method that improves model quality when clients have partially disjointed data (PDD) with respect to classes. PDD essentially means that some clients may not have data from all classes, and the paper claims that this would cause issues such as space invasion and mode collapse, hurting FL model quality. The solution is to use a manifold-reshaping approach to prevent degeneration caused by missing local classes. This is done by distributing global prototypes of all classes to all clients as references, establishing proper margins among classes to avoid space invasion. Tests on FMNIST, SVHN, CIFAR10, CIFAR100, and ISIC2019 show that FedMR achieves higher accuracy than other methods like FedAvg, FedPro, and FedProc.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. In Figure 1, the authors show that their approach help avoid space invasion. However, have the authors verified that for other FL methods (e.g., FedProx, FedProc, MOON), they also do not have this space invasion issue? \n\n2. It seems FedMR incurs more computation overhead than other approaches, where computation is the main factor that affects energy consumption on mobile/edge devices. A lite version is included, but the comparison of the lite version is only made against weak baseline FedAvg. Can the authors provide a more direct comparison to SOTA FL approaches in terms of computation overhead and accuracy?\n\n3. Please clarify whether FedMR requires heavy tuning of hyperparameters to achieve its best result. Can you provide a more direct comparison between FedAvg (or any existing method that is more directly comparable to FedMR but is not PDD aware) and FedMR, without tuning any hyperparameters but just adding the manifold reshaping terms? \n\n4. Please improve the writing and readability of the manuscript, e.g.,\n\t- The format of the references appears to be incorrect. References can be placed inside parentheses to distinguish them from the main text. \"Federated learning McMahan et al. (2017); Li et al. (2020a); Yang et al. (2019) has drawn…\" -> Federated learning (McMahan et al. (2017); Li et al. (2020a); Yang et al. (2019)) has drawn…\".\n\t- \"…by adding the proximal regularization on parameters in FedAvg\", the first mention of FedAvg is missing a reference.\n        - \"This motivations a plenty of explorations to address\" -> \"This motivates a plenty of explorations to address\"\n        ... ", "broader_impact_concerns": "No ethical concerns. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "6uplgUb34J", "reviewer": "Reviewer_9MLk", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n------------\n\n1. The paper introduces a novel approach called FedMR specifically designed to tackle the challenges posed by partially disjoint data (PDD) in federated learning (FL). By reshaping the feature space through the incorporation of intra-class and inter-class losses, FedMR addresses the unique collapse and invasion characteristics of PDD, which have been underexplored in previous FL methods.\n\n\n2. The paper provides extensive experimental evaluations on various datasets to demonstrate the effectiveness of FedMR. The results consistently show that FedMR achieves significantly higher accuracy and improved communication efficiency compared to existing FL methods. \n\n3. Overall the paper is nicely written with clear motivation with the help of examples. There are some things which could be improved to make it easier to follow. Please see the weaknesses part for these.\n\n\n\n\n\n\nWeaknesses and questions:\n-------------------------------\n\n1.  Section 3.3.1 lacks intuition and explanations. Please provide some discussion on why uniformity of eigenvalues would prevent dimension collapse? \n\n\n2.  Theorem 1. Assumes the dimensions are not correlated, which is fine to get a result. However, in the discussion the authors mention that the dimensions become uncorrelated by the use the intra-class loss. I cannot see a proof for this.  Also in practice it would probably not be possible to reach perfectly disentangled dimensions by minimizing the intra-class loss? If so, it would be better to generalize the theorem 1 to include this case.\n\n\n3. I like the motivating example (3.2), I am assuming the calculations are without the FedMR. Could you also demonstrate how FedMR works in this example?\n\n4. What is (p_k^c)^t ? \n5. Does theorem 1 depend on the class distributions and overlap across the clients?\n6. Why not use 1-hot vectors for the class prototypes? This could reduce communication cost.\n\t\t\t\t\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper addresses the challenge of statistical heterogeneity in federated learning (FL) and introduces a novel approach called FedMR. Existing FL methods often assume access to samples from all classes during local training, neglecting the practical scenario of partially disjoint data (PDD) where clients contribute only a subset of classes. FedMR reshapes the feature space in local training by incorporating intra-class and inter-class losses. The intra-class loss prevents feature collapse, while the inter-class loss ensures appropriate margins between categories. Experimental results demonstrate that FedMR outperforms existing FL methods in terms of accuracy and communication efficiency. FedMR effectively addresses the challenges posed by PDD, providing a promising solution for FL in real-world scenarios.\n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I don't expect any major changes. It would be great to provide clarification for the points listed above in the weaknesses section. \nIt would be even better to quantify the notion of DPP and study the performance of the method theoretically. ", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper considers a particular form of heterogeneity in federated networks, in which the aim is to solve a multi-class classification and each client only contains a subset of the classes.

All reviewers found that the work solves an interesting/relevant problem and found the empirical results compelling. One remaining concern is that it is unclear exactly why existing methods have suboptimal performance in this scenario, and in particular, whether the hypothesis that they suffer from dimensional collapse/space invasion has been effectively tested. 

I tend to agree with the authors that the work is already meaningful in that it aims to systematically address issues of space invasion/collapse and the proposed modifications seem to clearly improve accuracy in the scenario of interest. However, I also agree with Reviewer q3Gk that it would be beneficial to understand to what degree prior works may inherently address these issues (e.g., using the intuition the authors provided), and to determine whether these are key underlying issues leading to poor performance in this setting. I am thus recommending that the paper be accepted with a minor revision, where I encourage the authors to submit a minor revision of the work that incorporates this feedback via a more detailed discussion, analysis, and/or a simple experiment (e.g., perhaps expanding on the results in Table 7 for more methods) for their final camera-ready.
Decision: Accept (poster)