Federated Learning with Bilateral Curation for Partially Class-Disjoint Data
Authors: Ziqing Fan
Ruipeng Zhang
Jiangchao Yao
Bo Han
Ya Zhang
Yanfeng Wang
Venue: NeurIPS 2023 poster (2023)
Keywords: federated learning
data heterogeneity
partially class-disjoint data

Abstract: Partially class-disjoint data (PCDD), a common yet under-explored data formation where each client contributes a part of classes (instead of all classes) of samples, severely challenges the performance of federated algorithms. Without full classes, the local objective will contradict the global objective, yielding the angle collapse problem for locally missing classes and the space waste problem for locally existing classes. As far as we know, none of the existing methods can intrinsically mitigate PCDD challenges to achieve holistic improvement in the bilateral views (both global view and local view) of federated learning. To address this dilemma, we are inspired by the strong generalization of simplex Equiangular Tight Frame (ETF) on the imbalanced data, and propose a novel approach called FedGELA where the classifier is globally fixed as a simplex ETF while locally adapted to the personal distributions. Globally, FedGELA provides fair and equal discrimination for all classes and avoids inaccurate updates of the classifier, while locally it utilizes the space of locally missing classes for locally existing classes. We conduct extensive experiments on a range of datasets to demonstrate that our FedGELA achieves promising performance (averaged improvement of 3.9% to FedAvg and 1.5% to best baselines) and provide both local and global convergence guarantees.

Federated Learning with Bilateral Curation for Partially Class-Disjoint Data
Ziqing Fan Ruipeng 2 Jiangehao Yao Bo Han', Ya Zhang!2, Yanfeng Wang
'Cooperative Medianet Innovation Center, Shanghai Jiao Tong University,
Shanghaii Al Laboratory, 'Hong Kong Baptist University {zqfan_knight, zhangrp, sunarker)āsje tu. edu. en
bhanml bhanml@comp..o h.b.u...kbb..... edu. hk, [ya_zhang, wangyanf eng) Bs edu.. cn
Abstract
Partially class-disjoint data (PCDD), a common yet under-explored data formation where each client contributes part of classes (instead of all classes) of samples, severely challenges the performance of federated algorithms. Without full classes, the local objective will contradict the global objective, yielding the angle collapse problem for locally missing classes and the space waste problem for locally existing know, none of the existing methods can intrinsically mitigate PCDD challenges to achieve holistic improvement in the bilateral views (both global view and local view) of federated learning. To address this dilemma, we are inspired by the strong generalization of simplex Equiangular Tight Frame (ETF) on the imbalanced data, and propose a novel approach called FedGELA where the classifier is globally fixed as simplex ETF while locally adapted to the personal distributions. Globally, FedGELA provides fair and equal discrimination for all classes and avoids inaccurate updates of the classifier, while locally it utilizes the space of locally missing classes for locally existing classes. We conduct extensive experiments
classes. As far
range of datasets to demonstrate that our FedGELA achieves promising performance (averaged improvement of 3.9% to FedAvg and 1.5% to best baselines) and provide both local and global convergence guarantees. Source code is available at: https://github..omm www.m//eiiaaaannSSSU//eaaa.
Introduction
Partially dissodisjoint data (PCDD) [13, 18, 21] refers to emerging situation in federated learn- ing [14, 22, 43, 46, 50] where each client only possesses information on a subset of categories, but all clients in the federation provide the information
the whole categories. For instance, in landmark detection [39] for thousands of categories with data locally preserved, most contributors only have a subset of categories of landmark photos where they live traveled before; and in the diagnosis of Thyroid diseases, due to regional diversity different hospitals may have shared and distinct Thyroid diseases [10]. It is usually difficult for each party acquire the full classes of samples, as the participants may be lack of domain expertise or limited by demographic discrepancy. Therefore, how efficiently handle the partially class-disjoint data is critical (yet under-explored) problem in real-world federated learning applications for the pursuit of personal and generic interests.
Prevalent studies mainly focus the general heterogeneity without specially considering the PCDD challenges: generic federated leaning (G-FL) algorithms adopt a uniform treatment of all classes and mitigate personal differences by imposing constraints local training [17, 19]. modifying logits [21, 47] adjusting the weights submitted gradients [37] or generating synthetic data [54]; in contrast, personalized federated learning (P-FL) algorithms place relatively less emphasis missing classes and selectively share either partial network parameters [1, 6] or class prototypes [33] minimize the impact of personal characteristics, thereby separating the topics. Those methods
locally
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Expand
Expand
Angle
Space
Pollappe
Classifer Vectur
Strink
Fenture Expand
Feniure Shrink
Classiffer Vector of Massing Clases
Waste of Space
Shrink
Angle Collapse
(a) Ideal Stroeture
(b) Collapsed Structure A (c) Collapeed Straeture B (d) Restricted Structure
Figure 1: Illustration of feature spaces and classifier vectors trained on the global dataset, two partially class-disjoint datasets (A and B), and restricted by federated algorithms. (a) is trained on the globally balanced dataset with full classes. (b) and (c) are trained on datasets A and B, respectively, which suffer from different patterns of classifier angle collapse problems. (d) is averaged in the server or constrained by some federated algorithms.
might directly or indirectly help mitigate the data shifts caused by PCDD, however, as far as we know, none of the existing works can mitigate the PCDD challenges to achieve holistic improvement in the bilateral views (global and local views) of federated learning. Please refer to Table for a comprehensive comparison among range of FL methods from different aspects.
Without full classes, the local objective will contradict the global objective, yielding the angle collapse for locally missing classes and the waste of space for locally existing classes. Ideally, as shown in Figure (a)), global features and their corresponding classifier vectors shall maintain a proper structure to pursue the best separation of all classes. However, the angles of locally missing classes' classifier vectors will collapse, when trained on each client with partially class disso-d data, as depicted in Figure 1(b), 1(c). FedRS [21] notices the degenerated updates of the classifier and pursues the same symmetrical structure in the local by restricting logits of missing classes. Other traditional G-FL algorithms indirectly restrict the classifier by constraining logits, features, or model weights, which may also make effects on PCDD. However, they cause another problem: space waste for personal tasks. As shown in Figure (d), restricting local structure will waste feature space and limit the training of the local model on existing classes. P-FL algorithms utilize the wasted space by selectively sharing part of models but exacerbate the angle collapse of classifier vectors. Recent FedRod [3] attempts to bridge the gap between P-FL and G-FL by introducing a two-head framework with logit adjustment in the G-head, but still cannot address the angle collapse caused by PCDD.
To tackle the PCDD dilemma from both P-FL and G-FL perspectives, we are inspired by a promising classifier structure, namely simplex equiangular tight frame (ETF) [9. 26, 41J. which provides each class the same classification angle and generalizes well on imbalanced data. Motivated by its merits, we propose novel approach, called FedGELA, in which the classifier is Globally fixed a simplex ETF while Locally Adapted to personal tasks. In the global view, FedGELA merges class features and their corresponding classifier vectors, which converge to ETF. In the local view, it provides existing major classes with larger feature spaces and encourages to utilize the spaces wasted by locally missing classes. With such bilateral curation, we can explicitly alleviate the impact caused by PCDD. In a nutshell, our contributions can be summarized as the following three points:
We study a practical yet under-explored data formation in real-world applications of federated learning, termed as partially class- data (PCDD), and identify the angle collapse and space waste challenges that cannot be efficiently solved by existing prevalent methods (Sec. 3.2).
We propose novel method called FedGELA that classifier is globally fixed symmetrical structure ETF while locally adapted by personal distribution (Sec. 3.3), and theoretically show the local and global convergence analysis for PCDD with the experimental verification (Sec. 4.2).
We conduct a range of experiments on multiple benchmark datasets under the PCDD case and real-world dataset to demonstrate the bilateral advantages of FedGELA over the state-of-ther -arthe methods from multiple views like the larger scale of clients and straggler situations (Sec. 5.2). We also provide further analysis like classification angles during training and ablation study. (Sec. 5.3).
Related Work
2.1 Partially Class Class-Disjoint Data and Federated Learning algorithms
Partially class-disjoint data one common formation among clients that can significantly impede the convergence, performncee, and efficiency of algorithms in FL [18]. It belongs to the data heterogeneity

---

Table 1: Key differences between SOTA methods and our FedGELA categorized by targets (P-FL or G-FL), techniques (improve from the views of features, logits or model), and whether directly mitigate angle collapse of classifier vectors or save locally wasted feature spaces caused by PCDD.
Target
Research work
Feature View
Logit View
Model View
Mitigate Collapse
Save Space
FedProx
MOON
G-FI.
FedRS
v
FedGen
FedLC
FedRep
/
/
P-FL
FedProto
FedBABU
V
FedRod
G&P-FI.
FedGELA(ours)
case, but does have a very unique characteristic different from the ordinary heterogeneity problem. That is, if only each client only has a subset of classes, it does not share the optimal Bayes classifier with the global model that considers all classes on the server side. Recently, FedRS [21] has recognized the PCDD dilemma and directly mitigate the angle collapse issue by constraining the logits of missing classes. FedProx [19] also can lessen the collapse by constraining local model weights to stay close to the global model. Other G-FL algorithms try to address data heterogeneity from a distinct perspective. MOON [17] and FedGen [54] utilizes contrastive learning and generative learning to restrict local representations. And FedLC [47] introduces logit calibration to adjust the logits of the local model to match those of the global model, which might indirectly alleviate the angle collapse in the local. However, they all try to restrict local structure as global, resulting in the waste space for personal tasks shown in Figure ((d). P-FL algorithms try utilize the wasted space by encouraging the angle collapse of the local classifier. FedRep [6] only shares feature extractors among clients and FedProto [33] only submits class prototypes save communication costs and align the feature spaces. In FedBABU [25], the classifier is randomly initialized and fixed during federated training while fine- tuned for personalization during the evaluation. However, they all sacrifice the generic performance on all classes. FedRod [3] attempts to bridge this gap by introducing framework with two heads and employing logit adjustment in the global head to estimate generic distribution but cannot address angle collapse. In Table 1, we categorize these methods by targets (P-FL or G-FL), skews (feature, logit, or model weight), and whether they directly mitigate the angle collapse of local classifier or saving personal spaces for personal spaces. It is evident that none of these methods, except ours, can handle the PCDD problem in both P-FL and G-FL. Furthermore, FedGELA is the only method that can directly achieve improvements from all views.
2.2 Simplex Equiangular Tight Frame
The simplex equiangular tight frame (ETF) is phenomenon observed in neural collapse [26], which occurs in the terminal phase of a well-trained model on a balanced dataset. It is shown that the last-layer features of the model converge to within-class means, and all within-class means and their corresponding classifier vectors converge to symmetrical structure. To analyze this phenomenon, some studies simplify deep neural networks as last-layer features and classifiers with proper constraints (layer-peeled model) [9, 12, 40, 53] and prove that ETF emerges under the cross- entropy loss. However, when the dataset is imbalanced, the symmetrical structure of ETF will collapse [9]. Some studies try to obtain the symmetrical feature and the classifier structure on the imbalanced datasets by fixing the classifier as ETF [41, 53]. Inspired by this, we propose novel method called FedGELA that bilaterally curates the classifier to leverage ETF or its variants. See Appendix A for more details about ETF.
3 Method
3.1 Preliminaries
ETF under LPM. A typical L-layer DNN parameterized by W can be divided into the feature backbone parameterized by W-4 and the classifier parameterized by W From the view of layer- peeled model (LPM) [9, 12, 40, 53], training W with constraints on the weights can be considered
3

---

as training the C-class classifier W (Wi.... 3 and features H = ( h")...") of all samples output by last layer of the backbone with constraints Ew and Eµ on them respectively. On the balanced data, any solutions this model form simplex equiangular tight frame (ETF) that all last layer features h and corresponding classifier W of all classes converge as:
h2
W
me:
(I)
VEH
VEw
where m forms the ETF defined as M RAxC U RdC allows rotation and satisfies U Ic and 1, is an all-ones vector. ETF is an optimal classifier and feature structure in the balanced case of LPM.
V C
V dlc1)) Here M [mi.... " ,me
FedAvg. On the view of LPM, given N clients and each with nu samples, the vanilla federated learning via FedAvg consists of four steps [22]: 1) In round t, the server broadcasts the global model W [H', W..t) ; to clients that participate in the training (Note that here H is actually the global backbone W- L,t instead of real features); 2) Each local client receives the model and trains it on the personal dataset. After E epochs, acquire new local model W 3) The updated models are collected to the server as (Wi, W..... W ); 4) The server averages local models to acquire new global model as W = >> Pa Wi, k. where pk- nu = Alg. When the pre-defined maximal round 7 reaches, we will have the final optimized global model WT
3.2 Contradiction and Motivation
Contradiction. In G-FL,, the ideal global objective under LPM of federated learning is described as:
Lce (hi.cs W*)
min H.W
PL
T1k
eEC₁
Assuming global distribution balanced among classes, no matter whether local datasets have full or partial classes, the global objective with constraints on weights can be simplified as:
min 1 H.W.
Loe (hes w s.t. | ||
|
Fw,
Fu-
(2)
Similarly, the local objective of k-th client with a set of classes Ck can be described as:
1 E>Σ
nk.e
Loe (Mi.cs wE). s.t. |
||
Fww; ||ntac| || & F.s-
min
(3)
H2. WE
TER OECN 1=1
When PCDD exists (C& * C). we can see the contradiction between local and global objectives, which respectively forms two structures, shown Figure 3(a) and Figure 3(b). After aggregated in server or constrained by some FL methods, the structure in the local is restricted meet the global structure, causing space waste for personal tasks shown in Figure (d).
100
Angle
90
Nissing Global
Existing Angle Nizsing Angle in Local Existing Angle Local
Motivation. To verify the contradiction and related feature and classifier structures, we split CIFARIO into 10 clients and perform FedAvg on it with Dirichlet Distribution (Dir (3 0.))). As illustrated in Figure 2, the angle difference between existing classes and between missing classes becomes smaller and converges to a similar value in the global model. However, in the local training, angles between existing classes become larger while angles between missing classes become smaller, which indicates the contradiction. With this observation, to bridge the gap between Eq (3) and Eq (2) under PCDD, we need to construct the symmetrical and uniform classifier angles for all classes while encouraging local clients to expand exist- ing classes' feature space. Therefore, we propose our method FedGELA that classifier can be Globally fixed as ETF but Locally Adapted based on the local distribution matrix to
70
10
Rounds
Figure 2: Averaged angles of classi- fier vectors between locally existing classes (existing angle) and between locally missing classes (missing an- gle) on CIFARIO (Dir (3 = 0.1)) in local client and aggregated in global server (local epoch 10). In global, "existing" angle and "missing" angle converge to similar values while in the local, "existing" angle expands but "missing" angle shrinks.
utilize the wasted space for the existing classes.

---

3.3 FedGELA
Global ETF. Given the global aim of achieving an unbiased classifier that treats all classes equally and provides them with the same discrimination and classifier angles, we curate the global model's classifier as a randomly initialized simplex ETF with scaling VEw at the start of federated training:
w4 M.
Then the ETF is distributed to all clients to replace their local classifiers. In Theorem 1, we prove in federated training under some basic assumptions, by fixing the classifier as a randomly simplex ETF with scaling v Ew and constraints Eн on the last layer features, features output by last layer of backbone and their within class means will converge to the ETF similar to Eq (I), which meets the requirement of global tasks.
Local Adaptation. However, when PCDD exists in the local clients, naively combining ETF with FL does not meet the requirement of P-FL as analyzed in Eq (2) and Eq (3). To utilize the wasted space for locally missing classes, in the training stage, we curate the length of ETF received from the server based on the local distribution as below:
(4) where 4p is the distribution matrix of k-th client. Regarding the selection of 4r, it should satisfy basic rule for federated learning, wherein the aggregation of local classifiers aligns with the global classifier, thereby ensuring the validity of theoretical analyses from both global and local perspectives. Moreover, it is highly preferable for the selection process to avoid introducing any additional privacy leakage risks. To meet the requirement that averaged classifier should be standard ETF: W = Σ Pk: W in the globally balanced case, its row vectors are all one's vector multiple statistical values of personal distribution: (+7))
W $ W V M,
Ri.c 1 (7 is constant, and nee and n& are the nz7
e-th class sample number and total sample number of the k-th client) respectively. We set y [C] Finally, the local objective from Eq. (3) is adapted as:
G
mk.e
exp( 1 R.c W hi.e)
min Hy
log
Eeec exp( 2 k W k2,)
(5)
s.t. ||0|| & <i ne.
Total Framework. After introducing two key parts of FedGELA (Global ETF and Local Adaptation), we describe the total framework of FedGELA. As illustrated and highlighted in Algorithm (refer to Appendix D for the work- flow figure), the initializing stage, the server randomly generates an ETF as the global classi- fier and sends it to all clients while local clients adjust it based on the personal distribution ma- trix as Eq (4). At the training stage, local clients receive global backbones and train with adapted ETF in parallel. After E epochs, all clients sub- mit personal backbones to the server. In the server, personal backbones are received and ag- gregated to generic backbone, which is broad- cast all clients participating the next round. At the inference stage, on the client side, we obtain a generic backbone with standard ETF to handle the world data while on the client side, personal backbone with adapted ETF handle the personal data.
Algorithm 1 FedGELA
Input:(N, K, me, Ck2 H P,,, Ew, Eg, T, E)
Parallelly for all elients: W 4x E/ M.
for 0, T do
D on the server side
H' + =K Hi
sample K clients from all N clients.
the client side do in parallel for vk K clients
receive H from server, H; + H'.
for r = 0, 1,.,, E 1 do sample mini-batch _ in local data.
H + H F. W.W end for submit H; to the server. end in parallel end for Output:(H' W:") and (H) 4 W.
4 Theoretical Analysis
In this part, we first primarily introduce some notations and basic assumptions in Sec. 4.1 and then present the convergence guarantees of both local models and the global model under the PCDD with the proper empirical justification and discussion in Sec. 4.2. (Please refer to Appendix for entire proofs and Appendix D for details on justification experiments.)
5

---

Expand
90
Angle
60
Angle
60
30
FedAwa Local
FedETF in Global
PedETF in Local FedGELA Local
FedGELA Global
50
100
40
80
120
160
Shrink
Abandon
Communication Rounds
Local Epochs
(a) Global convergence verification.
(b) Local convergence verification.
(c) Adapted structure.
Figure 3: Illustration of local and global convergence verification together with the effect of 4b. (a) and (b) are the results of averaged angle between all class means and between locally existing class means FedAvg, FedGE, and FedGELA on CIFARIO under 50 clients and Dir (3 0.2). (c) the illustration of how local adaptation utilizes the wasted space of missing classes for existing classes,
4.1 Notations
We use f and T to denote curtain round and pre-defined maximum round after aggregation in federated training, tE denote the state that just finishing local training before aggregation in round and tE + to denote 7-th local iteration in round t and 0 S S E 1. The convergence follows some common assumptions in previous FL studies and helpful math results [15, 20, 29- 31, 33, 36, 38, 45, 51] including smoothness, convexity on loss function Fi, F2. ...... FN of all clients, bounded norm and variance of stochastic gradients on their gradient functions VF, F2..... FN and heterogeneity I'1 reflected the distance between local optimum W and global optimum W Please refer to the concrete descriptions those assumptions in Appendix B. Besides, in Appendix B, we additionally provide a convergence guarantee without bounded norm of stochastic gradients, as some existing works [24, 32] point out the contradiction to the strongly convex.
4.2 Convergence analysis
Here we provide the global and local convergence guarantee of our FedGELA compared with FedAvg and FedGE (FedAvg with only the Globally Fixed ETF) in Theorem I and Theorem 2. To better explain the effectiveness of our FedGELA in local and global tasks, we record the averaged angle between all class means in global and existing class means in local as shown in Figure 3(a) and Figure 3(b). Please refer to Appendix B for details on the proof and justification of theorems.
Theorem (Global Convergence). !f Fi,... FN are all L-smooth, j--ttrongy convex, and the vari- ance and norm of F...... ..., FN bounded by and G. Choose K = L/H and y max (8, E), for all classes e and sample expected global representation by cross-entropy loss will converge to: (WL. KE
E
log
2B
MY 2
E
W
-w1I)
(W
7+
where in FedGELA, B En (v202 2 Pk & WL W GLT, 8(E 1)2 Since WL WL. and (W...)ThE 2 B[(W4)Thi h., hi will converge hig
In Theorem the variable B represents the impact of algorithmic convergence (pio²), non-iid data distribution () and stochastic optimization (S(E G)²G²). The only difference between FedAvg, FedGE, and our FedGELA lies in the value of B while others are kept the same. FedGE and FedGELA have smaller G compared to FedAvg because they employ fixed ETF classifier that is predefined as optimal. FedGELA introduces minor additional overhead (p&l $ x W W I)) on the global convergence of FedGE due the incorporation of local adaptation to ETFs The cost might be negligible, as a, G, and I' are defined on the whole model weights while pk 22 W w is defined the classifier. To verify this, we conduct experiments Figure 3(a), and as can be seen, FedGE and FedGELA have similar quicker speeds and larger classification angles than FedAvg.
Theorem 2 (Local Convergence). If Fi,..., FN are L-smooth, variance and norm of their gradients are bounuled by and G, and the heterogeneity is bounded by 11, clients' expected local loss satisfies:
B[r(t+1)8
F.B
LEni
o² +TT-A,
2
where in FedGELA, A - & WT. there exist learning rate tμ making the expected local loss decreasing and converging.
Will, which means if A- LE(G)+02) $ 0.
G

---

In Theorem 2. only "A" is different on the convergence among FedAvg, FedGE, and FedGELA. Fixing the classifier as ETF and adapting the local classifier will introduce smaller G and additional cost of L WL ww1 respectively, which might limit the speed of local convergence. However, FedGELA might reach better local optimal by adapting the feature structure. As illustrated in Figure 3 (c), the adapted structure expands the decision boundaries of existing major classes and better utilizes the feature space wasted by missing classes. To verify this, in Figure 3(b), we record the averaged angles between the existing class means during the local training. It can be seen that FedGELA converges to a much larger angle than both FedAvg and FedGE, which suits our expectations. More angle results can be seen in Figure 5.
Experiments
5.1 Experimental Setup
Datasets. We adopt three popular benchmark datasets SVHN [23], CIFARIO/100 [16] in federated learning. As for data splitting, we utilize Dirichlet Distribution (Dir (3), == (10000, 0.5. 0.2, 0.13) to simulate the situations of independently identical distribution and different levels of PCDD. Besides, one standard real- world PCDD dataset, Fed-ISIC2019 [4, 7, 34, 35] is used, and we follow the setting in the Flamby benchmark [34]. Please refer to Appendix C for more details.
Denote PA as the personal accuracy, which is the mean of the accuracy computed on each client test dataset, and GA as the generic accuracy on global test dataset (mixed clients' test datasets). Since there is no global model P-FL methods. we calculate GA of them as the averaged accuracy of all best local models on global test dataset, which is the same as FedRod [3]. Regarding PA, we record the best results of personal models for P-FL methods while for G-FL methods we fine-tune the best global model in 10 epochs and record the averaged accuracy on all client test datasets. For FedRod and FedGELA, we can directly record the GA and PA (without fine-tuning) during training.
Metries.
Implementation. We compare FedGELA with FedAvg, FedRod [3], multiple state-of-the-art methods in G-FL (FedRS [21], MOON [17], FedProx [19], FedGen [54] and FedLC [47]) and in P-FL (FedRep [6]. FedProto [33] and FedBABU [25]). For SVHN, CIFARIO, and CIFAR100, 100, we adopt a commonly used Re [8, 17, 47, 48, 52] with one FC layer as the backbone, followed by a layer of classifier. FedGELA replaces the classifier as simple ETF, We use SGD with learning rate 0.01, weight decay 10 4, and momentum 0.9. The batch size is set as 100 and the local updates are set as 10 epochs for all approaches. As for methoo-specifii syper-parameet in FedProx, we tune it carefully. features with length (En = 1) and only tune the length scaling of classifier (Ew). All methods are implemented by PyTorch [27] with NVIDIA GeForce RTX 3090. See detailed information in Appendix C.
like the proximal term our method, there are Ew and En need to set, we normalize
5.2 Performance of FedGELA
In this part, we compare FedGELA with FedAvg, FedRod, three SOTA methods of P-FL (FedRep, FedProto, and FedBABUU, four SOTA methods of G-FL (FedProx, MOON, FedRS, FedLC and FedGen) on different aspects including the scale of clients, the level of PCDD, straggler situations, and real-world applications. Similar recent studies [8, 17, 44], we split SVHN CIFARIO, and CIFAR 100 into 10 and 50 clients and each round select 10 clients to join the federated training, denoted as full participation and partial participation (straggler situation), respectively. With the help of Dirichlet distribution [11], we verify all methods on IID, Non-IID (3 = 0.5), and extreme Non-III situations (3 = 0.1 or B 0.2). As the decreasing B, the level of PCDD increases and we show the heat map of data distribution in Appendix C. We set 3 0.2 in partial participation to make sure each client has at least one batch of samples. The training round for SVHN and CIFARIO is 50 in full participation and 100 in partial participation while for CIFAR 1 it is set to 100 and 200. Besides, we also utilize a real federated scenario Fed-ISIC2019 to verify the ability to real-world application.
Full participation and partial participation. As shown in Table 2, with the decreasing / or increasing number of clients, the generic performance of FedAvg and all other methods greatly drops while the personal performance of all methods greatly increases. This means under PCDD and the straggler problem, the performance of generic performance is limited but the personal distribution is easier to capture. As for P-FL methods, they fail in global tasks especially in severe PCDD
7

---

Table 2: Personal and generic performance on SVHN, CIFARIO, and CIFAR100. We use Dir (3
0.5) for medium heterogeneity and Dir (8 = 0.1) or Dir (33 = 0.2) for high-level heterogeneity. To verify the straggler situation, we split all datasets into 10 or 50 clients for full participation or partial participation, and in each round, 10 clients are selected in the federated training.
Method
Full Participation (10, 10)
Partial Participation (50, 10)
Dataset
#Partition
IID
B 0.5
B
0.1
IID
3
0.5
G = 0.2
#Metric
GA
PA
GA
PA
GA
GA
PA
GA
GA
FedAvg
93.01
92.61
93.95
91.24
98.10
75.24
91.44
91.29
92.70
89.29
95.31
84.70
FedProx
93.12
93.12
98.71
92.15
97.98
75.13
91.67
91.66
92.71
89.98
95.13
85.68
MOON
93.16
93.16
92.98
92.46
98.06
76.21
93.49
91.41
91.86
90.20
95.78
86.22
FedRS
93.29
93.21
93.92
92.33
98.04
76,26
91.63
91.59
93.51
91.70
96,20
87.78
FedGen
94.02
93.99
94.47
92.66
98.22
76.51
91.47
91.33
93.67
91.35
95.77
87.59
SVHN
FedLC
93.29
93.28
94.76
91.20
98.24
76.17
91.69
91.67
92.73
91.02
95.20
86.92
FedRep
93.01
92.61
94.77
91.24
97.87
68.52
91.77
89.20
93.14
80.94
95.38
67.77
FedProto
93.21
91.68
94.48
85.85
98.26
56.49
90.23
87.27
93.28
76.59
95.62
54.92
FedBABU
93.26
93.08
95.20
92.04
98.16
75.52
93.69
91.05
93.54
90.49
96.70
84.42
FedRod
93.50
93.22
95.47
92.09
98.06
76. 24
92.04
91.65
93.96
91. 20
95.68
86,98
FedGELA
94.84
94.66
96.27
93.66
98.52
78.88
94.68
93.59
95.54
93.29
96.85
89.58
FedAvg
73.17
72.8
81.67
67.28
92.66
54.57
66. 88
66.
7U.64
61.81
80.04
49.13
FedProx
73.69
73.69
81.95
67.53
92.94
56.13
67.67
67.27
73.62
60.80
80.66
50.82
MOON
73.29
73.29
82.27
68.34
92.90
55.61
67 58
67.56 58
74.64
61.81
83,42
52.19
FedRS
73.56
72.94
81.59
68.10
92.57
58.19
66.76
66.52
72.21
58.95
81.11
51.66
FedLC
73.05
73.00
81.99
67.97
92.48
57.02
67 46
67 13
72.57
61.31
82.14
65.15
CIFARIO
FedGen
73.72
73.49
82.22
69.33
92.79
58.04
68.74
68.02
75.52
62.44
81.07
53.46
FedRep
73.42
73.23
$3.30
47.96
92.92
38.32
67.85
67.74
77.28
42.64
84.52
33.22
FedProto
67.06
66.74
$1.03
46,99
93.17
32.13
61.85
52.76
72.89
37.47
81.73
26.07
FedBABU
73.86
72.30
81.40
65.03
92.94
53.65
66.99
64.90
77.59
58.17
82.92
49.90
FedRod
74.24
73.76
82.34
70.74
92.27
58.86
70.09
70.04
78.23
64.13
84.63
58.86
FedGELA
75.02
74.07
84.52
72.73
94.28
61.57
72.33
72.04
80,96
65.08
86.55
60.52
FedAvg
65.27
65.27
65.59
63.96
76.43
59.17
55. 16
55.29
55.36
54.15
58.85
53.39
FedProx
65.71
65.71
65.31
64.18
75.95
59.93
56.86
56. 89
56.89
56.08
59.27
55.25
MOON
65.33
65.33
65.23
64.79
75.45
60.12
56.91
56.86
56.72
56.14
59.51
55.53
FedRS
65.18
65.64
66.48
64.62
76.86
60.74
56.51
55.91
56.45
56.34
61.92
55.99
FedGen
65.74
65.75
66.72
64.33
76.92
60.43
56.77
56.74
57.43
56.27
60.09
55.27
CIFAR100
FedLC
65.83
65.84
65.91
65.02
75.67
60.07
56.87
56.04
56.56
56.28
60.89
55.95
FedRep
61.21
59.21
67.87
52.51
77.81
42.77
53.41
51.44
55.60
48.67
67.70
33.10
FedProto
56.56
56.26
66.08
46.88
77.68
37.63
52.41
50.04
54.05
42.88
63.22
28.74
FedBABU
65.63
65.28
71.30
64.54
80.33
60.99
56.91
54.57
60.14
54.40
68.44
54.24
FedRod
66. 17
66.17
72.05
65.19
80.46
61.01
57. 76
57 01
63.90
56.53
72.37
54.67
FedGELA
67.28
68.07
72.61
66.94
82.79
63.13
61.70
59.29
64.37
58.60
72.93
58.53
Table 3: Personal and generic performance on a real federated application Fed-ISIC2019. More results of other realworld dataset shown in the Appendix.
Method|
FedAvg
FedProx
MOON
FeaRS
FedGen
FedL.C
FedRep
FedProto
FedRABU
FedRod
FedGELA
PA
77.27 27 37 28
28 77.58 2u 76. 16. 77.80 78.91 20 78. 78.65 79.27 39.39
GA
73 17 73.69 73 a 74. 15 74.3720.77 74.20 as 68 ar 66.26 tn 74 74.98+0.21 75.85
situations since they do not consider the global convergence during training. As for G-FL methods, the performance better in generic tasks but limited in personalized tasks, especially in CIFAR 100. They constrain the model's ability to fit personalized distributions during local training, resulting in improved consistency during global optimization. As can be seen, our FedGELA consistently exceeds all basclines for all settings with averaged performance of 2.42%, 5.2% and 5.7% to FedAvg and 1.35%, 1.64% and 1.81% % to the best bascline on the three datasets respectively.
Performance in real-world applications. Except for the above three benchmarks, we also verify FedGELA with other methods under real PCDD federated application: Fed-ISIC2019. As shown in Table 3, our method achieves the best improvement of 2% and 2.26% relative to FedAvg and of 0.36% and 1.25% relative to the best baseline on personal and generic tasks respectively, which demonstrates that our method is robust to practical situations in the both views. In the Appendix D.5, we provide more results on additional two real- world applications named FEMNIST and SHAKESPEARE to further show the effectiveness of our method in the real-world scenarios.
8

---

0.69
Actimny
0.91
Ours
Accureey
Durs
Accuracy
Our
0.73
Our Reddug
0.09
Fedkeg
FedAug
0.00
Foddug
0.774
0.58
0.23
0.97
CFORAD Bucall
Hucall
0.03
BSC Iuccab
0.00
0.20
Accunscy
Accumeyy
Accomcy
Ours FeddAug
Ours FediAgg
Ours FedAug
Ours FedAug
0.03
800
0.00
0.22
0.71
0.82
0.77
Figure 4: Bilateral performance on four datasets by tuning log (x axis) of FedGELA.
%
90
Angle
Anple
60
30
30
30-
In Goba
PEOGELA Glosal
TeoGZLA Gubba
50
100
100
50
100
Communication Rounds
Communication Rounds
Communcatioo Rounds
(a) Global, Full Parti., 8 = 0.5 (b) Global, Full Parti., 3 = 0.1 (c) Global, Partial Parti., $ 0.5
120
120
120
Angle
60
Angle
60
Angle
60
Tedligg Local
Reddig
hal22 Local
Podit Local
PudGE iocal
0-
TedGiLAA local
0
FedGRLA Local
PediTTA local
40
B0
120
160
40
80
120
160
40
80
120 160
Local Epochs
Local Epochs
Local Epochs
(d) Local, Full Parti., B = 0.5
(c) Local, Full Parti., 8 = 0.1 (f) Local, Partial Parti., 3 = 0.5
Figure 5: Illustration of the averaged angle between locally existing classes and missing classes on the local client and global server of FedAvg, FedGE, and our FedGELA on CIFARIO.
Table Ablation study of FedGELA. GE and LA mean the global ETF and local adaptation.
GE
LA
SVHN
CIFARIO
CIFARI00
Fed-ISIC2019
#Purtition
Full Parti.
Partial Parti.
Full Parti.
Partial Parti.
Full Parti.
Partial Parti
Real World
#Merii
PA
GA
PA
GA
PA
GA
PA
GA
PA
GA
PA
GA
PA
GA
195.02
86.36
93.15
88.43
82.50
64,88
72.52
59.19
69.09
62.80
56.46
54.28
77.27
73.59
95.92
88.93
93.97
92.42
83.63
69.70
77.66
65.56
71.46
66.02
62.67
58.98
69.88
75.54
95.93
74.84
93.15
89.58
83.97
63.75
77.76
61.55
71.93
60.76
58.92
51.95
54.65
62.43
96.55
89.07
95.69
92.15
84.61
69.46
79.95
65.21
74.23
66.05
66.33
58.81
79.27
75.85
5.3 Further Analysis
More angle visualizations. In Figure 5, we show the effectiveness of local adaptation in FedGELA and verify the convergence of fixed classifier as ETF and local adaptation compared with FedAvg. Together with Figure 3, it can be seen that, compared with FedAvg, both FedGE and FedGELA converge faster to larger angle between all class means in global. In the meanwhile, the angle between existing classes of FedGELA in the local is much larger, which proves FedGELA converges better than FedAvg and the adaptation brings little limits to convergence but many benefits to local performance under different levels of PCDD.
Hyper-parameter. FedGELA introduces constrain Eh the features and the length Ew of classifier vectors. We perform L2 norm on all features in FedGELA, which means En = 1. For the length of the classifier, we tune it as hyper-parameter As shown in Figure 4, from a large range from 10e3 to 10e6 of Ew, our method achieves bilateral improvement compared to FedAvg on all datasets.
Ablation studies. Since our method includes two parts: global ETF and local adaptation, we illustrate the average accuracy of FedGELA on all splits of SVHN, CIFAR 10/100, and Fed-ISIC2019 without the global ETF or the local adaptation or both. As shown in Table 4, only adjusting the local classifier does not gain much in personal or global tasks, and compared with FedGE, FedGELA achieves similar generic performance on the four datasets but much better performance on the personal tasks.

---

Table 5: Performance of FedGELA compared with FedAvg and the best baseline under pure PCDD settings on CIFARIO and SVHN datasets. PoC's means that the dataset is divided into e clients and each client has classes. We show the improvement in red on each baseline compared to FedGELA.
Datasct (split)
Metric
FedAvg
Best Bascline
FedGELA
PA
92.08+3.76
94.07+1.77
95.84
CIFARIO(PI0C2)
GA
47.26+12.34
52.02+7.58
59.60
PA
91.74+3.68
93.22+2.20
95.42
CIFARIO(PSOC2)
GA
36.22+18.556
44.74+110.04
54.78
PA
95,64+3.11
97.02+1.73
98,75
SVHN(PI0C2)
GA
69.34414.22
76.06+7.50
83.56
PA
94.87+3.50
96.88+1.49
98.37
SVHN(P50C2)
GA
66.94+10.22
72.97+4.21
77.18
Table 6: Performance of choosing different Φ. Assuming the row vector of distribution matrix ( related to class distribution Mike and the relationship as Qx (*t,e) ). Except for Qu (x) 2,, we have ng
ny also considered employing alternative methods like employing an exponential Qx(x)) = e or power function Qu (z) = . of the number of samples.
Dataset (split)
Metric
(x) = e.
Qx(T) = .
Qx(T) = (>uurs)
PA
95.12
95.43
94.84
SVHN(ID)
GA
94.32
93.99
94.66
PA
96.18
95.56
96.27
SV = 0.5)
GA
93.28
93.22
93.66
PA
98.33
98.21
98.52
SVHN(B 0.1)
GA
78.95
77.18
78.88
Performance under pure PCDD setting. To verify our method under pure PCDD, we decouple the PCDD setting and the ordinary heterogeneity (Non-PCDD). In Table 5, the dataset divided in to x clients with classes, and in each round, 10 clients are selected into federated training. The training round is 100. According the results, FedGELA achieves significant improvement especially 18.56% to FedAvg and 10.04% to the best baseline on CIFARIO (P50C2).
PxCy to denote
Other types of 4. Considering the aggregation of local classifiers should align with the global classifier, which ensures the validity of theoretical analyses from both global and local perspectives,
Ek= Pk P should be (1 is all-one matrix). Assuming the row vector of distribution matrix (Φx)? is related to class distribution ahes and the relationship as Q& Pi.s)
nu The equation can be rewrite as: 7 E Pa (.Q(((())) (bis)) = 1, where 7 is the scaling constant. In our FedGELA, avoid sharing statistics for privacy, we only find one potential way that (=(*Aee The and 7 &. In this part, 7
we have also considered employing alternative methods like employing an exponential or power function of the number of samples. As shown the Table other methods need to share Qu( (*k.s) but achieve the similar performance compared to FedGELA, which exhibits the merit of our choice.
In Appendix D, we provide more experiments from other perspectives like communication efficiency and the local burden of storing and computation, to show promise of FedGELA.
Conclusion
In this work, study the problem of partially class-disjoint data (PCDD) in federated learning on both personalized federated learning (P-FL) and generic federated learning (G-FL), which is practical and challenging due the angle collapse of classifier vectors for the global task and the waste space for the personal task. We propose a novel method, FedGELA, to address the dilemma via bilateral curation. Theoretically, we show the local and global convergence guarantee of FedGELA and verify the justification on the angle of global classifier vectors and on the angle between locally existing classes. Empirically, extensive experiments show that FedGELA achieves promising improvements on FedAvg under PCDD and outperforms fftt--ofthhe-as methods in both P-FL and G-FL.
10

---

Acknowledgement
The work is supported by the National Key R&D Program of China (No. 20227I01160002, STCSM (No. 2251 106101, No.
No. 21DZ1100100; 111 plan (No. BP0719010) and National Natural Science Foundation of China (No. 62306178). Ziqing Fan and Ruipeng Zhang were partially supported by Wu Wen Jun Honorary Doctoral Scholarship. Al Institute, Shanghai Jiao Tong University. Bo Han was supported by the NSFC Young Scientists Fund No. 62006202, NSFC General Program No. 62376235, Guangdong Basic and Applied Basic Research Foundation No. 2022A151501 1652, and CCF-Baide Open Fund.
References
[1] Manoj Ghuhan Arivazhagan, Vinay Aggarwal, Aaditya Kumar Singh, and Sunav Choudhary. Federated learning with personalization layers. arXiv preprint arXiv: urXiv:1912.00818 2019.
[2] Sehastian Caldas, Sai Meher Karthik Duddu, Peter Wu, Tian Li, Jakub Konečný, H Brendan McMahan, Virginia Smith, and Ameet Talwalkar. Leaf: A benchmark for federated settings. arXiv preprint 10X2.0811000109 2018.
[3] Hong-You You Chen and Wei-Lun Chao. On bridging generic and personalized federated learning for image classification. In International Conference on Learning Representations, 2022.
[4] Noel CF Codella, David Gutman, et al. Skin lesion analysis toward melanoma detection: A challenge at the 2017 international symposium on biomedical imaging (isbi), hosted by the international skin imaging collaboration (isic). In ISBI, pages 168 168-1728 172, 2018.
[5] Gregory Cohen, Saced Afshar, Jonathan Tapson, and Andre Van Schaik. Emnist: Extending mnist to handwritten letters. In 2017 international joint conference on neural networks (IJCNN), pages 2921-299 2926. IEEE, 2017.
[6] Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai. Exploiting shared representations for personalized federated learning. In International Conference on Machine Learning, pages 2089-2099. PMLR, 2021.
[7] Marc Combalia, Noel CF Codella, Veronica Rotemberg, Brian Helba, Veronica Vilaplana, Ofer Reiter, Cristina Carrera, Alicia Barreiro, Allan C Halpern, Susana Puig, al. Ben20000: Dermoscopic lesions in the wild. arXiv preprint arXiv:7908.000
2019.
[8] Ziqing Fan, Yanfeng Wang, Jiangchao Yao, Lingjuan Lyu. Ya Zhang, and Qi Tian. Fedskip: Combatting statistical heterogeneity with federated skip aggregation. In 2022 IEEE International Conference on Data Mining (ICDM), pages 131-140, 2022. doi: 10. 109/LCDM5448442022
00023.
[9] Cong Fang, Hangfeng He, Qi Long, and Weijie J Su. Exploring deep neural networks via layer-peeled model: Minority collapse imbalanced training. Proceedings of the National 2021.
Academy of Sciences, 1 (18(43):e10039111
[10] Eduardo Gaitan, Norman C Nelson, and Galen V Poole. Endemic goiter and endemic thyroid disorders. World journal of surgery, 15(2):005-215, 1991.
[11] Jonathan Huang. Maximum likelihood estimation of dirichlet distribution parameters. CMU Technique Report, pages 1-9, 2005.
[12] Wenlong Ji, Yiping Lu, Yiliang Zhang, Zhun Deng, and Weijie Su. An unconstrained layer- peeled perspective on neural collapse. In International Conference on Learning Representations, 2021.
[13] Yilun Jin, Yang Liu, Kai Chen, and Qiang Yang. Federated learning without full labels: A survey. arXiv preprint arXiv:2303. 100003./4433 /4453, 2023.
[14] Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Ar- jun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings. al. Advances and open problems in federated learning. Foundations and Trends Machine Learning, 14:1-210, 2021.

---

[15] Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, et al. Scaffold: Stochastic controlled averaging for federated learning. In ICML, pages 5132-5143, 2020.
[16] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Toronto, ON, Canada, 2009.
[17] Qinbin Li, Bingsheng He, and Dawn Song. Model-contrastive federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10713-10722, 2021.
[18] Qinbin Li, Yiqun Diao, Quan Chen, and Bingsheng He. Federated learning on non-iid data silos: An experimental study. In 2022 IEEE 38th International Conference on Data Engineering (ICDE), pages 965- 166-978. IEEE, 2022.
[19] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Systems, 2:429-450, 2020.
[20] Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. On the convergence of fedavg on non-iid data. In International Conference on Learning Representations, 2020.
[21] Xin-Chun Li and De-Chuan Zhan. Fedrs: Federated learning with restricted softmax for label distribution non-iid data. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery Data Mining, pages 995- 2021.
[22] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficeen learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273-1282. PMLR, 2017.
[23] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop Deep Learning and Unsupervised Feature Learning 2011, 2011.
[24] Lam Nguyen, Phuong Ha Nguyen, Marten Dijk, Peter Richtárik, Katya Scheinberg, and Martin Takác. Sgd and hogwild! convergence without the bounded gradients assumption. In International Conference on Machine Learning, pages 3750-3758. PMLR, 2018.
[25] JAE HOON OH, Sangmook Kim, and Seyoung Yun. Fedbabu: Toward enhanced representation for federated image classification, In 10th International Conference on Learning Representa- tions, ICLR 2022. International Conference on Learning Representations (ICLR), 2022.
[26] Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117 (40):24652 24663, 2020.
[27] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32:8026-8037, 2019.
[28] William Shakespeare et al. William Shakespeare: the complete works. Barnes & Noble Publishing, 1989.
[29] Sebastian U. Stich. Local SGD converges fast and communicates little. In International Conference Learning Representations, 2019.
[30] Sebastian U Stich. Unified optimal analysis of the (stochastic) gradient method. arXiv preprint 1rXiv:1907.04232, 2019.
[31] Sebastian U Stich, Jean- Baptist Cordonnier, and Martin Jaggi. Sparsified sgd with memory. Advances Neural Information Processing Systems, 31, 2018.
[32] Canh T Dinh, Nguyen Tran, and Josh Nguyen. Personalized federated learning with moreau envelopes. Advances in Neural Information Processing Systems, 33:21394-214055 2020.
12

---

[33] Yue Tan, Guodong Long, Lu Liu, Tianyi Zhou, Qinghua Lu, Jing Jiang, and Chengqi Zhang. Fedproto: Federated prototype learning across heterogeneous clients. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 8432-8440, 2022.
[34] Jean Ogier du Terrail, Samy- Samy-Safwan Ayed, Edwige Cyffers, Felix Grimberg, Chaoyang He, Regis Loeb, Paul Mangold, Tanguy Marchand, Othmane Marfoq, Erum Mushtaq, et al. Flamby: Datasets and benchmarks for cross-silo federated learning in realistic healthcare settings. arXiv preprint arXiv: 10. 10.04620, 2022.
[35] Philipp Tschandl, Cliff Rosendahl, and Harald Kittler. The ham dataset, large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific data, 5(1): 1-9, 2018.
[36] Jianyu Wang and Gauri Joshi. Cooperative sgd: A unified framework for the design and analysis of local-update sgd algorithms. The Journal of Machine Learning Research, 22:9709-9758, 2021.
[37] Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H. Vincent Poor. Tackling the objective inconsistency problem in heterogeneous federated optimization. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 7611-7623. Curran Associates, Inc., 2020.
[38] Jianyu Wang, Zheng Xu, Zachary Garrett, Zachary Charles, Luyang Liu, and Gauri Joshi. Local adaptivity in federated learning: Convergence and consistency. arXiv preprint arXiv:2106.02::3 2021.
[39] Tobias Weyand, Andre Araujo, Bingyi Cao, and Jack Sim. Google landmarks dataset v2-a large- scale benchmark for instance-level recognition and retrieval. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 2575-2584, 2020.
[40] Lechao Xiao, Jeffrey Pennington, and Samuel Schoenholz. Disentangling trainability and generalization in deep neural networks. In International Conference on Machine Learning, pages 10462-10472. PMLR, 2020.
[41] Yibo Yang, Shixiang Chen, Xiangtai Li, Liang Xie, Zhouchen Lin, and Dacheng Tao. Inducing neural collapse in imbalanced learning: Do we really need a learnable classifier at the end of deep neural network? In Advances in Neural Information Processing Systems, 2022.
[42] Jiangchao Yao, Shengyu Zhang, Yang Yao, Feng Wang, Jianxin Ma, Jianwei Zhang, Yunfei Chu, Luo Ji, Kunyang Jia, Tao Shen, et al. Edge-cloud polarization and collaboration: A comprehensive survey for ai. IEEE Transactions on Knowledge and Data Engineering, 35(7): 6866 6886, 2022.
[43] Rui Ye, Zhenyang Ni, Fangzhao Wu, Siheng Chen, and Yanfeng Wang. Personalized federated learning with inferred collaboration graphs. 2023.
[44] Rui Ye, Mingkai Xu, Jianyu Wang. Chenxin Xu, Siheng Chen, and Yanfeng Wang. Feddisco: Federated learning with discrepancy- awaree collaboration. arXiv preprint arXiv:2305.19229, 2023.
[45] Hao Yu. Sen Yang, and Shenghuo Zhu. Parallel restarted sgd with faster convergence and less communication: Demystifying why model averaging works for deep learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 5693-5700, 2019.
[46] Honglin Yuan, Warren Richard Morningstar, Lin Ning. and Karan Singhal. What do we mean by generalization in federated learning? In ICLR, 2022.
[47] Jie Zhang, Zhiqi Li, Bo Li, Jianghe Xu, Shuang Wu, Shouhong Ding, and Chao Wu. Federated learning with label distribution skew via logits calibration. In International Conference on Machine Learning, pages 26311-26322 PMLR, 2022.
[48] Ruipeng Zhang, Qinwei Xu, Chaoqin Huang, Ya Zhang, and Yanfeng Wang. Semi-supervised domain generalization for medical image analysis. 2022 IEEE 19th International Symposium on Biomedical Imaging (ISBI), pages 1-5. IEEE, 2022.
13

---

[49] Ruipeng Zhang, Ziqing Fan, Qinwei Xu. Jiangchao Yao, Ya Zhang, and Yanfeng Wang. Grace: generalized and personalized federated learning method for medical imaging. International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 14-24. Springer, 2023.
[50] Ruipeng Zhang, Qinwei Xu, Jiangchao Yao, Ya Zhang, Qi Tian, and Yanfeng Wang. Feder- ated domain generalization with generalization adjustment. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3954-3963, 2023.
[51] Yuchen Zhang, Martin J Wainwright, and John C Duchi. Communication--fficcenn algorithms for statistical optimization. Advances neural information processing systems, 25, 2012.
[52] Zhihan Zhou, Jiangchao Yao, Yan-Feng Wang, Bo Han, and Ya Zhang. Contrastive learning with boosted memorization. In International Conference on Machine Learning, pages 27367-27377. PMLR, 2022.
[53] Zhihui Zhu, Tianyu Ding, Jinxin Zhou, Xiao Li, Chong You, Jeremias Sulam, and Qing Qu A geometric analysis of neural collapse with unconstrained features. Advances in Neural Information Processing Systems, 34:29820-29834, 2021.
[54] Zhuangdi Zhu, Junyuan Hong. and Jiayu Zhou. Data-free knowledge distillation for heteroge- neous federated learning. In International conference machine learning, pages (877-12889) PMLR, 2021.
14


Reviews:
[{"review_id": "hTdozBYRdV", "reviewer": "Reviewer_nvCV", "summary": "The authors study the problem of federated learning over partially class-disjoint data and propose using equiangular tight frame (ETF) techniques that allows achieving better performance in both the global and personal learning tasks. They show that the existing federated learning approaches suffer either from angle collapse for locally missing classes or from waste of space for locally existing classes, and propose their approach FedGELA which solves both the issues.", "questions": "I like the experimental evaluations and thorough comparison with the prior federated learning works, but I'm concerned about the algorithmic novelty of the proposed approach. Most of the techniques seem to be adapted from the prior known literature on EFT. Can the authors highlight the technical difficulties in directly applying the prior techniques in solving the PCDD problem?", "limitations": "I don't think there are any negative societal impacts of this work.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "+ Extensive experiments comparing the proposed approach, FedGELA, with the existing federated learning approaches.\n+ Detailed theoretical analysis of the proposed approach.\n+ Highlighting the issues of angle collapse and waste-of-space in the federated learning with partially class-disjoint data.", "weaknesses": "- Borrows the existing EFT techniques and hence the novelty seems to be limited.\n- Overall improvement in average accuracy is marginal (~1.5%) over the existing approaches.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Thanks for your positive support and the constructive comments. Regarding the questions and weaknesses mentioned by the reviewer, we provide the point-to-point response as below, and hope that can address your concern.  Let Q, W and A denote the shorthand of Question, Weaknesses and Answer respectively.**\n> **W1 and Q1:** Borrows the existing ETF techniques and hence the novelty seems to be limited. I like the experimental evaluations and thorough comparison with the prior federated learning works, but I'm concerned about the algorithmic novelty of the proposed approach. Most of the techniques seem to be adapted from the prior known literature on ETF. Can the authors highlight the technical difficulties in directly applying the prior techniques in solving the PCDD problem?\n\n**A:** The main problem in addressing the PCDD dilemma with ETF lies in the inability to remedy the loss in the clients. Although ETF can guarantee the global optimum on the server side, from the local view, only a subset of classes are present, and their optimal Bayes classifier is no longer shared with the server, where all classes are considered. Note that, this is the most distinction for PCDD from the ordinary heterogeneity (Non-PCDD) where all classes appear but only differ in the distribution shift.\nHow to deal with this problem is a biggest technical challenge different from directly using ETF. This is the reason why we propose the **Bilateral Curation** in principle. To further show this distinction, we summarize the comparision between our FedGELA and directly using ETF (termed as FedGE for simplicity) in the following table.\n\n|  | GE | LA | CIFAR100 |  |  |  | Fed-ISIC2019 |  |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- |\n|  | #Partition |  | Full Parti. |  | Partial Parti. |  | Real World |  |\n|  | #Metric |  | PA | GA | PA | GA | PA | GA |\n| FedAvg | - | - | 69.09 | 62.80 | 56.46 | 54.28 | 77.27 | 73.59 |\n| FedGE | $\\checkmark$ | - | 71.46 | 66.02 | 62.67 | 58.98 | 69.88 | 75.54 |\n| FedGELA | $\\checkmark$ | $\\checkmark$ | 74.23 | 66.05 | 66.33 | 58.81 | 79.27 | 75.85 |\n|\n\nAs can be seen, FedGELA consistently maintains the advantage in the metric of PA (personalization performance for local clients), while FedGE is even significantly worse than FedAvg on the real-world dataset Fed-ISIC2019 in the metric of PA.  Another noticeable point is that we are the first to employ ETF techniques into FL and before us, the corresponding convergence analysis is unknown. We provide both local and global convergence guarantee for naively combining ETF with FedAvg(FedGE) and our FedGELA.\n\n**We will follow the reviewer's advice to further highlight the technical challenge and add these discussion in the submission.**\n\n> **W2:** Overall improvement in average accuracy is marginal (~1.5%) over the existing approaches.\n\n**A:** 1)We must note that the **reported average improvement of 1.5% encompasses all settings, including both IID and Non-IID scenarios**. Acutally, there is a marginal space for all algorithms to enhance FedAvg in IID situations, while in Non-IID situations, our method has a larger improvement of compared to existing approaches, particularly with a 7% and 11.39% generic improvement over FedAvg on CIFAR10, and a 2.12% and 2.58% generic improvement over the best baseline on CIFAR100. These results demonstrate the effectiveness of FedGELA.\n**To further address the reviewer's concern, we decouple the PCDD setting and the ordinary data heterogeneity (Non-PCDD), and conduct the corresponding experiments on pure PCDD settings.** In the following table, we use PxCy to denote the dataset is divided into x clients with y classes of samples, and in each round, 10 clients are selected into federated training. The training round is 100. According to the experimental results, we can see that our **FedGELA achieves significant improvement especially 18.56% to FedAvg and 10.04% to the best baseline on CIFAR10(P50C2).**\n\n| Dataset(Split) | Metric | FedAvg | Best Baseline | FedGELA |\n| --- | --- | --- | --- | --- |\n| CIFAR10(P10C2) | PA | 92.08 **+3.76** | 94.07 **+1.77** | **95.84** |\n|  | GA | 47.26 **+12.34** | 52.02 **+7.58** | **59.60** |\n| CIFAR10(P50C2) | PA | 91.74 **+3.68** | 93.22 **+2.20** | **95.42** |\n|  | GA | 36.22 **+18.56** | 44.74 **+10.04** | **54.78** |\n| SVHN(P10C2) | PA | 95.64 **+3.11** | 97.02 **+1.73** | **98.75** |\n|  | GA | 69.34 **+14.22** | 76.06 **+7.50** | **83.56** |\n| SVHN(P50C2) | PA | 94.87 **+3.50** | 96.88 **+1.49** | **98.37** |\n|  | GA | 66.94 **+10.24** | 72.97 **+4.21** | **77.18** |\n|\n\n2)Additionally, our algorithm is **easy to reproduce, requiring almost zero burden** in terms of local storage, local computation, and communication costs compared to FedAvg. Unlike other methods, our approach **does not require fine-tuning for personalization**. This makes it more accessible and practical for implementation in real-world scenarios.\n"}, {"review_id": "yv7IEdrftV", "reviewer": "Reviewer_UNoN", "summary": "This paper introduces a novel Federated Learning Algorithm to address the Partially class-disjoint data (PCDD) problem. The approach is based on the simplex equiangular tight frame (ETF) phenomenon to solve the angle collapse issue and introduces a second projection to personalize an adapted structure to save space. The main contributions can be summarized in three aspects: identifying the angle collapse and space waste challenges in the PCDD problem, introducing the novel FedGELA algorithm, and conducting a range of experiments to evaluate its performance. The paper also includes a theoretical analysis with convergence analysis.", "questions": "see weaknesses", "limitations": "see weaknesses", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written with a proper structure and clear explanations. The presentation of the authors' ideas is easy to follow due to the effective use of figures and notations.\n2. The methodology of the FedGELA algorithm is interesting, and the mathematical deductions are sufficient. The algorithm is clear and provides enough information for reproducibility.\n3. The algorithm has been wisely experimented, and the plots are suitable and clear.\n\n", "weaknesses": "1. The authors claim that \"none of the existing methods can intrinsically mitigate PCDD challenges to achieve holistic improvement in the bilateral views of federated learning.\" However, the PCDD problem seems closely related to the general non-iid (non-independent and identically distributed) problem. The main differences between these two problems have not been explained.\n2. Based on my understanding, if PCDD is different from the non-iid problem, it should perhaps be related to the multi-label problem. However, the presentation of the paper, the experimental data, and the methods of experimental comparison all tend to be more inclined towards non-iid problems. Non-iid is a common problem setting, which contradicts the first author's claim of contributions.\n3. The performance improvement is limited.\n4. I disagree with the statement that \"restricting local structure will waste feature space and limit the training of the local model on existing classes.\" I believe the notion of \"waste of space\" is unfounded as it appears to have no impact on computational efficiency or performance improvement.\n\nConclusion:\nThe methodology and algorithm presented in this paper are interesting, and the paper is written in high quality. However, there seems to be an important flaw in the problem setting. PCDD appears not to be a new issue but rather a non-iid problem under some special conditions.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**We really appreciate your positive support and the constructive comments. Regarding the weakness mentioned by the reviewer, we provide the detailed response as below, and hope that can address your concern. Let W denotes the shorthand of Weaknesses.**\n\n**Reply to W1 and W2:**  \n1)We would like to explain a bit about the PCDD problem. **It actually belongs to the data heterogeneity case, but does have a very unique characteristic different from the ordinary heterogeneity problem.** That is, if each client only has a subset of classes, it does not share the optimal Bayes classifier with the global model that considers all classes on the server side. While in the ordinary heterogeneity where local clients have all classes of samples but only differ in the class distributions, they do share an optimal Bayes classifier including the global model on the server side.  \n2)Regarding the claim of the first contribution, the contextual description (Line 63) is for the PCDD problem (Line 64). We did not mean to imply that Non-IID is yet under-explored.   \n**In the revision, we will carefully consider the reviewer's question and add more explanation in terms of the relationship between PCDD and the data heterogeneity (i.e., multiple Non-IID distributions) for clarity.**\nWe will refine this description to avoid misunderstanding.\n\n**Reply to W3:**  \nWe would like to kindly argue about the effectiveness of our method by the following three points.\n\n1)We must note that the reported average improvement of 1.5% to the best baseline encompasses all settings and all datasets, including both Non-IID and IID scenarios. Acutally, there is a marginal space for all algorithms to enhance FedAvg in IID situations, while in Non-IID situations, our method has a larger improvement compared to existing approaches, particularly with a 7% and 11.39% generic improvement over FedAvg on CIFAR10, and a 2.12% and 2.58% generic improvement over the best baseline on CIFAR100. These results demonstrate the effectiveness of FedGELA.  \n\n2)For the consideration of practical data distribution and cohesion with most previous works, we use dirichlet distribution to split the dataset, which will generate heterogeneity data more than PCDD and might limit the potential better improvement. **To further address the reviewer's concern, we decouple the PCDD setting and the ordinary data heterogeneity (Non-PCDD), and conduct the corresponding experiments on pure PCDD settings.** In the following table, we use PxCy to denote the dataset is divided in to x clients with y classes of samples, and in each round, 10 clients are selected into federated training. The training round is 100. According to the experimental results, we can see that our **FedGELA achieves significant improvement especially 18.56% to FedAvg and 10.04% to the best baseline on CIFAR10(P50C2)**.\n\n| Dataset(Split) | Metric | FedAvg | Best Baseline | FedGELA |\n| --- | --- | --- | --- | --- |\n| CIFAR10(P10C2) | PA | 92.08 **+3.76** | 94.07 **+1.77** | **95.84** |\n|  | GA | 47.26 **+12.34** | 52.02 **+7.58** | **59.60** |\n| CIFAR10(P50C2) | PA | 91.74 **+3.68** | 93.22 **+2.20** | **95.42** |\n|  | GA | 36.22 **+18.56** | 44.74 **+10.04** | **54.78** |\n| SVHN(P10C2) | PA | 95.64 **+3.11** | 97.02 **+1.73** | **98.75** |\n|  | GA | 69.34 **+14.22** | 76.06 **+7.50** | **83.56** |\n| SVHN(P50C2) | PA | 94.87 **+3.50** | 96.88 **+1.49** | **98.37** |\n|  | GA | 66.94 **+10.24** | 72.97 **+4.21** | **77.18** |\n|\n\n3)Additionally, our algorithm is **easy to reproduce, requiring almost zero burden** in terms of local storage, local computation, and communication costs compared to FedAvg. Unlike other methods, our approach **does not require fine-tuning for personalization**. This makes it more accessible and practical for implementation in real-world scenarios.\n\n**Reply to W4:**  \nProbably, it is because our words \"restricting local structure\" that confuses the reviewer. Actually this sentence discusses that if we align the global classifier structure as the local classifier structure, it does greatly limit the performance of personalization, since under PCDD, the global model (with the support of full classes) and the local models (with the support of a subset of classes) do not intrinsically share an optimal Bayes classifier (please refer to the explanation for W1 & W2). Recent work FedRod [1] that configures different classifiers on the server and client sides also supports this point.  \nRegarding \"waste of space\", it is a further possible explanation about why aligning the structures under PCDD induces the performance degeneration in terms of the PA metric (for personalization). An intuitive illustration is shown in Figure 2. Empirically as shown in Table 4, aligning the local classifier structure as the global structure, namely ETF (we denote this method FedGE) can not achieve best personal performance. Especially on the real-world PCDD dataset Fed-ISIC2019, the restricting greatly limits the personal performance. Here we show part of results of Table 4:\n\n|  | GE | LA | CIFAR100 |  |  |  | Fed-ISIC2019 |  |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- |\n|  | #Partition |  | Full Parti. |  | Partial Parti. |  | Real World |  |\n|  | #Metric |  | PA | GA | PA | GA | PA | GA |\n| FedAvg | - | - | 69.09 | 62.80 | 56.46 | 54.28 | 77.27 | 73.59 |\n| FedGE | $\\checkmark$ | - | 71.46 | 66.02 | 62.67 | 58.98 | 69.88 | 75.54 |\n| FedGELA | $\\checkmark$ | $\\checkmark$ | 74.23 | 66.05 | 66.33 | 58.81 | 79.27 | 75.85 |\n|\n\n**We will refine this sentence to clarify our meaning and provide the detailed explanations about our statement.**  \n[1]On bridging generic and personalized federated learning for image classification\n\n\n"}, {"review_id": "E4E3nhZoze", "reviewer": "Reviewer_SUwJ", "summary": "This paper mainly focuses on the partially class-disjoint data (PCDD) problem in federated learning (FL) settings, which is a common yet challenging problem in distributed data sources. Inspired by a classifier structure (simplex equiangular tight frame, ETF), the authors of the paper propose FedGELA to tackle the PCDD problem. FedGELA is a variant of FedAvg with local model adaptation (personalization): They first define the classifier $W$, which is the ETF that the classifier should converge to. Here, they also take the client local data distribution ($\\phi$) into account. Afterwards, the feature extractor $H$ will be optimized locally at each client and communicated via server-client communication. Finally, the global feature extractor and the $W$ at central server, as well as the local feature extractors and the adapted $W$ at clients will be returned.", "questions": "1.\tYou claim that FedGELA could mitigate PCDD from both Global and Local view. But in Algorithm 1, the global model $H^T$ is simply an average of the client model and there is no server-side optimization. Could you please explain this?\n2.\tIn Table 1, you claim that FedGELA could mitigate the model skew, could you please explain this point in more details? Also, in terms of “save space”, since you are transmitting the whole feature backbone $H$, which is the resnet18 in your experiments, what is the reason of space saving?\n3.\tWhy do you use $H$ to represents 2 different terms, features (Line 114) and global backbone (in the Algorithm)? It’s a bit confusing during reading.\n4.\tCould you shortly explain the converged angles in your figures? Is there a specific meaning of the values? E.g. in Figure 2. \n5.\tIn Equation 4, you model the client data skew in the label space via $\\phi$, which is based on the number of samples from different classes. Have you experimented with other options? This looks a bit too straightforward to me.\n6.\tFedGen[1] is a method which augment the feature embedding space using a shared feature generator, which could possibly mitigate the issue of “waste of space” by generating synthetic embeddings from minority classes. Could you provide a comparison with this work?\n[1] Zhu, Zhuangdi, Junyuan Hong, and Jiayu Zhou. \"Data-free knowledge distillation for heterogeneous federated learning.\" ICML, 2021.\n[2] Tan, Yue, et al. \"Fedproto: Federated prototype learning across heterogeneous clients.\" AAAI 2022.\nMinors:\n1.\tLine 115, $E_W$ and $E_H$ should be introduced at their first appearance.\n2.\tLine 298, the selection of $E_W$ seems to be dataset specific, is there any default values suggestions?\n\n", "limitations": "FedGELA is only tested in the client data with label skew. Is it also applicable to the data with feature skew?", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The proposed method is motivated very well. The schematic illustration is also clear. The theoretical analysis is sound. Experiments and the results are good.", "weaknesses": "From my understanding, FedGELA focuses only on the alignment in the feature embedding space, which has been done by many previous works (FedGen [1], FedProto [2], …). Therefore this paper lacks significance to some extent. Also, the definition of $W$ (ETF, the global classifier), as well as the locally adapted ones looks straightforward.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**We really appreciate your constructive comments. Regarding the questions from the reviewer, we provide detailed response as below, and hope that can address your concern. Let's use Q as a shorthand for Question.**\n\n**Weakness and Q6:**\n\n1)**Technical Innovation.** We would like to kindly argue that our core contributions lie in identifying the practical yet under-explored PCDD problem and proposing a **bilateral curation** method in principle to combat the challenges. Why we need to **curate the local classifier instead of keeping all classifiers same is not studied under PCDD**, and we give the contraction analysis in Eq. (2) and (3). Besides, such a non-parameteric bilateral curation is not straightforward, as the form in the **local curation is not given in previous study and Eq. (4) must insure that global and local convergence hold simultaneously**(See the reply of Q5). Other forms might not enjoy such a theoretical merit.\n\n2)**Comparison to FedGen.** **FedGELA focuses on bilateral curation in the parameter space, instead of embedding space.** Although parameter space and embedding space might follow the same spirit, in the embedding space, we need to upload prototypes like FedProto or train generator and generate features in each round like FedGen. But our bilateral ETFs reach consensus in advance and can be guaranteed in principle without the transmission of personal classifiers and burden on local training as shown in Algorithm 1 in Page 5. Note that, the calculation of $\\Phi$ is negligible. Empirically, FedProto is included in the paper and here we provide the comparision to FedGen on SVHN in the following table. **We will add FedGen as baseline into the submission**.\n\n||Partition|Metric|FedAvg|FedGen|FedGELA|\n|---|---|---|---|---|---|\n| Full Parti.(10 clients) |IID|PA|93.01|94.02|94.84|\n|||GA|92.61|93.99|94.66|\n||$\\beta=0.5$|PA|93.95|94.47|96.27|\n|||GA|91.24|92.66|93.66|\n||$\\beta=0.1$|PA|98.10|98.22|98.52|\n|||GA|75.24|76.51|78.88|\n|Partial Parti.(50 clients)|IID|PA|91.44|91.47|94.68|\n|||GA|91.29|91.33|93.59|\n||$\\beta=0.5$|PA|92.70|93.67|95.54|\n|||GA|89.29|91.35|93.29|\n||$\\beta=0.2$|PA|95.31|95.77|96.85|\n|||GA|84.70|87.59|89.58|\n|\n\n**Q1:**\nWe kindly point out that the global and local views mean FedGELA can improve the performance of both the global model and the local models. It does not suggest that FedGELA has both client-side and server-side optimization.\n\n**Q2:**\nAs described in the caption of Table 1, 1) model skew is the bias of local model and global model to their optimal weights. We mitigate this by the curation on the classification heads; 2) \"save space\" here means saving locally wasted space instead of the storage size of the model. We will emphasize these points in the caption to avoid misunderstanding.\n\n**Q3:**\n$H$ follows the notation in Layer Peeled Model[1] to better retrospect previous theory about ETF. We specially correct this notation for the global backbone in Line 122. We apologize that we haven't highlighted this point and will improve it to avoid the confusion.\n\n**Q4:**\nIn Figure 2, the angle converges to a proper value, meaning that the optimal separation structure is approaching. The value of the converged angle reflects the mean seperation between different classes when the algorithm converges. \n\n**Q5:** The selection of $\\Phi_k$ should reflect personal class distribution and satisfy a basic rule for federated learning, wherein the aggregation of local classifiers aligns with the global classifier(line 168 in the paper), thereby ensuring the validity of theoretical analyses from both global and local views. The convergence requirement can be rewrite as $\\gamma\\sum_k p_kQ_k(\\frac{n_{k,c}}{n_k})=1$, where $\\gamma$ is the scaling constant and $Q_k(\\frac{n_{k,c}}{n_k})$ denotes the potential way to select $\\Phi$. It is highly preferable for the selection process to avoid transmitting $Q_k(\\frac{n_{k,c}}{n_k})$, which might induce additional privacy risks. There are indeed many ways to select $\\Phi_k$. **However, setting $Q_k(\\frac{n_{k,c}}{n_k})=\\frac{n_{k,c}}{n_k}$  and $\\gamma=\\frac{1}{C}$ is the only potential way we can find to determine $\\gamma$ that both satisfy the convergence and privacy requirement.** Besides, we have also considered employing alternative methods like employing an exponential or power function of the number of samples, which need to share  $Q_k(\\frac{n_{k,c}}{n_k})$  but achieve similar performance. The related experiments are shown in the following table. **We will disscuss more and highlight our design intuition in the submission.**\n\n|Partition|Metric|$Q_k(x)=e^{x}$|$Q_k(x)=x^{\\frac{1}{2}}$|$Q_k(x)=x$(ours)|\n|---|---|---|---|---|\n|IID|PA|95.12|95.43|94.84|\n||GA|94.32|93.99|94.66|\n|$\\beta=0.5$|PA|96.18|95.56|96.27|\n||GA|93.28|93.22|93.66|\n|$\\beta=0.1$|PA|98.33|98.21|98.52|\n||GA|78.95|77.18|78.88|\n|\n\n**Q7:**\nWe'll explain notations when they first appear.\n\n**Q8:**\nWe recommand the default value to 10e3. With larger $E_w$ (even 10e7), FedGELA can also perform better than most of the methods and far better than FedAvg.\n\n**Limitaion:**\nWe conduct the additional experiments on the PACS dataset[2], which is commonly used for analyzing feature heterogeneity. FedISIC-2019 used in our paper also includes feature shifts as the images are collected by different hospitals in different situations. As shown below, FedGELA remains applicable and achieves commendable performance even under feature heterogeneity. Our speculation is that, the local classifiers trained on distinct feature domains may exhibit bias. Using the optimal separation structure like FedGELA aids in enhancing performance.\n\n|Dataset|split|FedAvg|Best Baseline|FedGELA|\n|---|---|---|---|---|\n|PACS|PA|97.60|98.99|98.65|\n||GA|82.30|84.42|85.06|\n|Fed-ISIC2019|PA|77.27|78.91|79.27|\n||GA|73.59|74.98|75.85|\n|\n\n[1]Exploring deep neural networks via layer-peeled model: Minority collapse in imbalanced training.(NeurIPS21)\n\n[2]Deeper, broader and artier domain generalization.(ICCV17)"}, {"review_id": "yh2xVdXXgc", "reviewer": "Reviewer_J5UX", "summary": "This paper addresses a challenge in Federated Learning referred to as partially class-disjoint data (PCDD), where each client contributes a part of classes (instead of all classes) of samples. Without full classes, the local objective will contradict the global objective, yielding the angle collapse problem for locally missing classes and the space waste problem for locally existing classes. This is a real-world challenge since it is not uncommon for example that some classes will be well sampled in certain regions, but not others. Prior art mainly focus on the general heterogeneity without specially considering partially class disjoint challenges. Without full classes, the local objective will contradict the global objective, yielding the angle collapse for locally missing classes and the waste of space for locally existing classes. The goal is to achieve holistic improvement in the bilateral views (both global view and local view) of federated learning. \n\nThe authors propose FedGELA where the classifier is globally fixed as a simplex ETF while locally adapted to the personal distributions. Globally, FedGELA provides fair and equal discrimination for all classes and avoids inaccurate updates of the classifier, while locally it utilizes the space of locally missing classes for locally existing classes. \n\nThe proposed approach builds upon simplex equiangular tight frame (ETF), which provides each class the same classification angle and generalizes well on imbalanced data. Specifically, in their FedGELA approach, the classifier is globally fixed as a simplex ETF while locally adapted based on the local distribution matrix to utilize the wasted space for the existing classes. In the global view, FedGELA merges class features and their corresponding classifier vectors, which converge to ETF. In the local view. it provides existing major classes with larger feature spaces and encourages to utilize the spaces wasted by locally missing classes. \n\nContributions are summarized as :\n- Study algorithmic implication of a real-world  challenge (partially class-disjoint data (PCDD), namely angle collapse and space waste \n- Propose FedGELA and theoretically show the local and global convergence analysis for PCDD with the experimental verification \n- Evaluate on multiple benchmark datasets under the PCDD case and a real-world dataset to demonstrate the bilateral advantages of FedGELA over the state of the art methods.\n\n", "questions": "1. Given the performance improvements were in the <3% against the best baselines - can you provide more information? For example does it perform better worse under certain conditions? if so, pls distinguish, and if possible explain what this may imply about either the algorithm and limitations, or perhaps a limitation in the benchmark wrt fully characterizing the real world challenge?", "limitations": "No negative societal effects. Limitations explained in Weaknesses section.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Related work well covers comparison among a range of FL methods and why PCDD not covered. Examples of why prior art does not address PCDD include: generic federated leaning adopt a uniform treatment of all classes, then attempt mitigate personal differences; personalized federated learning places less emphasis on locally missing classes and selectively shares parameters/prototypes to minimize the impact of personal characteristics. While these methods might directly or indirectly help mitigate the data shifts caused by PCDD, neither achieve holistic improvement for global and local views\n2. Performance evaluaton on 3 relevant datasets (SVHN, CIFAR10, CIFAR100), against all the top state of the art algorithms as baselines, and showng it outperforms. FedGELA consistently exceeds all baselines.\n\n", "weaknesses": "Overall, the paper represents is a solid contribution - well defined problem not addressed by prior art and representative of real-world problem for fedrated learing. Solid treatment of prior art, and differentiation from prior methods. \nWeaknesses:\n1. Only 1 real PCDD federated application Fed-ISIC2019 was evaluated - however I am not aware of other benchmarks I would recommend.\n2. Performance improvements against to the best baseline for all tests were all <3% performance improvement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\n**We really appreciate your positive support and the constructive comments. In the following, we provide the detailed response and hope that can address your concerns. Let W, Q and A denote the shorthand of Weaknesses, Question and Answer respectively.** \n\n> **W1:** Only 1 real PCDD federated application Fed-ISIC2019 was evaluated - however I am not aware of other benchmarks I would recommend.\n\n**A:** To address the reviewer's concern, we **test FedGELA on FEMNIST[1] and SHAKESPEARE[2] (two datasets also satisfy the PCDD setting)** compared with all related approches in the paper. FEMNIST includes complex 62-class handwriting images from 3500 clients and SHAKESPEARE is a next word prediction task with 1129 clients. Most of the clients only have a subset of class samples. With help of LEAF[3], we choose 50 clients of each dataset into federation and in each round we randomly select 10 clients into training. The total round is set to 20 and the model structure is a simple CNN and 2 layer of LSTM for FEMNIST and SHAKESPEARE, respectively. It can be seen **in the following table**, **our method achieves best results of both personal and generic performance on all three real-world challenges**. \n\n| Dataset | split | FedAvg | Best Baseline | FedGELA |\n| --- | --- | --- | --- | --- |\n| SHAKESPEARE | PA | 49.56 **+4.07**  | 51.66 **+1.97** | **53.63** |\n|  | GA | 44.53 **+3.86** | 47.29 **+1.10** | **48.39** |\n| FEMNIST | PA | 67.02 **+4.82** | 69.54 **+2.3** | **71.84** |\n|  | GA | 59.54 **+2.54** | 61.22 **+0.86** | **62.08** |\n| FedISIC-2019 | PA | 77.27 **+2.00** | 78.91 **+0.36** | **79.27** |\n|  | GA | 73.59 **+2.26** | 74.98 **+0.96** | **75.85** |\n|\n\n[1]EMNIST: Extending MNIST to handwritten letters  \n[2]William Shakespeare: the complete works  \n[3]Leaf: A benchmark for federated settings  \n\n> **W2 and Q1:** Performance improvements against to the best baseline for all tests were all <3% performance improvement. Given the performance improvements were in the <3% against the best baselines - can you provide more information? For example does it perform better worse under certain conditions? if so, pls distinguish, and if possible explain what this may imply about either the algorithm and limitations, or perhaps a limitation in the benchmark wrt fully characterizing the real world challenge?\n\n**A:** Thank you for the suggestion. We explain this in the following two points.\n\n1)**Significant Improvement on Pure PCDD settings.** For the consideration of practical data distribution and cohesion with most previous works, we use dirichlet distribution to split the dataset, which will generate heterogeneity data more than PCDD and might limit the potential better improvement. To further address the reviewer's concern, **we decouple the PCDD setting and the ordinary heterogeneity(Non-PCDD), and conduct the corresponding experiments on pure PCDD situations.** In the following table, we use PxCy to denote the dataset is divided in to x clients with y classes of samples, and in each round, 10 clients are selected into federated training. The training round is 100. According to the experimental results, we can see that our FedGELA achieves **significant improvement especially 18.56% to FedAvg and 10.04% to the best baseline** on CIFAR10(P50C2).\n\n| Dataset(Split) | Metric | FedAvg | Best Baseline | FedGELA |\n| --- | --- | --- | --- | --- |\n| CIFAR10(P10C2) | PA | 92.08 **+3.76** | 94.07 **+1.77** | **95.84** |\n|  | GA | 47.26 **+12.34** | 52.02 **+7.58** | **59.60** |\n| CIFAR10(P50C2) | PA | 91.74 **+3.68** | 93.22 **+2.20** | **95.42** |\n|  | GA | 36.22 **+18.56** | 44.74 **+10.04** | **54.78** |\n| SVHN(P10C2) | PA | 95.64 **+3.11** | 97.02 **+1.73** | **98.75** |\n|  | GA | 69.34 **+14.22** | 76.06 **+7.5** | **83.56** |\n| SVHN(P50C2) | PA | 94.87 **+3.5** | 96.88 **+1.49** | **98.37** |\n|  | GA | 66.94 **+10.24** | 72.97 **+4.21** | **77.18** |\n|\n\n2)Regarding the potential limitations, we can think that curating the structure of the last layer really builds on top of the power deep neural networks, whose fitting ability is sufficiently powerful. Thus regularzing structure of the classification layer does not hurt the overall model capacity too much, and promote the training calibration under PCDD. But when it comes to the shalow models, regularizing the structure of the classification layer might be too harsh and might hurt the training. Another point that is worth exploring in the future is that in the self-supervised learning, it might not contain the label information avaliable and how to curation the model under the PCDD scenario to promote the training remains unknown.  \n\n**We appreciate the reviewer's advice and will include the discussion of this question into the submission for better clarity and future explorations.**\n"}]

Author rebuttal:
We would like to thank all the reviewers(nvCV, UNoN, SUwJ and J5UX) for their thoughtful suggestions on our paper, and appreciate that the reviewers have multiple positive impressions of our work, including: 
- **well defined problem (J5UX) and a clear motivation (SUwJ and J5UX)**
- **a novel and interesting algorithm (UNoN)**
- **solid theoretical justification (nvCV and UNoN) and sufficient mathematical deductions (SUwJ)**
- **extensive and reasonable experiments(nvCV, UNoN and J5UX) with good results (SUwJ)**
- **well-written paper with a proper structure (UNoN) and clear illustration (SUwJ).**

We provide a summary of our responses, and we will add all corresponding discussions, reviewer-recommended related work and experimental results into the manuscript. For detailed responses, please refer to the feedback of each comment/question point-by-point.

**Introduction and Related Works:**

- We clarify some statements like "global and local view", "model skew" and "save space".(for the question of the Reviewer J5UX) 
- We explain the relationship between PCDD and Non-IID and the difference between PCDD and traditional label heterogeneity in federated learning.(for the question of the Reviewer UNoN) 
- We further analyze the statements "waste of space" and "converged angles". (for the question of the Reviewers UNoN and SUwJ respectively) 

**Method:**

- We highlight and clarify the technique difficulties in applying simplex ETF when solving PCDD challenges. (for the question of the Reviewers nvCV)
- We further analyze the chosen of the personal distribution matrix. (for the question of the Reviewer SUwJ) 
- We highlight our technical innovation and provide the comparison to FedGen. (for the question of the Reviewer SUwJ) 
- We add more descriptions about some notations. (for the question of the Reviewer SUwJ) 

**Experiments:**

- We add FedGEN into baselines. (for the question of the Reviewer SUwJ)
- We decouple the PCDD settings and the ordinary (Non-PCDD) data heterogeneity and verify our algorithm at pure PCDD settings to further show the effectiveness. (for the question of the Reviewers nvCV, UNoN and J5UX)
- We verify our algorithm on two more real-world PCDD federated challenges. (for the question of the Reviewer J5UX)
- We provide suggestions about choosing the hyper-parameter. (for the question of the Reviewer SUwJ)
- We test our method under feature heterogeneity. (for the question of the Reviewer SUwJ)

**We appreciate all reviewers’ time and effort again. We will add all corresponding discussions, reviewer-recommended related work and experimental results into the manuscript. We are looking forward to your reply!**



Meta-review: This paper tackles federated learning over class-disjoint data and proposes a technique based on equiangular tight frame to increase performance on a number of benchmarks. The authors analyze the downsides of existing approaches wrt locally disjoint classes and show that the proposed method improves both. Reviewers found the problem interesting and the method to be well-defined and effective, though some had concerns about the novelty of the method vs. novelty of the application and the size of the performance improvement. While I think this paper is borderline, I lean towards acceptance.
Decision: Accept (poster)