Title: Federated Learning with Bilateral Curation for Partially Class-Disjoint Data
Venue: NeurIPS 2023
Accepted: yes
Keywords: federated learning

data heterogeneity

partially class-disjoint data
Abstract: Partially class-disjoint data (PCDD), a common yet under-explored data formation where each client contributes a part of classes (instead of all classes) of samples, severely challenges the performance of federated algorithms. Without full classes, the local objective will contradict the global objective, yielding the angle collapse problem for locally missing classes and the space waste problem for locally existing classes. As far as we know, none of the existing methods can intrinsically mitigate PCDD challenges to achieve holistic improvement in the bilateral views (both global view and local view) of federated learning. To address this dilemma, we are inspired by the strong generalization of simplex Equiangular Tight Frame (ETF) on the imbalanced data, and propose a novel approach called FedGELA where the classifier is globally fixed as a simplex ETF while locally adapted to the personal distributions. Globally, FedGELA provides fair and equal discrimination for all classes and avoids inaccurate updates of the classifier, while locally it utilizes the space of locally missing classes for locally existing classes. We conduct extensive experiments on a range of datasets to demonstrate that our FedGELA achieves promising performance (averaged improvement of 3.9% to FedAvg and 1.5% to best baselines) and provide both local and global convergence guarantees.

Federated Learning with Bilateral Curation for Partially Class-Disjoint Data

Abstract

Partially class-disjoint data (PCDD), a common yet under-explored data formation where each client contributes a part of classes (instead of all classes) of samples, severely challenges the performance of federated algorithms. Without full classes, the local objective will contradict the global objective, yielding the angle collapse problem for locally missing classes and the space waste problem for locally existing classes. As far as we know, none of the existing methods can intrinsically mitigate PCDD challenges to achieve holistic improvement in the bilateral views (both global view and local view) of federated learning. To address this dilemma, we are inspired by the strong generalization of simplex Equiangular Tight Frame (ETF) on the imbalanced data, and propose a novel approach called FedGELA where the classifier is globally fixed as a simplex ETF while locally adapted to the personal distributions. Globally, FedGELA provides fair and equal discrimination for all classes and avoids inaccurate updates of the classifier, while locally it utilizes the space of locally missing classes for locally existing classes. We conduct extensive experiments on a range of datasets to demonstrate that our FedGELA achieves promising performance (averaged improvement of 3.9% to FedAvg and 1.5% to best baselines) and provide both local and global convergence guarantees. Source code is available at: https://github.com/MediaBrain-SJTU/FedGELA.

1 Introduction

Partially class-disjoint data (PCDD) [13, 18, 21] refers to an emerging situation in federated learning [14, 22, 43, 46, 50] where each client only possesses information on a subset of categories, but all clients in the federation provide the information on the whole categories. For instance, in landmark detection [39] for thousands of categories with data locally preserved, most contributors only have a subset of categories of landmark photos where they live or traveled before; and in the diagnosis of Thyroid diseases, due to regional diversity different hospitals may have shared and distinct Thyroid diseases [10]. It is usually difficult for each party to acquire the full classes of samples, as the participants may be lack of domain expertise or limited by demographic discrepancy. Therefore, how to efficiently handle the partially class-disjoint data is a critical (yet under-explored) problem in real-world federated learning applications for the pursuit of personal and generic interests.

Prevalent studies mainly focus on the general heterogeneity without specially considering the PCDD challenges: generic federated leaning (G-FL) algorithms adopt a uniform treatment of all classes and mitigate personal differences by imposing constraints on local training [17, 19], modifying logits [21, 47] adjusting the weights of submitted gradients [37] or generating synthetic data [54]; in contrast, personalized federated learning (P-FL) algorithms place relatively less emphasis on locally missing classes and selectively share either partial network parameters [1, 6] or class prototypes [33] to minimize the impact of personal characteristics, thereby separating the two topics. Those methods wangyanfeng}@sjtu.edu.cn

Expand

Shrink

Expand

Angle

Collapse

Shrink

Waste of Space

of Missing Classe

s

Feature Expand

Feature Shrink

(a) Ideal Structure

(b) Collapsed Structure A

(c) Collapsed Structure B

(d) Restricted Structure

might directly or indirectly help mitigate the data shifts caused by PCDD, however, as far as we know, none of the existing works can mitigate the PCDD challenges to achieve holistic improvement in the bilateral views (global and local views) of federated learning. Please refer to Table 1 for a comprehensive comparison among a range of FL methods from different aspects.

Without full classes, the local objective will contradict the global objective, yielding the angle collapse for locally missing classes and the waste of space for locally existing classes. Ideally, as shown in Figure 1(a), global features and their corresponding classifier vectors shall maintain a proper structure to pursue the best separation of all classes. However, the angles of locally missing classes' classifier vectors will collapse, when trained on each client with partially class-disjoint data, as depicted in Figure 1(b), 1(c). FedRS [21] notices the degenerated updates of the classifier and pursues the same symmetrical structure in the local by restricting logits of missing classes. Other traditional G-FL algorithms indirectly restrict the classifier by constraining logits, features, or model weights, which may also make effects on PCDD. However, they cause another problem: space waste for personal tasks. As shown in Figure 1(d), restricting local structure will waste feature space and limit the training of the local model on existing classes. P-FL algorithms utilize the wasted space by selectively sharing part of models but exacerbate the angle collapse of classifier vectors. Recent FedRod [3] attempts to bridge the gap between P-FL and G-FL by introducing a two-head framework with logit adjustment in the G-head, but still cannot address the angle collapse caused by PCDD.

To tackle the PCDD dilemma from both P-FL and G-FL perspectives, we are inspired by a promising classifier structure, namely simplex equiangular tight frame (ETF) [9, 26, 41], which provides each class the same classification angle and generalizes well on imbalanced data. Motivated by its merits, we propose a novel approach, called FedGELA , in which the classifier is Globally fixed as a simplex ETF while Locally Adapted to personal tasks. In the global view, FedGELA merges class features and their corresponding classifier vectors, which converge to ETF. In the local view, it provides existing major classes with larger feature spaces and encourages to utilize the spaces wasted by locally missing classes. With such a bilateral curation, we can explicitly alleviate the impact caused by PCDD. In a nutshell, our contributions can be summarized as the following three points:

We study a practical yet under-explored data formation in real-world applications of federated learning, termed as partially class-disjoint data (PCDD), and identify the angle collapse and space waste challenges that cannot be efficiently solved by existing prevalent methods (Sec. 3.2).

We propose a novel method called FedGELA that classifier is globally fixed as a symmetrical structure ETF while locally adapted by personal distribution (Sec. 3.3), and theoretically show the local and global convergence analysis for PCDD with the experimental verification (Sec. 4.2).

We conduct a range of experiments on multiple benchmark datasets under the PCDD case and a real-world dataset to demonstrate the bilateral advantages of FedGELA over the state-of-the-art methods from multiple views like the larger scale of clients and straggler situations (Sec. 5.2). We also provide further analysis like classification angles during training and ablation study. (Sec. 5.3).

2 Related Work

2.1 Partially Class-Disjoint Data and Federated Learning algorithms

Partially class-disjoint data is one common formation among clients that can significantly impede the convergence, performance, and efficiency of algorithms in FL [18]. It belongs to the data heterogeneity

case, but does have a very unique characteristic different from the ordinary heterogeneity problem. That is, if only each client only has a subset of classes, it does not share the optimal Bayes classifier with the global model that considers all classes on the server side. Recently, FedRS [21] has recognized the PCDD dilemma and directly mitigate the angle collapse issue by constraining the logits of missing classes. FedProx [19] also can lessen the collapse by constraining local model weights to stay close to the global model. Other G-FL algorithms try to address data heterogeneity from a distinct perspective. MOON [17] and FedGen [54] utilizes contrastive learning and generative learning to restrict local representations. And FedLC [47] introduces logit calibration to adjust the logits of the local model to match those of the global model, which might indirectly alleviate the angle collapse in the local. However, they all try to restrict local structure as global, resulting in the waste space for personal tasks shown in Figure 1(d). P-FL algorithms try to utilize the wasted space by encouraging the angle collapse of the local classifier. FedRep [6] only shares feature extractors among clients and FedProto [33] only submits class prototypes to save communication costs and align the feature spaces. In FedBABU [25], the classifier is randomly initialized and fixed during federated training while fine-tuned for personalization during the evaluation. However, they all sacrifice the generic performance on all classes. FedRod [3] attempts to bridge this gap by introducing a framework with two heads and employing logit adjustment in the global head to estimate generic distribution but cannot address angle collapse. In Table 1, we categorize these methods by targets (P-FL or G-FL), skews (feature, logit, or model weight), and whether they directly mitigate the angle collapse of local classifier or saving personal spaces for personal spaces. It is evident that none of these methods, except ours, can handle the PCDD problem in both P-FL and G-FL. Furthermore, FedGELA is the only method that can directly achieve improvements from all views.

2.2 Simplex Equiangular Tight Frame

The simplex equiangular tight frame (ETF) is a phenomenon observed in neural collapse [26], which occurs in the terminal phase of a well-trained model on a balanced dataset. It is shown that the last-layer features of the model converge to within-class means, and all within-class means and their corresponding classifier vectors converge to a symmetrical structure. To analyze this phenomenon, some studies simplify deep neural networks as last-layer features and classifiers with proper constraints (layer-peeled model) [9, 12, 40, 53] and prove that ETF emerges under the crossentropy loss. However, when the dataset is imbalanced, the symmetrical structure of ETF will collapse [9]. Some studies try to obtain the symmetrical feature and the classifier structure on the imbalanced datasets by fixing the classifier as ETF [41, 53]. Inspired by this, we propose a novel method called FedGELA that bilaterally curates the classifier to leverage ETF or its variants. See Appendix A for more details about ETF.

3 Method

3.1 Preliminaries

ETF under LPM. A typical L-layer DNN parameterized by W can be divided into the feature backbone parameterized by W -L and the classifier parameterized by W L . From the view of layerpeeled model (LPM) [9, 12, 40, 53], training W with constraints on the weights can be considered

as training the C-class classifier W L = { W L 1 , ..., W L C } and features H = { h 1 , ..., h n } of all n samples output by last layer of the backbone with constraints E W and E H on them respectively. On the balanced data, any solutions to this model form a simplex equiangular tight frame (ETF) that all last layer features h i, ∗ c and corresponding classifier W L, ∗ c of all classes converge as:

where m ∗ c forms the ETF defined as M = √ C C -1 U ( I C -1 C 1 C 1 T C ) . Here M = [ m ∗ 1 , · · · , m ∗ C ] ∈ R d × C , U ∈ R d × C allows a rotation and satisfies U T U = I C and 1 C is an all-ones vector. ETF is an optimal classifier and feature structure in the balanced case of LPM.

FedAvg. On the view of LPM, given N clients and each with n k samples, the vanilla federated learning via FedAvg consists of four steps [22]: 1) In round t , the server broadcasts the global model W t = { H t , W L,t } to clients that participate in the training (Note that here H is actually the global backbone W -L,t instead of real features); 2) Each local client receives the model and trains it on the personal dataset. After E epochs, we acquire a new local model W t k ; 3) The updated models are collected to the server as { W t 1 , W t 2 , . . . , W t N } ; 4) The server averages local models to acquire a new global model as W t +1 = ∑ N k =1 p k W t k , where p k = n k / ∑ N k ′ =1 n k ′ . When the pre-defined maximal round T reaches, we will have the final optimized global model W T .

3.2 Contradiction and Motivation

Contradiction. In G-FL, the ideal global objective under LPM of federated learning is described as:

Assuming global distribution is balanced among classes, no matter whether local datasets have full or partial classes, the global objective with constraints on weights can be simplified as:

Similarly, the local objective of k-th client with a set of classes C k can be described as:

̸

When PCDD exists ( C k = C ), we can see the contradiction between local and global objectives, which respectively forms two structures, shown in Figure 3(a) and Figure 3(b). After aggregated in server or constrained by some FL methods, the structure in the local is restricted to meet the global structure, causing space waste for personal tasks shown in Figure 1(d).

Motivation. To verify the contradiction and related feature and classifier structures, we split CIFAR10 into 10 clients and perform FedAvg on it with Dirichlet Distribution (Dir ( β = 0 . 1 )). As illustrated in Figure 2, the angle difference between existing classes and between missing classes becomes smaller and converges to a similar value in the global model. However, in the local training, angles between existing classes become larger while angles between missing classes become smaller, which indicates the contradiction. With this observation, to bridge the gap between Eq (3) and Eq (2) under PCDD, we need to construct the symmetrical and uniform classifier angles for all classes while encouraging local clients to expand existing classes' feature space. Therefore, we propose our method FedGELA that classifier can be Globally fixed as ETF but Locally Adapted based on the local distribution matrix to utilize the wasted space for the existing classes.

0

5

10

Rounds

70

80

90

100

Angle

Missing Angle in Global

Existing Angle in Global

Missing Angle in Local

Existing Angle in Local

3.3 FedGELA

Global ETF. Given the global aim of achieving an unbiased classifier that treats all classes equally and provides them with the same discrimination and classifier angles, we curate the global model's classifier as a randomly initialized simplex ETF with scaling √ E w at the start of federated training:

Then the ETF is distributed to all clients to replace their local classifiers. In Theorem 1, we prove in federated training under some basic assumptions, by fixing the classifier as a randomly simplex ETF with scaling √ E W and constraints E H on the last layer features, features output by last layer of backbone and their within class means will converge to the ETF similar to Eq (1), which meets the requirement of global tasks.

Local Adaptation. However, when PCDD exists in the local clients, naively combining ETF with FL does not meet the requirement of P-FL as analyzed in Eq (2) and Eq (3). To utilize the wasted space for locally missing classes, in the training stage, we curate the length of ETF received from the server based on the local distribution as below:

where Φ k is the distribution matrix of k-th client. Regarding the selection of Φ k , it should satisfy a basic rule for federated learning, wherein the aggregation of local classifiers aligns with the global classifier, thereby ensuring the validity of theoretical analyses from both global and local perspectives. Moreover, it is highly preferable for the selection process to avoid introducing any additional privacy leakage risks. To meet the requirement that averaged classifier should be standard ETF: W L = ∑ N k =1 p k W L k in the globally balanced case, its row vectors are all one's vector multiple statistical values of personal distribution: ( Φ T k ) c = n k,c n k γ 1 ( γ is a constant, and n k,c and n k are the c-th class sample number and total sample number of the k-th client) respectively. We set γ to 1 | C | . Finally, the local objective from Eq. (3) is adapted as:

Total Framework. After introducing two key parts of FedGELA (Global ETF and Local Adaptation), we describe the total framework of FedGELA. As illustrated and highlighted in Algorithm 1 (refer to Appendix D for the workflow figure), at the initializing stage, the server randomly generates an ETF as the global classifier and sends it to all clients while local clients adjust it based on the personal distribution matrix as Eq (4). At the training stage, local clients receive global backbones and train with adapted ETF in parallel. After E epochs, all clients submit personal backbones to the server. In the server, personal backbones are received and aggregated to a generic backbone, which is broadcast to all clients participating in the next round. At the inference stage, on the client side, we obtain a generic backbone with standard ETF to handle the world data while on the client side, a personal backbone with adapted ETF to handle the personal data.

4 Theoretical Analysis

In this part, we first primarily introduce some notations and basic assumptions in Sec. 4.1 and then present the convergence guarantees of both local models and the global model under the PCDD with the proper empirical justification and discussion in Sec. 4.2. (Please refer to Appendix B for entire proofs and Appendix D for details on justification experiments.)

Shrink

0

50

100

Communication Rounds

0

30

60

90

Angle

FedAvg in Global

FedETF in Global

FedGELA in Global

(a) Global convergence verification.

0

40

80

120

160

Local Epochs

0

30

60

90

Angle

FedAvg in Local

FedETF in Local

FedGELA in Local

(b) Local convergence verification.

Shrink

Expand

Abandon

(c) Adapted structure.

4.1 Notations

We use t and T to denote a curtain round and pre-defined maximum round after aggregation in federated training, tE to denote the state that just finishing local training before aggregation in round t , and tE + τ to denote τ -th local iteration in round t and 0 ≤ τ ≤ E -1 . The convergence follows some common assumptions in previous FL studies and helpful math results [15, 20, 2931, 33, 36, 38, 45, 51] including smoothness, convexity on loss function F 1 , F 2 , · · · , F N of all clients, bounded norm and variance of stochastic gradients on their gradient functions ∇ F 1 , ∇ F 2 , · · · , ∇ F N and heterogeneity Γ 1 reflected as the distance between local optimum W ∗ k and global optimum W ∗ . Please refer to the concrete descriptions of those assumptions in Appendix B. Besides, in Appendix B, we additionally provide a convergence guarantee without a bounded norm of stochastic gradients, as some existing works [24, 32] point out the contradiction to the strongly convex.

4.2 Convergence analysis

Here we provide the global and local convergence guarantee of our FedGELA compared with FedAvg and FedGE (FedAvg with only the Globally Fixed ETF) in Theorem 1 and Theorem 2. To better explain the effectiveness of our FedGELA in local and global tasks, we record the averaged angle between all class means in global and existing class means in local as shown in Figure 3(a) and Figure 3(b). Please refer to Appendix B for details on the proof and justification of theorems.

Theorem 1 (Global Convergence) . If F 1 , ..., F N are all L-smooth, µ -strongly convex, and the variance and norm of ∇ F 1 , ..., ∇ F N are bounded by σ and G . Choose κ = L/µ and γ = max { 8 κ, E } , for all classes c and sample i , expected global representation by cross-entropy loss will converge to:

where in FedGELA, B = ∑ N k =1 ( p 2 k σ 2 + p k || Φ k W L -W L || ) + 6 L Γ 1 + 8( E -1) 2 G 2 . Since W L = W L, ∗ and ( W L, ∗ ) T h i, ∗ c i ≥ E [( W L ) T h i c i ] , h i c i will converge to h i, ∗ c i .

In Theorem 1, the variable B represents the impact of algorithmic convergence ( p 2 k σ 2 ), non-iid data distribution ( 6 L Γ 1 ), and stochastic optimization ( 8( E -1) 2 G 2 ). The only difference between FedAvg, FedGE, and our FedGELA lies in the value of B while others are kept the same. FedGE and FedGELA have a smaller G compared to FedAvg because they employ a fixed ETF classifier that is predefined as optimal. FedGELA introduces a minor additional overhead ( p k || Φ k W L -W L || ) on the global convergence of FedGE due to the incorporation of local adaptation to ETFs. The cost might be negligible, as σ , G , and Γ 1 are defined on the whole model weights while p k ∥ Φ k W L -W L ∥ is defined on the classifier. To verify this, we conduct experiments in Figure 3(a), and as can be seen, FedGE and FedGELA have similar quicker speeds and larger classification angles than FedAvg.

Theorem 2 (Local Convergence) . If F 1 , ..., F N are L-smooth, variance and norm of their gradients are bounded by σ and G , and the heterogeneity is bounded by Γ 1 , clients' expected local loss satisfies:

where in FedGELA, A = ( η t -L 2 η 2 t ) EG 2 -L ∥ ∥ Φ k W L -W L ∥ ∥ , which means if A -G 4 LE ( G 2 + σ 2 ) ≤ 0 , there exist learning rate η t making the expected local loss decreasing and converging.

In Theorem 2, only 'A' is different on the convergence among FedAvg, FedGE, and FedGELA. Fixing the classifier as ETF and adapting the local classifier will introduce smaller G and additional cost of L ∥ ∥ Φ k W L -W L ∥ ∥ respectively, which might limit the speed of local convergence. However, FedGELA might reach better local optimal by adapting the feature structure. As illustrated in Figure 3 (c), the adapted structure expands the decision boundaries of existing major classes and better utilizes the feature space wasted by missing classes. To verify this, in Figure 3(b), we record the averaged angles between the existing class means during the local training. It can be seen that FedGELA converges to a much larger angle than both FedAvg and FedGE, which suits our expectations. More angle results can be seen in Figure 5.

5 Experiments

5.1 Experimental Setup

Datasets. We adopt three popular benchmark datasets SVHN [23], CIFAR10/100 [16] in federated learning. As for data splitting, we utilize Dirichlet Distribution (Dir ( β ), β = { 10000 , 0 . 5 , 0 . 2 , 0 . 1 } ) to simulate the situations of independently identical distribution and different levels of PCDD. Besides, one standard real-world PCDD dataset, Fed-ISIC2019 [4, 7, 34, 35] is used, and we follow the setting in the Flamby benchmark [34]. Please refer to Appendix C for more details.

Metrics. Denote PA as the personal accuracy, which is the mean of the accuracy computed on each client test dataset, and GA as the generic accuracy on global test dataset (mixed clients' test datasets). Since there is no global model in P-FL methods, we calculate GA of them as the averaged accuracy of all best local models on global test dataset, which is the same as FedRod [3]. Regarding PA, we record the best results of personal models for P-FL methods while for G-FL methods we fine-tune the best global model in 10 epochs and record the averaged accuracy on all client test datasets. For FedRod and FedGELA, we can directly record the GA and PA (without fine-tuning) during training.

Implementation. We compare FedGELA with FedAvg, FedRod [3], multiple state-of-the-art methods in G-FL (FedRS [21], MOON [17], FedProx [19], FedGen [54] and FedLC [47]) and in P-FL (FedRep [6], FedProto [33] and FedBABU [25]). For SVHN, CIFAR10, and CIFAR100, we adopt a commonly used ResNet18 [8, 17, 47, 48, 52] with one FC layer as the backbone, followed by a layer of classifier. FedGELA replaces the classifier as a simple ETF. We use SGD with learning rate 0.01, weight decay 10 -4 , and momentum 0.9. The batch size is set as 100 and the local updates are set as 10 epochs for all approaches. As for method-specific hyper-parameters like the proximal term in FedProx, we tune it carefully. In our method, there are E W and E H need to set, we normalize features with length 1 ( E H = 1 ) and only tune the length scaling of classifier ( E W ). All methods are implemented by PyTorch [27] with NVIDIA GeForce RTX 3090. See detailed information in Appendix C.

5.2 Performance of FedGELA

In this part, we compare FedGELA with FedAvg, FedRod, three SOTA methods of P-FL (FedRep, FedProto, and FedBABU), four SOTA methods of G-FL (FedProx, MOON, FedRS, FedLC and FedGen) on different aspects including the scale of clients, the level of PCDD, straggler situations, and real-world applications. Similar to recent studies [8, 17, 44], we split SVHN, CIFAR10, and CIFAR100 into 10 and 50 clients and each round select 10 clients to join the federated training, denoted as full participation and partial participation (straggler situation), respectively. With the help of Dirichlet distribution [11], we verify all methods on IID, Non-IID ( β = 0 . 5 ), and extreme Non-IID situations ( β = 0 . 1 or β = 0 . 2 ). As the decreasing β , the level of PCDD increases and we show the heat map of data distribution in Appendix C. We set β = 0 . 2 in partial participation to make sure each client has at least one batch of samples. The training round for SVHN and CIFAR10 is 50 in full participation and 100 in partial participation while for CIFAR100, it is set to 100 and 200. Besides, we also utilize a real federated scenario Fed-ISIC2019 to verify the ability to real-world application.

Full participation and partial participation. As shown in Table 2, with the decreasing β or increasing number of clients, the generic performance of FedAvg and all other methods greatly drops while the personal performance of all methods greatly increases. This means under PCDD and the straggler problem, the performance of generic performance is limited but the personal distribution is easier to capture. As for P-FL methods, they fail in global tasks especially in severe PCDD

situations since they do not consider the global convergence during training. As for G-FL methods, the performance is better in generic tasks but limited in personalized tasks, especially in CIFAR100. They constrain the model's ability to fit personalized distributions during local training, resulting in improved consistency during global optimization. As can be seen, our FedGELA consistently exceeds all baselines for all settings with averaged performance of 2.42%, 5.2% and 5.7% to FedAvg and 1.35%, 1.64% and 1.81% to the best baseline on the three datasets respectively.

Performance in real-world applications. Except for the above three benchmarks, we also verify FedGELA with other methods under a real PCDD federated application: Fed-ISIC2019. As shown in Table 3, our method achieves the best improvement of 2% and 2 . 26% relative to FedAvg and of 0 . 36% and 1 . 25% relative to the best baseline on personal and generic tasks respectively, which demonstrates that our method is robust to practical situations in the both views. In the Appendix D.5, we provide more results on additional two real-world applications named FEMNIST and SHAKESPEARE to further show the effectiveness of our method in the real-world scenarios.

3

4

5

6

0.87

0.89

0.91

SVHN (Global)

Ours

FedAvg

3

4

5

6

0.61

0.65

0.69

CIFAR10 (Global)

Ours

FedAvg

3

4

5

6

0.58

0.60

0.62

CIFAR100 (Global)

Ours

FedAvg

3

4

5

6

0.73

0.74

0.75

0.76

ISIC (Global)

Ours

FedAvg

3

4

5

6

0.93

0.95

0.97

SVHN (Local)

Ours

FedAvg

3

4

5

6

0.77

0.80

0.83

CIFAR10 (Local)

Ours

FedAvg

3

4

5

6

0.62

0.66

0.70

CIFAR100 (Local)

Ours

FedAvg

3

4

5

6

0.77

0.78

0.79

0.80

ISIC (Local)

Ours

FedAvg

0

50

100

Communication Rounds

0

30

60

90

Angle

FedAvg in Global

FedGE in Global

FedGELA in Global

(a) Global, Full Parti.,

β

= 0

.

5

0

50

100

Communication Rounds

0

30

60

90

Angle

FedAvg in Global

FedGE in Global

FedGELA in Global

(b) Global, Full Parti.,

β

= 0

.

1

0

50

100

Communication Rounds

0

30

60

90

Angle

FedAvg in Global

FedGE in Global

FedGELA in Global

(c)

Global, Partial Parti.,

β

= 0

.

5

0

40

80

120

160

Local Epochs

0

60

120

Angle

FedAvg in Local

FedGE in Local

FedGELA in Local

(d) Local, Full Parti.,

β

= 0

.

5

0

40

80

120

160

Local Epochs

0

60

120

Angle

FedAvg in Local

FedGE in Local

FedGELA in Local

(e) Local, Full Parti.,

β

= 0

.

1

0

40

80

120

160

Local Epochs

0

60

120

Angle

FedAvg in Local

FedGE in Local

FedGELA in Local

(f)

Local, Partial Parti.,

β

= 0

.

5

5.3 Further Analysis

More angle visualizations. In Figure 5, we show the effectiveness of local adaptation in FedGELA and verify the convergence of fixed classifier as ETF and local adaptation compared with FedAvg. Together with Figure 3, it can be seen that, compared with FedAvg, both FedGE and FedGELA converge faster to a larger angle between all class means in global. In the meanwhile, the angle between existing classes of FedGELA in the local is much larger, which proves FedGELA converges better than FedAvg and the adaptation brings little limits to convergence but many benefits to local performance under different levels of PCDD.

Hyper-parameter. FedGELA introduces constrain E H on the features and the length E W of classifier vectors. We perform L 2 norm on all features in FedGELA, which means E H = 1 . For the length of the classifier, we tune it as hyper-parameter. As shown in Figure 4, from a large range from 10e3 to 10e6 of E W , our method achieves bilateral improvement compared to FedAvg on all datasets.

Ablation studies. Since our method includes two parts: global ETF and local adaptation, we illustrate the average accuracy of FedGELA on all splits of SVHN, CIFAR10/100, and Fed-ISIC2019 without the global ETF or the local adaptation or both. As shown in Table 4, only adjusting the local classifier does not gain much in personal or global tasks, and compared with FedGE, FedGELA achieves similar generic performance on the four datasets but much better performance on the personal tasks.

Performance under pure PCDD setting. To verify our method under pure PCDD, we decouple the PCDD setting and the ordinary heterogeneity (Non-PCDD). In Table 5, we use PxCy to denote the dataset is divided in to x clients with y classes, and in each round, 10 clients are selected into federated training. The training round is 100. According to the results, FedGELA achieves significant improvement especially 18 . 56% to FedAvg and 10 . 04% to the best baseline on CIFAR10 (P50C2).

Other types of Φ . Considering the aggregation of local classifiers should align with the global classifier, which ensures the validity of theoretical analyses from both global and local perspectives, ∑ N k =1 p k Φ k should be 1 ( 1 is all-one matrix). Assuming the row vector of distribution matrix (Φ k ) T c is related to class distribution n k,c n k and the relationship as Q k ( n k,c n k ) . The equation can be rewrite as: γ ∑ N k =1 p k Q k ( n k,c n k ) = 1 , where γ is the scaling constant. In our FedGELA, to avoid sharing statistics for privacy, we only find one potential way that Q k ( n k,c n k ) = n k,c n k and γ = 1 C . In this part, we have also considered employing alternative methods like employing an exponential or power function of the number of samples. As shown in the Table 6, other methods need to share Q k ( n k,c n k ) but achieve the similar performance compared to FedGELA, which exhibits the merit of our choice.

In Appendix D, we provide more experiments from other perspectives like communication efficiency and the local burden of storing and computation, to show promise of FedGELA.

6 Conclusion

In this work, we study the problem of partially class-disjoint data (PCDD) in federated learning on both personalized federated learning (P-FL) and generic federated learning (G-FL), which is practical and challenging due to the angle collapse of classifier vectors for the global task and the waste of space for the personal task. We propose a novel method, FedGELA, to address the dilemma via a bilateral curation. Theoretically, we show the local and global convergence guarantee of FedGELA and verify the justification on the angle of global classifier vectors and on the angle between locally existing classes. Empirically, extensive experiments show that FedGELA achieves promising improvements on FedAvg under PCDD and outperforms state-of-the-art methods in both P-FL and G-FL.

References

Manoj Ghuhan Arivazhagan, Vinay Aggarwal, Aaditya Kumar Singh, and Sunav Choudhary. Federated learning with personalization layers. arXiv preprint arXiv:1912.00818 , 2019.

Sebastian Caldas, Sai Meher Karthik Duddu, Peter Wu, Tian Li, Jakub Koneˇ cn` y, H Brendan McMahan, Virginia Smith, and Ameet Talwalkar. Leaf: A benchmark for federated settings. arXiv preprint arXiv:1812.01097 , 2018.

Hong-You Chen and Wei-Lun Chao. On bridging generic and personalized federated learning for image classification. In International Conference on Learning Representations , 2022.

Noel CF Codella, David Gutman, et al. Skin lesion analysis toward melanoma detection: A challenge at the 2017 international symposium on biomedical imaging (isbi), hosted by the international skin imaging collaboration (isic). In ISBI , pages 168-172, 2018.

Gregory Cohen, Saeed Afshar, Jonathan Tapson, and Andre Van Schaik. Emnist: Extending mnist to handwritten letters. In 2017 international joint conference on neural networks (IJCNN) , pages 2921-2926. IEEE, 2017.

Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai. Exploiting shared representations for personalized federated learning. In International Conference on Machine Learning , pages 2089-2099. PMLR, 2021.

Marc Combalia, Noel CF Codella, Veronica Rotemberg, Brian Helba, Veronica Vilaplana, Ofer Reiter, Cristina Carrera, Alicia Barreiro, Allan C Halpern, Susana Puig, et al. Bcn20000: Dermoscopic lesions in the wild. arXiv preprint arXiv:1908.02288 , 2019.

Ziqing Fan, Yanfeng Wang, Jiangchao Yao, Lingjuan Lyu, Ya Zhang, and Qi Tian. Fedskip: Combatting statistical heterogeneity with federated skip aggregation. In 2022 IEEE International Conference on Data Mining (ICDM) , pages 131-140, 2022. doi: 10.1109/ICDM54844.2022. 00023.

Cong Fang, Hangfeng He, Qi Long, and Weijie J Su. Exploring deep neural networks via layer-peeled model: Minority collapse in imbalanced training. Proceedings of the National Academy of Sciences , 118(43):e2103091118, 2021.

Eduardo Gaitan, Norman C Nelson, and Galen V Poole. Endemic goiter and endemic thyroid disorders. World journal of surgery , 15(2):205-215, 1991.

Jonathan Huang. Maximum likelihood estimation of dirichlet distribution parameters. CMU Technique Report , pages 1-9, 2005.

Wenlong Ji, Yiping Lu, Yiliang Zhang, Zhun Deng, and Weijie J Su. An unconstrained layerpeeled perspective on neural collapse. In International Conference on Learning Representations , 2021.

Yilun Jin, Yang Liu, Kai Chen, and Qiang Yang. Federated learning without full labels: A survey. arXiv preprint arXiv:2303.14453 , 2023.

Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. Foundations and Trends® in Machine Learning , 14:1-210, 2021.

Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, et al. Scaffold: Stochastic controlled averaging for federated learning. In ICML , pages 5132-5143, 2020.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Toronto, ON, Canada , 2009.

Qinbin Li, Bingsheng He, and Dawn Song. Model-contrastive federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10713-10722, 2021.

Qinbin Li, Yiqun Diao, Quan Chen, and Bingsheng He. Federated learning on non-iid data silos: An experimental study. In 2022 IEEE 38th International Conference on Data Engineering (ICDE) , pages 965-978. IEEE, 2022.

Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Systems , 2:429-450, 2020.

Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. On the convergence of fedavg on non-iid data. In International Conference on Learning Representations , 2020.

Xin-Chun Li and De-Chuan Zhan. Fedrs: Federated learning with restricted softmax for label distribution non-iid data. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining , pages 995-1005, 2021.

Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics , pages 1273-1282. PMLR, 2017.

Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011 , 2011.

Lam Nguyen, Phuong Ha Nguyen, Marten Dijk, Peter Richtárik, Katya Scheinberg, and Martin Takác. Sgd and hogwild! convergence without the bounded gradients assumption. In International Conference on Machine Learning , pages 3750-3758. PMLR, 2018.

JAE HOON OH, Sangmook Kim, and Seyoung Yun. Fedbabu: Toward enhanced representation for federated image classification. In 10th International Conference on Learning Representations, ICLR 2022 . International Conference on Learning Representations (ICLR), 2022.

Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences , 117 (40):24652-24663, 2020.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems , 32:8026-8037, 2019.

William Shakespeare et al. William Shakespeare: the complete works . Barnes & Noble Publishing, 1989.

Sebastian U. Stich. Local SGD converges fast and communicates little. In International Conference on Learning Representations , 2019.

Sebastian U Stich. Unified optimal analysis of the (stochastic) gradient method. arXiv preprint arXiv:1907.04232 , 2019.

Sebastian U Stich, Jean-Baptiste Cordonnier, and Martin Jaggi. Sparsified sgd with memory. Advances in Neural Information Processing Systems , 31, 2018.

Canh T Dinh, Nguyen Tran, and Josh Nguyen. Personalized federated learning with moreau envelopes. Advances in Neural Information Processing Systems , 33:21394-21405, 2020.

Yue Tan, Guodong Long, Lu Liu, Tianyi Zhou, Qinghua Lu, Jing Jiang, and Chengqi Zhang. Fedproto: Federated prototype learning across heterogeneous clients. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 8432-8440, 2022.

Jean Ogier du Terrail, Samy-Safwan Ayed, Edwige Cyffers, Felix Grimberg, Chaoyang He, Regis Loeb, Paul Mangold, Tanguy Marchand, Othmane Marfoq, Erum Mushtaq, et al. Flamby: Datasets and benchmarks for cross-silo federated learning in realistic healthcare settings. arXiv preprint arXiv:2210.04620 , 2022.

Philipp Tschandl, Cliff Rosendahl, and Harald Kittler. The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific data , 5(1): 1-9, 2018.

Jianyu Wang and Gauri Joshi. Cooperative sgd: A unified framework for the design and analysis of local-update sgd algorithms. The Journal of Machine Learning Research , 22:9709-9758, 2021.

Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H. Vincent Poor. Tackling the objective inconsistency problem in heterogeneous federated optimization. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 7611-7623. Curran Associates, Inc., 2020.

Jianyu Wang, Zheng Xu, Zachary Garrett, Zachary Charles, Luyang Liu, and Gauri Joshi. Local adaptivity in federated learning: Convergence and consistency. arXiv preprint arXiv:2106.02305 , 2021.

Tobias Weyand, Andre Araujo, Bingyi Cao, and Jack Sim. Google landmarks dataset v2-a largescale benchmark for instance-level recognition and retrieval. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 2575-2584, 2020.

Lechao Xiao, Jeffrey Pennington, and Samuel Schoenholz. Disentangling trainability and generalization in deep neural networks. In International Conference on Machine Learning , pages 10462-10472. PMLR, 2020.

Yibo Yang, Shixiang Chen, Xiangtai Li, Liang Xie, Zhouchen Lin, and Dacheng Tao. Inducing neural collapse in imbalanced learning: Do we really need a learnable classifier at the end of deep neural network? In Advances in Neural Information Processing Systems , 2022.

Jiangchao Yao, Shengyu Zhang, Yang Yao, Feng Wang, Jianxin Ma, Jianwei Zhang, Yunfei Chu, Luo Ji, Kunyang Jia, Tao Shen, et al. Edge-cloud polarization and collaboration: A comprehensive survey for ai. IEEE Transactions on Knowledge and Data Engineering , 35(7): 6866-6886, 2022.

Rui Ye, Zhenyang Ni, Fangzhao Wu, Siheng Chen, and Yanfeng Wang. Personalized federated learning with inferred collaboration graphs. 2023.

Rui Ye, Mingkai Xu, Jianyu Wang, Chenxin Xu, Siheng Chen, and Yanfeng Wang. Feddisco: Federated learning with discrepancy-aware collaboration. arXiv preprint arXiv:2305.19229 , 2023.

Hao Yu, Sen Yang, and Shenghuo Zhu. Parallel restarted sgd with faster convergence and less communication: Demystifying why model averaging works for deep learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 33, pages 5693-5700, 2019.

Honglin Yuan, Warren Richard Morningstar, Lin Ning, and Karan Singhal. What do we mean by generalization in federated learning? In ICLR , 2022.

Jie Zhang, Zhiqi Li, Bo Li, Jianghe Xu, Shuang Wu, Shouhong Ding, and Chao Wu. Federated learning with label distribution skew via logits calibration. In International Conference on Machine Learning , pages 26311-26329. PMLR, 2022.

Ruipeng Zhang, Qinwei Xu, Chaoqin Huang, Ya Zhang, and Yanfeng Wang. Semi-supervised domain generalization for medical image analysis. In 2022 IEEE 19th International Symposium on Biomedical Imaging (ISBI) , pages 1-5. IEEE, 2022.

Ruipeng Zhang, Ziqing Fan, Qinwei Xu, Jiangchao Yao, Ya Zhang, and Yanfeng Wang. Grace: Ageneralized and personalized federated learning method for medical imaging. In International Conference on Medical Image Computing and Computer-Assisted Intervention , pages 14-24. Springer, 2023.

Ruipeng Zhang, Qinwei Xu, Jiangchao Yao, Ya Zhang, Qi Tian, and Yanfeng Wang. Federated domain generalization with generalization adjustment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3954-3963, 2023.

Yuchen Zhang, Martin J Wainwright, and John C Duchi. Communication-efficient algorithms for statistical optimization. Advances in neural information processing systems , 25, 2012.

Zhihan Zhou, Jiangchao Yao, Yan-Feng Wang, Bo Han, and Ya Zhang. Contrastive learning with boosted memorization. In International Conference on Machine Learning , pages 27367-27377. PMLR, 2022.

Zhihui Zhu, Tianyu Ding, Jinxin Zhou, Xiao Li, Chong You, Jeremias Sulam, and Qing Qu. A geometric analysis of neural collapse with unconstrained features. Advances in Neural Information Processing Systems , 34:29820-29834, 2021.

Zhuangdi Zhu, Junyuan Hong, and Jiayu Zhou. Data-free knowledge distillation for heterogeneous federated learning. In International conference on machine learning , pages 12878-12889. PMLR, 2021.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses a challenge in Federated Learning referred to as partially class-disjoint data (PCDD), where each client contributes a part of classes (instead of all classes) of samples. Without full classes, the local objective will contradict the global objective, yielding the angle collapse problem for locally missing classes and the space waste problem for locally existing classes. This is a real-world challenge since it is not uncommon for example that some classes will be well sampled in certain regions, but not others. Prior art mainly focus on the general heterogeneity without specially considering partially class disjoint challenges. Without full classes, the local objective will contradict the global objective, yielding the angle collapse for locally missing classes and the waste of space for locally existing classes. The goal is to achieve holistic improvement in the bilateral views (both global view and local view) of federated learning. \n\nThe authors propose FedGELA where the classifier is globally fixed as a simplex ETF while locally adapted to the personal distributions. Globally, FedGELA provides fair and equal discrimination for all classes and avoids inaccurate updates of the classifier, while locally it utilizes the space of locally missing classes for locally existing classes. \n\nThe proposed approach builds upon simplex equiangular tight frame (ETF), which provides each class the same classification angle and generalizes well on imbalanced data. Specifically, in their FedGELA approach, the classifier is globally fixed as a simplex ETF while locally adapted based on the local distribution matrix to utilize the wasted space for the existing classes. In the global view, FedGELA merges class features and their corresponding classifier vectors, which converge to ETF. In the local view. it provides existing major classes with larger feature spaces and encourages to utilize the spaces wasted by locally missing classes. \n\nContributions are summarized as :\n- Study algorithmic implication of a real-world  challenge (partially class-disjoint data (PCDD), namely angle collapse and space waste \n- Propose FedGELA and theoretically show the local and global convergence analysis for PCDD with the experimental verification \n- Evaluate on multiple benchmark datasets under the PCDD case and a real-world dataset to demonstrate the bilateral advantages of FedGELA over the state of the art methods.\n\nSTRENGTHS:\n1. Related work well covers comparison among a range of FL methods and why PCDD not covered. Examples of why prior art does not address PCDD include: generic federated leaning adopt a uniform treatment of all classes, then attempt mitigate personal differences; personalized federated learning places less emphasis on locally missing classes and selectively shares parameters/prototypes to minimize the impact of personal characteristics. While these methods might directly or indirectly help mitigate the data shifts caused by PCDD, neither achieve holistic improvement for global and local views\n2. Performance evaluaton on 3 relevant datasets (SVHN, CIFAR10, CIFAR100), against all the top state of the art algorithms as baselines, and showng it outperforms. FedGELA consistently exceeds all baselines.\n\nWEAKNESSES:\nOverall, the paper represents is a solid contribution - well defined problem not addressed by prior art and representative of real-world problem for fedrated learing. Solid treatment of prior art, and differentiation from prior methods. \nWeaknesses:\n1. Only 1 real PCDD federated application Fed-ISIC2019 was evaluated - however I am not aware of other benchmarks I would recommend.\n2. Performance improvements against to the best baseline for all tests were all <3% performance improvement.\n\nQUESTIONS:\n1. Given the performance improvements were in the <3% against the best baselines - can you provide more information? For example does it perform better worse under certain conditions? if so, pls distinguish, and if possible explain what this may imply about either the algorithm and limitations, or perhaps a limitation in the benchmark wrt fully characterizing the real world challenge?\n\nLIMITATIONS:\nNo negative societal effects. Limitations explained in Weaknesses section.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper mainly focuses on the partially class-disjoint data (PCDD) problem in federated learning (FL) settings, which is a common yet challenging problem in distributed data sources. Inspired by a classifier structure (simplex equiangular tight frame, ETF), the authors of the paper propose FedGELA to tackle the PCDD problem. FedGELA is a variant of FedAvg with local model adaptation (personalization): They first define the classifier $W$, which is the ETF that the classifier should converge to. Here, they also take the client local data distribution ($\\phi$) into account. Afterwards, the feature extractor $H$ will be optimized locally at each client and communicated via server-client communication. Finally, the global feature extractor and the $W$ at central server, as well as the local feature extractors and the adapted $W$ at clients will be returned.\n\nSTRENGTHS:\nThe proposed method is motivated very well. The schematic illustration is also clear. The theoretical analysis is sound. Experiments and the results are good.\n\nWEAKNESSES:\nFrom my understanding, FedGELA focuses only on the alignment in the feature embedding space, which has been done by many previous works (FedGen [1], FedProto [2], …). Therefore this paper lacks significance to some extent. Also, the definition of $W$ (ETF, the global classifier), as well as the locally adapted ones looks straightforward.\n\nQUESTIONS:\n1.\tYou claim that FedGELA could mitigate PCDD from both Global and Local view. But in Algorithm 1, the global model $H^T$ is simply an average of the client model and there is no server-side optimization. Could you please explain this?\n2.\tIn Table 1, you claim that FedGELA could mitigate the model skew, could you please explain this point in more details? Also, in terms of “save space”, since you are transmitting the whole feature backbone $H$, which is the resnet18 in your experiments, what is the reason of space saving?\n3.\tWhy do you use $H$ to represents 2 different terms, features (Line 114) and global backbone (in the Algorithm)? It’s a bit confusing during reading.\n4.\tCould you shortly explain the converged angles in your figures? Is there a specific meaning of the values? E.g. in Figure 2. \n5.\tIn Equation 4, you model the client data skew in the label space via $\\phi$, which is based on the number of samples from different classes. Have you experimented with other options? This looks a bit too straightforward to me.\n6.\tFedGen[1] is a method which augment the feature embedding space using a shared feature generator, which could possibly mitigate the issue of “waste of space” by generating synthetic embeddings from minority classes. Could you provide a comparison with this work?\n[1] Zhu, Zhuangdi, Junyuan Hong, and Jiayu Zhou. \"Data-free knowledge distillation for heterogeneous federated learning.\" ICML, 2021.\n[2] Tan, Yue, et al. \"Fedproto: Federated prototype learning across heterogeneous clients.\" AAAI 2022.\nMinors:\n1.\tLine 115, $E_W$ and $E_H$ should be introduced at their first appearance.\n2.\tLine 298, the selection of $E_W$ seems to be dataset specific, is there any default values suggestions?\n\nLIMITATIONS:\nFedGELA is only tested in the client data with label skew. Is it also applicable to the data with feature skew?", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel Federated Learning Algorithm to address the Partially class-disjoint data (PCDD) problem. The approach is based on the simplex equiangular tight frame (ETF) phenomenon to solve the angle collapse issue and introduces a second projection to personalize an adapted structure to save space. The main contributions can be summarized in three aspects: identifying the angle collapse and space waste challenges in the PCDD problem, introducing the novel FedGELA algorithm, and conducting a range of experiments to evaluate its performance. The paper also includes a theoretical analysis with convergence analysis.\n\nSTRENGTHS:\n1. The paper is well-written with a proper structure and clear explanations. The presentation of the authors' ideas is easy to follow due to the effective use of figures and notations.\n2. The methodology of the FedGELA algorithm is interesting, and the mathematical deductions are sufficient. The algorithm is clear and provides enough information for reproducibility.\n3. The algorithm has been wisely experimented, and the plots are suitable and clear.\n\nWEAKNESSES:\n1. The authors claim that \"none of the existing methods can intrinsically mitigate PCDD challenges to achieve holistic improvement in the bilateral views of federated learning.\" However, the PCDD problem seems closely related to the general non-iid (non-independent and identically distributed) problem. The main differences between these two problems have not been explained.\n2. Based on my understanding, if PCDD is different from the non-iid problem, it should perhaps be related to the multi-label problem. However, the presentation of the paper, the experimental data, and the methods of experimental comparison all tend to be more inclined towards non-iid problems. Non-iid is a common problem setting, which contradicts the first author's claim of contributions.\n3. The performance improvement is limited.\n4. I disagree with the statement that \"restricting local structure will waste feature space and limit the training of the local model on existing classes.\" I believe the notion of \"waste of space\" is unfounded as it appears to have no impact on computational efficiency or performance improvement.\n\nConclusion:\nThe methodology and algorithm presented in this paper are interesting, and the paper is written in high quality. However, there seems to be an important flaw in the problem setting. PCDD appears not to be a new issue but rather a non-iid problem under some special conditions.\n\nQUESTIONS:\nsee weaknesses\n\nLIMITATIONS:\nsee weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
The authors study the problem of federated learning over partially class-disjoint data and propose using equiangular tight frame (ETF) techniques that allows achieving better performance in both the global and personal learning tasks. They show that the existing federated learning approaches suffer either from angle collapse for locally missing classes or from waste of space for locally existing classes, and propose their approach FedGELA which solves both the issues.

STRENGTHS:
+ Extensive experiments comparing the proposed approach, FedGELA, with the existing federated learning approaches.
+ Detailed theoretical analysis of the proposed approach.
+ Highlighting the issues of angle collapse and waste-of-space in the federated learning with partially class-disjoint data.

WEAKNESSES:
- Borrows the existing EFT techniques and hence the novelty seems to be limited.
- Overall improvement in average accuracy is marginal (~1.5%) over the existing approaches.

QUESTIONS:
I like the experimental evaluations and thorough comparison with the prior federated learning works, but I'm concerned about the algorithmic novelty of the proposed approach. Most of the techniques seem to be adapted from the prior known literature on EFT. Can the authors highlight the technical difficulties in directly applying the prior techniques in solving the PCDD problem?

LIMITATIONS:
I don't think there are any negative societal impacts of this work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-23