FedLoGe: Joint Local and Generic Federated Learning under Long-tailed Data
Authors: Zikai Xiao
Zihan Chen
Liyinglan Liu
YANG FENG
Joey Tianyi Zhou
Jian Wu
Wanlu Liu
Howard Hao Yang
Zuozhu Liu
Venue: ICLR 2024 poster (2024)
Keywords: Personalized federated learning
Data heterogeneity
Long-tailed Learning
Primary area: general machine learning (i.e., none of the above)

Abstract: Federated Long-Tailed Learning (Fed-LT), a paradigm wherein data collected from decentralized local clients manifests a globally prevalent long-tailed distribution, has garnered considerable attention in recent times. In the context of Fed-LT, existing works have predominantly centered on addressing the data imbalance issue to enhance the efficacy of the generic global model while neglecting the performance at the local level. In contrast, conventional Personalized Federated Learning (pFL) techniques are primarily devised to optimize personalized local models under the presumption of a balanced global data distribution. This paper introduces an approach termed Federated Local and Generic Model Training in Fed-LT (FedLoGe), which enhances both local and generic model performance through the integration of representation learning and classifier alignment within a neural collapse framework. Our investigation reveals the feasibility of employing a shared backbone as a foundational framework for capturing overarching global trends, while concurrently employing individualized classifiers to encapsulate distinct refinements stemming from each client’s local features. Building upon this discovery, we establish the Static Sparse Equiangular Tight Frame Classifier (SSE-C), inspired by neural collapse principles that naturally prune extraneous noisy features and foster the acquisition of potent data representations. Furthermore, leveraging insights from imbalance neural collapse's classifier norm patterns, we develop Global and Local Adaptive Feature Realignment (GLA-FR) via an auxiliary global classifier and personalized Euclidean norm transfer to align global features with client preferences. Extensive experimental results on CIFAR-10/100-LT, ImageNet, and iNaturalist demonstrate the advantage of our method over state-of-the-art pFL and Fed-LT approaches.

Published as a conference paper at ICLR 2024
FEDLOGE: JOINT LOCAL AND GENERIC FEDERATED LEARNING UNDER LONG-TAILED DATA
Zikai Xiao" Zihan Chen², Liyinglan Liu', Yang Feng", Jian Wu', Wanlu Liu',
Joey Tianyi S Howard Hao Yang'. Zuozhu Liu
I 'Zhejiang University,
SSinaapore University of Technology and Design,
UUniversity of Electronic Science and Technology of China,
4 Angelalign Technology Inc,
'IHPC, Agency for Science, Technology and Research, Singapore, *CAR Agency for Science, Technology and Research, Singapore zikai z ju. on
ABSTRACT
Federated Long-Tailed Learning (Fed-LT), a paradigm wherein data collected from decentralized local clients manifests a globally prevalent long-tailed dis- tribution, has garnered considerable attention recent times. In the context of Fed-LT, existing works have predominantly centered on addressing the data im- balance issue to enhance the efficacy of the generic global model while neglecting the performance at the local level. In contrast, conventional Personalized Fed- erated Learning (pFL) techniques are primarily devised to optimize personalized local models under the presumption of balanced global data distribution. This paper introduces an approach termed Federated Local and Generie Model Train- ing in Fed-LT (FedLoGe), which enhances both local and generic model perfor- mance through the integration of representation learning and classifier alignment within neural collapse framework. Our investigation reveals the feasibility of employing shared backbone as a foundational framework for capturing overar- ching global trends, while concurrently employing individualized classifiers en- capsulate distinct refinements stemming from each client's local features. Build- ing upon this discovery, we establish the Static Sparse Equiangular Tight Frame Classifier (SSE-C). inspired by neural collapse principles that naturally prune ex- traneous noisy features and foster the acquisition of potent data representations. Furthermore, leveraging insights from imbalance neural collapse's 's classifier norm patterns, we develop Global and Local Adaptive Feature Realignment (GLA- FR) via auxiliary global classifier and personalized Euclidean norm transfer to align global features with client preferences. Extensive experimental results CIFAR- 10/100-LT. ImageNet-LT, and iNaturalist demonstrate the advantage our method over ofate-of-the--eee pFL and Fed-LT approaches. Our codes are available http://hww....oootmt////////aaaa
INTRODUCTION
Federated learning (FL) enables collaborative model training across decentralized clients without exposing local private data (McMahan et al.) 2017 Kairouz et al., 2021). Recent work further inves- tigates the federated long-tailed learning (Fed-LT) task, where the global data exhibits long-tailed distributions and local clients hold heterogencous distributions (Chen et al. 2022b (Shang et al.) 2022b). They usually learn a well-trained generic global model, whose performance might degrade when universally applied to all clients with diverse data, degrading its practical applicability. For ample, in the realm of smart healthcare, as demonstrated by Lee & Shin (2020). Chen et al. (2022a),
"Co-first author.
1 'Correpoondigg author.

---

Published as a conference paper at ICLR 2024
and Elbatel et al. (003) the capacity of the global model deliver high-quality diagnosties is lim- ited, as patient distributions vary across specialized hospitals. Additionally, in cross institutional financial applications like credit scoring (Dastile al. 2020) and fraud detection (Awoyemi et al.
2017). individuals from different regions or age groups may exhibit dissimilar credit patterns.
A high-quality global model with balanced performance could speed up local adaptation and attract new clients, while personalized models aim to provide enhanced local performance by considering local data characteristics. Nevertheless, existing works Fed-LT have primarily focused on ad- dressing the imbalance in the context of the global long-tailed data (Yang et al. 2023a Qian et al,) 2023 Xiao et al. 2023), neglecting the tailoring of models to the needs of individual clients, since local data statistics could be diverse and not necessarily long-tailed. Personalized federated learning (pFL.) (Tan et al., 2022). which trains customized local models for a single or a group of clients, offers an alternative solution to prioritize each client's is (or group's) distinct data statistics and pref- erences, in which the global generic model is deemed as bridge for training local personalized models and boosting the local performance with expressive representations (Li et al. 202 lb Collins
ct al. 202T Li et al.
2021a). However, conventional pFL approaches are not supposed to attain superior global generic model in Fed-LT.
ligh quality
Low quality
Enhance
Mask
20
Dominant
Mean
Relative Variance
Relative Variance
15
Relative Variance
3-
Mean
Noises
Mean
10
10
1-
5-
Negligible
0-
0
100
200
100
400
500
100
200
300
-400
500
Feature
Feature (b)
Figure (a): The mean (sorted descending order) and variance of class means unveil feature degeneration: The feature collapse property ceases to prevail, and features with diminished means exhibit substantial variance rather than being zero; (b): After training the backbone with SSE-C, in which noisy features with bigger variance are partially pruned (gray shaded vertical lines), and enhance the quality (smaller variance) of dominant features.
Designing framework simultaneously train global and local models in the presence of Fed-LT remains a critical challenge. Inspired by Kang et al.] (2019), we find that the generality and trans- ferability of feature extractors significantly superior classifiers (Kim et al. 2022) Vasconcelos jet al. 2022). On the contrary, adjusting the classifier has proven to be quite effective in address- ing the imbalance and heterogeneity issues (Li al., 2022al Zhang al., 2022a) In other words, the feature extractor can serve as the cornerstone to reflect global trends, while adjusting classifiers can induce the model to adaptively achieve superior personalized performance across the server and heterogeneous clients. Consequently, we conceptualize our model learning with two intertwined processes: global representation learning and mabalanced/heteroggneee classifier adaptation. Adopting this viewpoint, we identify two key challenges for personalized Fed-IT::
C1: How to learn effective representations under heterogeneous and imbalanced data?
Due to heterogeneity, each client captures different feature distributions, leading to divergence dur- ing model aggregation and inferior global performance. Recent studies show that training with a fixed classifier can reduce divergence among heterogeneous clients to improve performance (Oh) et al. 2022, Dong et al., 2022]. The fixed classifier serves as a consistent criterion for learning rep- resentations across clients over time, rather than being optimal choice itself. For instance, Yang ctt al. (2022) proposes to initialize the classifier as a simplex equiangular tight frame (ETF) with maximal pairwise angles under imbalanced learning. In general, these fixed classifiers force the feature prototypes to converge to an optimal structure to improve representation learning. However, their effectiveness in resolving Fed-LT regarding both global and local models is not investigated.
We examine the effectiveness of training with fixed classifiers in Fed-LT from the perspective of neural collapse (NC) (Papyan al. 2020). The NC identifies salient property of the feature
2

---

Published as a conference paper at ICLR 2024
space, showing that all within-same However, as shown in Fig. - preliminary experiments benchmarking ETF with CIFAR- 100 in Fed-LT suggest that only a few features have relatively large means, while most of the small- mean features are contaminated by severe noise. Such observations are inconsistent with the feature collapse property, and we coin it as feature degeneration. More details of the computation process for the data in Fig. D as well as the necessary explanation can be found in Appendix A.2.
features tend to collapse to their respective class means.
To resolve the feature degeneration for improved representation learning, we propose the Static Sparse Equiangular Tight Frame Classifier (SSE-C) inspired by the sparse coding theories (Frankle) & Carbin 2019 Glorot et al., 201T). The assumption of SSE-C that the small-mean degenerated features contribute little to model performance while forcing sparsity on them would help learn more expressive representations. We refer to the small-mean features as negligible features and the large- mean features as dominant features. The SSE-C dynamically prunes the classifier weights of those small-mean noisy features, while holding more expressive dominant features. Probing into weights trained with SSE-C validates our assumption, as shown in Fig. [bb) and experiments.
C2: How to conduct effective feature realignment to improve the performance of both the generic and personalized models bused on data preferences?
In Fed-LTT, the long- requirements learn different global and personalized local models for satisfactory performance. Thus, the feature extractor trained with a fixed classifier needs to be realigned to both global and local models. For the global model, it is necessary realign the model to improve its performance on global tail classes. For the personalized model, the classifier needs to align features with respective heterogeneous local data preferences.
global data distribution and heterogeneous local distributions raise the
In this work, we unify the feature realign- ment for both the server and clients un- der the neural collapse framework. The key idea to align both global and local classifiers based on the weight norm of the classifiers. Previous works show that clas- sifier weight norms are closely correlated with the corresponding class cardinalities
Input
Backhone
Classifiers
Norms
Realignment
vissuc
1el
Generi
9c
i
Local
&ec 1%a 11obol
(Kang et al. 2019 Tan et al. 1 Li
F Forward
Shared
Transter
9G Generic Prediction
ctt al., 2020b). Further research provides both empirical explanation and thcoretical justification that the classifier weight norm larger for majority classes while smaller for minorities (Kim & Kim| 2020 Dang al. 2023 Thrampoulidis al. (022)..
Backward
Personalized
Decline
DP
Local Prediction
The framework of FedLoGe. The SSE-C enhances the capability of the backbone (feature ex- tractor); during feature alignment, the model transfers the most crucial information from the global classifier to the personalized model, omitting information per- taining to categories with low usage.
Figure 2:
We propose the Global and Local Adap- tive Feature realignment (GLA-FR) mod- ule to align the backbone trained with SSE-C to the server and clients. In particular, we devise auxiliary classifier heads for the global (t) and K local classifiers (1 kk )), which are trained alternately with SSE-C in each epoch, see Algorithm [ The alignment includes two stages: global alignment and local alignment. The global realignment is simple yet effective, adjusting the weights based on the norms of v (Eq 6 to tackle with global balanced test set. The alignment for person- alized models is a bit different, as their data distributions differ greatly from the global distribution. We integrate the global trends with each local client's preference by adjusting the global classifiers ii with the norms of local classifier Φ (Eq
Our work represents pioneering efforts to achieve a harmonious integration of global and personal- ized model learning under Fed-LT, thereby facilitating each participating institution in obtaining a model that is more adeptly tailored to its inherent characteristics and preferences. Comprehensive
'Our eSssE-C leads to two notable improvements in Fig. (b) over Fig. 1 (a): First, it masks noisy features with enhanced, as shown by reduced variances among those with larger means, reflecting their increased precision and efficacy.
large variances, now in sparse areas marked by grey. Second, the role of dominant features is
3

---

Published as a conference paper at ICLR 2024
experiments representative datasets CIFAR-10/100-LT ImageNet, and iNaturalist demonstrate the outperformance and efficacy of both global generic and local personalized Fed-LT models.
RELATED WORK
2.1 FEDERATED LEARNING
Federated Long-tailed Learning Recent research launched attempts to resolve the Fed-LT task. Model decoupling methods explore frameworks such as classifier retraining (Shang et al. 2022b) and prototype-basee classifier rebalancing (Yang et al. 2023a Dai et al. 2023) for Fed-LT. Shang cct al. (2022a) and Wang et al. (2022) investigate calibration and distillation methods to improve the model performance. Many efforts also have been made from the perspective of meta-learning (Qian 2023 Shen et al., (021), client selection (Zhang et al. 2023 Yang et al. 2021). re-weighting
et al.
Wann et al. 2021: Shen
2021), and aggregation (Chou al. 2022).
Personalized Federated Learning (pFL) To deal with the poor generalization performance of the single generic global at local data, pFL has been vastly investigated. A group of works seeks train local personalized models via transferring knowledge from the generic global model (Li & Wang, methods are ao been explored with client clustering (Sattler et al. 2020 Briggs et al. 2020 Ghosh et al.,
2019 Dinh al., 2020 Fallah et al. 2020 Chen al. (023)). Multi-task learning-
2020 and model interpolation (Deng al. 2020
at. 2021a Diao al. 2020).
For neural network-basse FL framework, parameter decoupling methods have gained popularity due their simplicity. Parameter decoupling aims achieve personalization by decoupling the local private model parameters from the global model parameters. For horizontal decoupling, [i] al. (202Tb) personalizes the batch normalization layers. PPilutta er al. (2022) explores decoupling different parts, and personalizing the last layer is adopted An/vriaznag ct al. (2019), Collins et al.
(2021), and Briggs et al. (2020). For vertical decoupling, [Shen et al. (2022) personalizes channels.
2.2 NEURAL COLLAPSE FOR REPRESENTATION LEARNING
of four interconnected phenomena that demonstrate a pervasive induc- tive bias in the terminal phase training, as shown by Papyan et al. (2020). Subsequently. several works have sought to explain the neural collapse phenomena from the perspective of peeled mod- els (Ji et al. 2021 Fang et al. 2021). unconstrained feature models (Tirer & Bruna 2022 Mixon al. 2020 Zhu al. 2021). and Riemannian manifolds (Yaras al. 2022). Building upon the findings on neural collapse, Yang et al. (2022) first proposed fixing the classifier to an ETF structure and introduced dot regression loss. The ETF structure was later utilized for semantic segmentation (Zhong et al. 2023), handling heterogeneity in federated learning (Li et al.) (003). transfer learning Li et al. 2022b). incremental learning (Yang et al. 2023b), and object detection (Ma et al. 2023)
Neural collapse refers
In summary, our method stands out by introducing personalization within the neural collapse frame- work, effectively overcoming global imbalanced data and enhancing local model personalization through tailored feature distribution alignment. Contrary to FedETF's 's fixed classifier approach and FedRod's dual classifier structure, our method ensures superior handling of global long-tail bias and precise local model tuming, marking novel advancement in Fed-LT. For detailed comparisons, please see Section A.13 in the Appendix.
3 PROPOSED METHOD
In this section, we introduce our proposed Ped-LoGe (see Algorithm []. a simple yet effective framework to achieve joint personalized and generic model learning for Fed-L.T. To boost representa- tion learning and address the global-local inconsistency, we introduce training paradigm consisting of sparsified ETF module and global-local feature alignment modules.
3.1 PRELIMINARIES
We consider an FL system with K clients and a server. The overall objective is to train 1 + K models: 1 generic and K personalized models. Specifically, the generic model is parameterized
4

---

Published as a conference paper at ICLR 2024
by w {0, ) whereas the hthh personalized model for each client k [K] is denoted as wa- We decoupled the neural network models into a feature extractor f(z, f) and a set classifiers. The feature extractor, parameterized by 8, transforms input into features h. The generic classifier g(h, 15) and personalized classifiers g(h, Φx) then map these features to the output labels. The overall global and local objective functions be respectively expressed as:
Σ D D
Global:
min
Cu(wk: Dí), Local: min [(, oR: Dk), {0,001
127
where D (Dx) is the global long-tailed dataset composed of K heterogeneous local datasets, each with size [Dx]. The training executed over T rounds. In each round t, the server distributes the current global model w(()) all clients for local updates. Furthermore, given that e represents the class index, for Ve E C, the classifier vector is ce. and the corresponding features are he.
3.2 STATIC SPARSE EQUIANGULAR TIGHT FRAME CLASSIFIER (SSE-C)
Initializing the classifier as ETF and subsequently freezing it during training has proven to be an effective strategy in federated learning, attributed to ETF's inherent structure, which ideally exhibits commendable properties of feature collapse under balanced data. However, we found that in the context of Fed-LT, the feature collapse property was not satisfied when initializing the fixed classi- fiers due to feature degeneracy (the existence of high-noise feature with a small norm), which is illustrated in Fig. (a). Accordingly, we propose the Static Sparse Equiangular Tight Frame Classi- fier (SSE-C) via fxing classifier to learn higher-quality features and reduce the impact of negligible features so as to achieve effective representation learning. The server will obtain SSE-C with LSSE-C prior to the local training, and all clients will fix SSE-C throughout the training.
We first initialize the classifier as conventional ETF matrix by
C -o
where e = [v 1. c.cl R E RdxC allows any rotation and satisfies U Ic, Ic Ic is the identity matrix. d is the dimension of the classifier vector, and Ic is all-ones vector. We can deduce the important property that all class vector has the equal l2 norm and maximal pair-wise angle -- in Rdd Note that randomly assigning 3 proportion of weights in the ETF matrix to 0 will disrupt the ETF condition: the class vector angles cease to be maximal and equal, and the norms necessary to train sparse ETF structure that satisfies the ETF geometric conditions. We introduce sparse indicator matrix S with the same dimensions as th, where 33 proportion of the elements are randomly set to Then, the sparsified matrix v can be represented as s/' & @ S. We design the Equal Norm Loss and Maximal Angle Loss to optimize the geometrie structure to meet the conditions of ETF. First, the &g norm of class vectors should be equal. We constrain all norms of class vector values predetermined
of the classifier vector become unequal. As such,
Iuorm (1/, ,), - Σ (I S.ill2 7)
Second, we maximize the minimum angle between class vector pairs. Following MMA Wang et al. 20201, we normalize the classifier vector by Vh,, with the formula:
vV. and maximize only the minimum angle I
langte (3, S) -- 1 - cos cos-1
OS.) o...))
max JE(L.2....CMA
(4)
By integrating the hurm and langle we obtain the CSSE. for the following training:
LSSE- Inom (1b,, 3%,,7,55 S) + langle (,),
(5)
Then solve the objective e'ssE- arg ming LSSE-C by SGD. By regularizing class vector norms and maximizing their minimum angle, the classifier exhibits sparsity while maintaining ETF prop- erties, effectively guiding the model in learning robust features.
5

---

Published as a conference paper at ICLR 2024
Algorithm 1 An overview of FedLoGe framework
Input: w w- (0(0) ( 1 Jk=1 K, T,C
(0) 1
Output: ( = (0(T) ()) [wx] 1K (0( ((2)) K
0x k==
Stage 1: Representation Learning with SSE-C
1: Obtain //ssE-c by Equation 5
2: for each round = 1 to T do
3: St < subset of selected clients
4: for each client A: S₁ in parallel do
5: 0g, of + CLIENTUPDATE(K; (((-1), (6(111), e/ssE-c
6: 000) .6((0) AGGREGATION (1-1) .k ( for all lo E Se
Stage 2: Global Feature Realignment
7: for each classifier vector to C do (Equation 6
26 (T)
8:
Align the long- -ailed norm balanced norm
Stage 3: Local Feature Realignment
9:
for each client k: = to K do for each classifier vector e = 1 to C' do
10:
11:
Ok.c
((T)
(T) f v/e kk.
(Equation 7
Incorporate Global Classifier with local statistics
12:
(T) + Ok (T) V (F .kk x)
\> Finelune Pk
(T)
13:
return w (T) (B(T), (b(T),, {wa)
k= (0(T), .kk 1k=1
CrreiionCIENNTUDAATEI 1: ((() (0))) g(t), si(t)
0(0), 2,1), vSSEEC
2: for each local epoch i = 1 to E do
3: Compute features h; + f(z,, 010)
4: 0((+1) 0%0 (V((((ssst.c;az))
Fix (0 120 Update 0 with ussse
(1+1) + u
(t) (C(( -( h:)) }
Fix ( Update (0( 0k
(1)
5: (11))
0k f Ok ( (0()) ):e))
6: return (1+1)) (1+1)
Wk
3.3 GLOBAL AND LOCAL ADAPTIVE FEATURE REALIGNMENT (GLA-FR)
To be adapted in both global and local models, the feature extractor trained with a fixed classifier needs to be realigned to address the imbalance and heterogeneity. Hence, we conduct feature re- alignment to both global and personalized models after training the SSE-C guided feature extractor, where the realignment should be consistent with the local data statistics/class cardinality.
To obtain a good estimation of class cardinality, in prior work such as Kang et al. (2019 Tan et al. (2021): Li et al. (2020)), the classifier weight norms ||wa|| are found to be correlated with the corresponding class cardinalities Ne. in which the is the classifier weight vector for the c-th class. Kim & Kim (2020) provides explanation from the perspective of decision boundaries the weight vector norm for more frequent classes is larger, biasing the decision boundary towards less frequent classes. Also, for the neural collapse framework with imbalanced data, the relations between the weight norm of classifiers and the class cardinality also exist (Thrampoulidis et al.)
2022 Dang et al., 2023). These findings are consistent, which motivates us measure/estimate local data statistics based on the norm weight of the classifier.
The frozen ETF classifier is not suitable for feature alignment, attributed the lack of valid norms to estimate class cardinality. We design a new auxiliary global head & to obtain valid norms, which participate in gradient updates and weight aggregation alongside the backbone. After T' rounds of training, we get the global weight TT) {((T), (T)) The ((T) is well trained with eisSE-C-
For the global adaptive feature distribution process (GA-FR), let the denote a classifier vector corre- sponding to the ddll class, where E C. The aligned classifier vector v can be obtained by dividing

---

Published as a conference paper at ICLR 2024
Non-IID
0.5
Dataset
Imbalance Factor
1F=50
1F=100
IF=50
IF=100
Method/Model
GM
PM
GM
PM
GM
PM
GM
PM
FedAvg
0.7988
0.8722
0,7214
0.8664
0,7890
0.8916
0.7231
0.8935
FedProx
0.7869
0.8653
0.7160
0.8617
0.7790
0,8888
0.7266
0,8897
FedBN
0.7604
0.8837
0.6984
0.8847
0.7596
0.8916
0.7033
0.8925
FedPer
0.8935
0.8931
0.8918
0.8999
FedRep
0.7938
0,8988
0.7218
0,8984
0.7816
0,9043
0.7271
0,9065
CIFAR-IO-LT
Ditto
0,7813
0.8926
0,7180
0.8874
0,7804
0.8967
0.7210
0,8973
FedROD
0,7862
0,9030
0,7137
0.8941
0.7776
0.9025
0.7236
0,9041
FedBABU
0.7851
0.8664
0.7280
0.8613
0.7819
0.8914
0.7316
0.8898
FedETF
0.8056
0.7446
0.6709
0.7323
0.7453
0.8106
0.6615
0.7917
Ratio Loss
0,7995
0.8791
0.7290
0.8724
0.7857
0.8938
0.7342
0.8934
FedLoGe
0.8277
0.9112
0.7672
0.9096
0.8189
0.9104
0.7593
0.9073
FedAvg
0.4271
0.6019
0.3818
0.6214
0:4215
0.6086
0,3797
0.6269
FedProx
0,4276
0.6071
0,3856
0.6214
0.4187
0.6089
0.3788
0.6261
FedBN
0,4332
0.6657
0,3895
0,6703
0.4229
0.6373
0,3847
0.6562
FedPer
0.6888
0.7007
0.6476
0.6737
FedRep
0,4355
0.6911
0.3907
0.6917
0,4283
0.6539
0,3828
0.6831
CIFAR-100
Ditto
0.4312
0.6441
0.3847
0.6588
0,4188
0.6182
0.3822
0.6320
FedRDD
0,4384
0,7124
D.30
"0,6999
"..4230
0.0667
0,302
0.6917
FedBABU
0,4415
0.6416
0.3921
0,6480
0,4336
0.6354
0.3907
0.6598
FedETF
0.4223
0.605S
0,3825
0.6421
0.4278
0.6302
0.4278
0.6507
Loss
0,4326
0.6152
0.3912
0.6348
0.4253
0.6213
0,3839
0.6344
FedLoGe
0.4762
0. 0.7229
0.4233
0.7285
0.4860
0.7099
0.4330
0.7195
Table 1: Test accuracies of our and SOTA methods on CIFAR-10/100-111 with diverse imbalanced and heterogeneous data settings. GM/PM denotes Global/Pensonalized model.
2/ by its l norm as follows:
V/ - vol |1bll2
(6)
Here, ||deel2 represents the & norm of e/c. Each vi will be unit vector, preserving the direction of Ve and possessing the magnitude of 1.
For personalized adaptive feature realignment (LA--RR) adapt the global auxiliary classifier vb to the personalized classifier by multiplying the norm of ox. which implies that clients will leverage information from categories with larger sample size while omitting information pertaining the rare categories. For local classifier vector Øx,c at client k the process of LA-FR is as follows:
Kh.c - vh * 1|02,,ll22
3.4 ALGORITHMS
Overall, our framework Fed-LoGe consists of three critical stages: representation learning with SSE-C, global feature realignment, and local feature realignment, for the training of the shared backbone 0, global auxiliary classifier v. and K local classifiers (( respectively.
In the first stage, the server first constructs the SSE-C with Eq. 5 before the training and then dis- tributes it to all clients. Upon receiving SSE-C, each client fixes it as the classifier to train the backbone 6, global classifier v. and local classifier Φ alternately. Specifically, we update 0 with fixed e'SsE c- Subsequently, the 0 is frozen to update the global head şi and each local classifier dx- At the end of each round, the and s are aggregated at the server, while Φ is retained locally.
Global adaptive feature realignment (GA-FR) is performed in the second stage, where each class vector is redistributed by the server according to its individual norms, as outlined by Eq. Subse- quently, in the third phase, personalized adaptive feature realignment (LA-FR) for the class vectors of the global auxiliary head & is performed. Following LA-FR, local finetuning could be further conducted to boost the model performance. summary of Fed-LoGe is given in Algorithm I
EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Dataset, Models and Metrics: We consider image classification tasks for performance evalua- tion benchmark long-tailed datasets: CIFAR-10/100-1.T, ImageNet-LT, and iNaturalist-sser
7

---

Published as a conference paper at ICLR 2024
Dataset
ImageNet
Inaturalist
Method/Model
Many
Med
Few
GM
PM
Many
Med
Few
GM
PM
RedAvg
0.481
0.307
0.159
0.329
0.528
0.591
0.418
0.238
0.425
0.590
FedProx
0,493
0.318
0.180
0.343
0.500
0.525
0,484
0.223
0.432
0.596
FedBN
0.471
0.300
0.168
0.319
0.504
0.573
0.396
0.221
0.413
0.563
FedPer
0.653
0.638
FedRep
0,460
0.309
0.187
0.330
0.574
0.571
0,453
0.237
0.429
0.627
Ditto
0.492
0.319
0.176
0.342
0.674
0.598
0.452
0.243
0.437
0.584
FedROD
0.483
0.305
0.165
0.331
0,7033
0.585
0.416
0.243
0.421
0.699
FedBABU
0.443
0.240
0.055
0.230
0.425
0.561
0.401
0.199
0.377
0.696
FedETF
0,425
0.239
0.05
0.222
0.418
0.587
0,431
0.245
0.437
0.713
Ratio Loss
0.495
0.337
0.189
0.351
0.521
0.587
0.454
0.290
0.452
0.589
FedLoGe
0.430
0.373
0.285
0.356
0.726
0.519
0.508
0.473
0.503
0.759
Table 2: Test accuracies of our and SOTA methods on ImageNet-LT and iNaturalist- 160k with
diverse heterogeneous data settings.
160k Van Horn et al. 2018 The CIFAR-10/100-11 10/100-1TTT datasets are sampled into a long-tailed distribution employing exponential distribution governed by the Imbalance Factor (IF) in Cao iet al. (211). All experiments are conducted with non-IID data partitions, implemented by the Dirichlet distributions-bass
approach with parameter a to control the non-IIDness (Chen & Chao 2022). ResNet- 18 is trained over K 40 clients on CIFAR-IO-LT, while ResNet-34 and ResNet-50 are implemented on CIFAR- 100-TT and ImageNet-LT, respectively, with K 20 clients. The con- figurations for iNaturalist- 1 60k align with those utilized for ImageNet-LT. We use a 1. 0.5 and IF 50, 100 CIFAR- 10/110000---11 We use a 0.1, 0.5 in ImageNet-LT and iNaturalist, respectively.
A global balanced dataset is used for the calculation of test accuracy to evaluate global model (GM) performance. We also report the accuracy across many, medium, and few classes. The detailed categorization for many/med/few classes can be found in the Appendix.
For personalized model (PM) evaluation, we use local test accuracy, and the local test set is sampled from the global test set. Each local test set has an identical distribution to the local training set. The accuracy of the PM is the arithmetic mean of local test accuracy across all clients.
Compared Methods: In addition to FedAvg and FedProx (Li et al., 2020a) which are included for
ETF
SSE-C
GA-FR
LA-FR
GM
PM
0.3825
0.6421
types of -fate-of-theear baselines: (1) pFL methods, including FedBN (Li
reference, consider
0.4175
0,6865
0.4350
0.6868
al. 2021b), FedPer (Arivazhagan et al. 2019)
0.4175
0.7339
FedRep (Collins et al., 2021). Ditto (Li et al.
0.3988
0.7043
2021ap, and FedROD (Chen & Chao 2022). (2)
0.4350
0.7343
Federated (Long-tailed) Representation learning. in-
Table 3: Ablations of SSE-C and GLA-FR.
cluding FedBABU (Oh al., [2222) FecETF
et al. 2023) and Ratio Loss (Wang et al. 2021)
4.2 PERFORMANCE COMPARISON
For evaluation on CIFAR-IO-LT and 100AR-100-LT, FedLoGe consistently outperforms all base- lines over all settings, achieving the highest overall accuracies in all settings for both GM and PMs; see Tab. [ For ImageNet-LT and iNaturalist- 160k, Tab. 2 highlights that FedLoGe consistently surpasses all baselines, marking significant accuracy improvements, particularly in middle and tail classes. Overall, benefiting from the enhanced representation learning and classifier realignments, FedLoGe attains superior GM performance, attributed the SSE-C design for representation learn- ing, and simultaneously obtains impressive PM performance together with refined feature alignment.
4.3 ABLATION STUDY AND SENSITIVITY ANALYSIS
In the ablation study, we evaluate the SSE-C and GA-FR/LA- FR individual impacts with respect to both GM and PM performance, over
Ablations of SSE-C and GLA-FR:
(IF=100,
a = 0.5), as given in Tab. 5 The results lead to the following conclusions: Compared to dense ETF, SSE-C can train superior backbone, enhancing both GM and PM performance. GA-FR
8

---

Published as a conference paper at ICLR 2024
can be combined with any backbone boost GM performance, while LA-FR can be used with any backbone to boost PM performance. Employing SSE-C and GLA-FR simultaneously yields significant enhancements in both GM and PM performance.
Negligible and dominant features with SSE-C: We investigated the effects of pruning on different features over CIFAR- 100-LT (a=0.5, IF=100). We computed the class mean for each category. Within each class mean, we ordered the means and pruned the corresponding classifier vector weights in descending (from dom- inant features) and ascending order (from negligible features) with pruning ratios from 0 to 100%. After training with SSE-C, pruning dominant features drasti- cally decreases performance, while prun- ing negligible features barely affects per- formance, which is visualized in Fig. 3 (a, b). This observation indicates that SSE-C can adeptly learn more effective features while autonomously disregarding high-noise features.
05-
Naple
Pees
nee Derman
66
Prining Ratio
Pruting Ratio
(a)
(b)
Figure 3: (a): Pruning features from smaller means and larger means, respectively. Negligible features exert minor impact on model performance; (b): Pruning Ex- periments with SSE-C. The model optimally enhances the dominant features, rendering the impact of negligi- ble features imperceptible.
We evaluate the model performance with various values of (which is assigned as the norm of SSE-C in norm equal loss) over CIFAR-100-LT (IF=1000, 0.5). as depicted in Fig. (a). Our observations indicate that a smaller norm leads to reduced gradient during backpropagation, producing subpar performance. Conversely, larger norm enables faster convergence. For CIFAR--0/100-I.T, the optimal value is approximately 0.1, whereas, for ImageNet and iNaturalist, it is approximately 1.6.
Sensitivity analysis for y in SSE-C:
TettAcuacyy
0.4
CIFAR-10LLT
0444
CITAR-100-LE
Ten Accunny
85 o7 03 Test Aoceeeeceee
0-1
00
Ber
160
05
05 de de
Norm SSSCC
Ratio SSE
Sparse Ratio of
Figure 4: (a): The impact of varying norms y in SSE-C on model performance within CIFAR-100- LT: (b) The impact of sparse ratio on model performance in CIFAR-IO-LT, (c): The impact of sparse ratio 3 on model performance in CIFAR-IO0-LT.
Sensitivity analysis for Sparse Ratio B in SSE-C: The indicates the pruning proportion in SSE-C. We evaluate the performance with the sparsity from O to 90% at intervals of 10% on CIFAR10/100-LT (IF 100, = 0.5). shown Fig. (b) and (c), minor sparsification yields slight model performance enhancement, and the performance is obtained around 60% sparsity. Surprisingly, large sparsity ratio still remains superior compared to models without sparsification.
CONCLUSION
This paper presented FedLoGe, a model training framework that enhances the performance of both local and generic models Fed-LT settings in the unified perspective of neural collapse. The proposed framework is comprised of SSE-C, component developed inspired by the feature collapse phenomenon to enhance representation learning, and GLA-FR, which enables fast adaptive feature realignment for both global and local models. As result, FedLoGe attains significant performance gains over current methods in personalized and long-tai tail federated learning. Future research will explore adaptive sparsity and expand the framework diverse loss functions and tasks.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
This work is supported by the National Natural Science Foundation of China (Grant No. 62106222, No. 62201504), the Natural Science Foundation of Zhejiang Province, China No. LZ23F020008, No. LG/22F010001), Zhejiang Lab Open Research Project (No. K2022PD0AB05) and the Zhejiang University-Angelaiig Inc. R&D Center Intelligent Healthcare.
REFERENCES
Manoj Ghuhan Arivazhagan, Vinay Aggarwal, Aaditya Kumar Singh, and Sunav Choudhary. Fed- erated learning with personalization layers. arXiv preprint arXiv:1912.00811 2019.
John O Awoyemi, Adebayo O Adetummbi, and Samuel A Oluwadare. Credit card fraud detection using machine learning techniques: comparative analysis. In 2017 international conference on computing networking and informatics (ICCNI). pp. 1-9. IEEE, 2017.
Christopher Briggs, Zhong Fan, and Peter Andras. Federated learning with hierarchical clustering of local updates to improve training on non-iid data. In 2020 International Joint Conference on Neural Networks (IJCNN), pp. 1-9. IEEE, 2020.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with lahel-distribution-aaar margin loss. Advances in neural information processing systems, 32, 2019.
Hong- You Chen and Wei-Lun Chao. On bridging generic and personalized federated learning for image classification. In International Conference on Learning Representations, 2022. URL
Zhen Chen, Chen Yang, Meilu Zhu, Zhe Peng. and Yixuan Yuan. Personalized retrogrees- federated learning toward imbalanced medical data. IEEE Transactions on Medical Imaging, (12):3663-3674, 2022a.
Zihan Chen, Songshang Liu, Hualiang Wang, Howard H Yang, Tony QS Quek, and Zuozhu Liu. Towards federated long-tailed learning. arXiv preprint arXiv:2206. 14988, 2022b.
Zihan Chen, Howard Hao Yang, Tony Quek, and Kai Fong Ernest Chong. Spectral co-distillation for personalized federated learning. In Thirty-seventh Conference Neural Information Processing Systems, 2023.
Yen-Hsiu Chou, Shenda Hong, Chenxi Sun, Derun Cai, Moxian Song, and Hongyan Li. Grp- fed: Addressing client imbalance federated learning via global-regularized personalization. In Proceedings of the 2022 SIAM International Conference Data Mining (SDM), pp. 451-458. SIAM, 2022.
Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai. Exploiting shared repre- sentations for personalized federated learning. In International conference on machine learning, pp.
PMLR, 2021.
Rong Dai, Li Shen, Fengxiang He, Xinmei Tian, and Dacheng Tao. Dispfl: Towards communication efficient personalized federated learning via decentralized sparse training. arXiv preprint arXiv:2206.00183 2022.
Yutong Dai, Zeyuan Chen, Jumnan Li, Shelby Heinecke, Lichao Sun. and Ran Xu. Tackling data heterogeneity in federated learning with class prototypes. In Proceedings of the AAAI Conference Artificial Intelligence, volume 37, pp. 7314-7322, 2023.
Hien Dang, Tan Nguyen, Tho Tran, Hung Tran, and Nhat Ho. Neural collapse in deep linear net- work: From balanced imbalanced data. arXiv preprint arXiv:2301.004 2023.
Xolani Dastile, Turgay Celik, and Moshe Potsane. Statistical and machine learning models in credit scoring: A systematic literature survey. Applied Soft Computing, 91:1 91:106263, 2020.
10

---

Published as a conference paper at ICLR 2024
Yuyang Deng, Mohammad Mahdi Kamani, and Mehrdad Mahdavi. Adaptive personalized federated learning. arXiv preprint arXiv:2003. 2020.
Enmao Diao, Jie Ding, and Vahid Tarokh. Heterofl: Computation and communication efficient
federated learning for heterogeneous clients. arXiv preprint arXiv:2010.01266 2020.
Xin Dong, Sai Qian Zhang, Ang Li, and HT Kung. Spherefed: Hyperspherical federated learning.
In European Conference on Computer Vision, pp. 165-184. Springer, 2022.
Marawan Elbatel, Hualiang Wang, Robert Martí, Huazhu Fu, and Xiaomeng Li. Federated model aggregation via supervised priors for highly imbalanced medical image classification. arXiv preprint MtxXiv:3307.19599 2023.
Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Personalized federated learning with the- oretical guarantees: A model-agnostic meta-learning approach. Advances in Neural Information Processing Systems, 33:3557-3560 2020.
Cong Fang, Hangfeng He, Qi Long, and Weijie J Su. Exploring deep neural networks via layer- peeled model: Minority collapse in imbalanced training. Proceedings of the National Academy of Sciences, (18(43))e2103091 118, 2021.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019. URL https:// openreview
Avishek Ghosh, Jichan Chung, Dong Yin, and Kannan Ranchandran. An efficient framework for clustered federated learning. Advances in Neural Information Processing Systems, 33:19586- 19597, 2020.
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In Proceedings the fourteenth international conference on artificial intelligence and statistics, pp- 315-323. JMLR Workshop and Conference Proceedings, 2011.
Yangsibo Huang, Samyak Gupta, Zhao Song, Kai Li, and Sanjeev Arora. Evaluating gradient in- version attacks and defenses in federated learning. Advances in Neural Information Processing Systems, 14:7232-7241, 2021.
Wenlong Ji, Yiping Lu, Yiliang Zhang, Zhun Deng, and Weijie J Su. An unconstrained layer-peeled perspective on neural collapse. arXiv preprint 1000099 2021.
Peter Kairouz, Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Ad- vances and open problems federated learning. Foundations and Trends@ in Machine Learning. 14(1-2):1-21 2021.
Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition. arXiv preprint 1rXiv:1910.09217, 2019.
Byungju Kim and Junmo Kim. Adjusting decision boundary for class imbalanced learning. IEEE Access, 8:81674-81685, 2020.
Donghyun Kim, Kaihong Wang, Stan Sclaroff, and Kate Saenko. A broad study pre- pre-training for domain generalization and adaptation. In European Conference on Computer Vision, pp. 621- 638. Springer, 2022.
Geun Hyeong Lee and Soo- Yon- Shin. Federated learning on clinical benchmark data: performance assessment. Journal of medical Internet research, 22(10):e2089, 2020.
Daliang Li and Junpu Wang. Fedmd: Heterogenous federated learning via model distillation. arXiv
preprint arXiv: 1910.03581, 2019.
Mengke Li, Yiu-ming Cheung, and Yang Lu. Long-tailed visual recognition via gaussian clouded logit adjustment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2022a.
11

---

Published as a conference paper at ICLR 2024
Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. Federated learning: Challenges, methods, and future directions. IEEE signal processing magazine, 37(3):50 2020a.
Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. Ditto: Fair and robust federated learning through personalization. In International Conference on Machine Learning, pp. 6357- 6368. PMLR. 2021a.
Xiao Li, Sheng Liu, Jinxin Zhou, Xinyu Lu, Carlos Fernandez-Granda, Zhihui Zhu, and Qing Qu. Principled and efficient transfer learning of deep models via neural collapse. arXiv preprint arXiv:221..12206 2022b.
Xiaoxiao Li. Meirui Jiang, Xiaofei Zhang, Michael Kamp, and Qi Dou. Fed Federated learning on non- (ind features via local batch normalization. In International Conference on Learning Representations, 2021b. URL https:: wttr:iiewwweeeeeeeeeeeeeeeeeeeee weettet.ffftfee pdf www.::idd=--BB--- 6YEQUN0OICG
Yu Li, Tao Wang, Bingyi Kang, Sheng Tang, Chunfeng Wang. Jintao Li, and Jiashi Feng. Over- coming classifier imbalance for long-tail object detection with balanced group softmax. In Proceedings of the IEEE/CVP conference on computer vision and pattern recognition, pp. 10991-
confercce
11000, 2020b.
Zexi Li, Xinyi Shang, Rui Hc, Tao Lin, and Chao Wu. No fear of classifier biases: Neural collapse inspired federated learning with synthetic and fixed classifier. arXiv preprint arXiv:2303. 2023.
Ziwei Liu. Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X Yu. Large- scale long-tailed recognition in an open world. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2537-2546, 2019.
Jiawei Ma, Yulei Niu, Jincheng Xu, Shiyuan Huang, Guangxing Han, and Shih-Fu Chang, Digco: Discriminative geometry-aware learning for generalized few-shot object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3208-3218, 2023.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera Ar- cas. Communication-effiiinn learning of deep networks from decentralized data. In Artificial intelligence and statistics, pp. 2773-1283 PMLR, 2017.
Dustin G Mixon, Hans Parshall, and Jianzong Pi. Neural collapse with unconstrained features. arXiv preprint arXiv:2011.11615 2020.
Jachoon Oh, SangMook Kim, and Young Yun. FedBABU: Toward enhanced representation for
federated image classification. In International Conference Learning Representations, 2022.
URL
net ittpreeeeeieee.eee/oooroiaiiiii (or.mmmiidtaaaaoQoecsss
Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase deep learning training. Proceedings of the National Academy Sciences, 117(40): 24652 24663, 2020.
Krishna Pillutla, Kshitiz Malik, Abdel-Rahman Mohamed, Mike Rabbat, Maziar Sanjabi, and Lin Xiao. Federated learning with partial model personalization. In International Conference Machine Learning, pp. 17716-17758, PMLR, 2022.
Pinxin Qian, Yang Lu, and Hanzi Wang. Long-tailed federated learning via aggregated meta map- ping. In 2023 IEEE International Conference on Image Processing (ICIP), pp. 2010-2014. IEEE, 2023.
Felix Sattler, Klaus-Robert Müller, and Wojeiech Samek. Clustered federated learning: Model- agnostic distributed multitask optimization under privacy constraints. IEEE transactions on neural networks and learning systems, 3273):371-3322,, 2020.
Xinyi Shang, Yang Lu, Yiu-ming Cheung, and Hanzi Wang. Fedic: Federated learning non- iid and long-tailed data via calibrated distillation. In 2022 IEEE International Conference on Multimedia and Expo (ICME), pp. 1-6. IEEE, 2022a.
12

---

Published as a conference paper at ICLR 2024
Xinyi Shang, Yang Lu, Gang Huang, and Hanzi Wang- Federated learning on heteroge- neous and long-tailed data via classifier re- re-training with federated features. arXiv preprint arXiv:2204.13399 2022b.
Yiqing Shen, Yuyin Zhou, and Lequan Yu. Cd2-pfed: Cyclic distillation destillatioo-uuided channel decoupling for model personalization in federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10041-10050, 2022.
Zebang Shen, Juan Cervino, Hamed Hassani, and Alejandro Ribeiro. An agnostic approach to fed- erated learning with class imbalance. In International Conference Learning Representations, 2021.
Canh T Dinh, Nguyen Tran, and Josh Nguyen. Personalized federated learning with moreau en- velopes. Advances in Neural Information Processing Systems, 33:21394 21402,2105 2020.
Alysa Ziying Tan, Han Yu, Lizhen Cui, and Qiang Yang. Towards personalized federated learning. IEEE Transactions on Neural Networks and Learning Systems, 2022.
Jingru Tan, Xin Lu, Gang Zhang, Changqing Yin, and Quanquan Li. Equalization loss v2: A new gradient balance approach for long-tailed object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1685-1694 2021.
Christos Thrampoulidis, Ganesh Ramachandra Kini, Vala Vakilian, and Tina Behnia. Imbal- ance trouble: Revisiting neural-collapse geometry. Advances in Neural Information Processing Systems, 35:27225-2733 2022.
Tom Tirer and Joan Bruna. Extended unconstrained features model for exploring deep neural col- lapse. International Conference on Machine Learning, pp. 21478-21505 PMLR, 2022.
Grant Van Horn Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun. Alex Shepard, Hartwig Adam, Pictro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8769-8778, 2018.
Cristina Vasconcelos, Vighnesh Birodkar, and Vincent Dumoulin. Proper reuse of image classifica- tion features improves object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 13628-13637, 2022.
Huan Wang, Lijuan Wang, and Jun Shen. Logit calibration for non-iid and long-tailed data in fed- erated learning. In 2022 IEEE Intl Conf on Parallel & Distributed Processing with Applications, Big Data & Cloud Computing, Sustainable Computing & Communications, Social Computing & Networking (SSAABDDCouBSeciiCComSSSsssiiii
pp. 782-789. IEEE, 2022.
Lixu Wang, Shichao Xu, Xiao Wang, and Qi Zhu. Addressing class imbalance in federated learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 10165-10173, 2021.
Zhennan Wang, Canqun Xiang, Wenbin Zou, and Chen Xu. Mma regularization: Decorrelating weights of neural networks by maximizing the minimal angles. Advances in Neural Information Processing Systems, 33: 33:19099-19110, 2020.
Zikai Xiao, Zihan Chen, Songshang Liu, Hualiang Wang, YANG FENG, Jin Hao, Joey Tianyi Zhou, Jian Wu, Howard Hao Yang, and Zuozhu Liu. Fed-grab: Federated long-tailed learning with self-adjusting gradient balancer. In seirth- Conference on Neural Information Processing Systems, 2023.
Miao Yang, Ximin Wang, Hongbin Zhu, Haifeng Wang. and Hua Qian. Federated learning with class imbalance reduction. In 2021 29th European Signal Processing Conference (EUSIPCO), pp. 2174-2178 IEEE, 2021.
Wenkai Yang, Deli Chen, Hao Zhao, Fandong Meng, Jie Zhou, and Xu Sun. Integrating local real data with global gradient prototypes for classifier re-balancing in federated long-tain tailed learning. arXiv preprint arXiv:2301. 10394, 2023a.
13

---

Published as a conference paper at ICLR 2024
Yibo Yang, Shixiang Chen, Xiangtai Li, Liang Xie, Zhouchen Lin, and Dacheng Tao. Inducing
neural collapse in imbalanced learning: Do we really need a learnable classifier at the end of deep neural network? Advances in Neural Information Processing Systems, 35:37991-38002, 2022.
Yibo Yang, Haobo Yuan, Xiangtai Li, Zhouchen Lin, Philip Tor, and Dacheng Tao. Neural col- lapse inspired feature-classffier alignment for few-shot class incremental learning. arXiv preprint arXiv:2302.03004 2023b.
Can Yaras, Peng Wang, Zhihui Zhu, Laura Balzano, and Qing Qu. Neural collapse with normalized features: geometric analysis over the riemannian manifold. Advances in neural information processing systems, 35:11547-11560, 2022.
Jianyi Zhang, Ang Li, Minxue Tang, Jingwei Sun, Xiang Chen, Fan Zhang, Changyou Chen, Yiran Chen, and Hai Li. Fed-cbs: heterogeneity- client sampling mechanism for federated learning via class-imbalance reduction. In International Conference on Machine Learning, pp- 41354 41381. PMLR, 2023.
Jie Zhang, Zhiqi Li, Bo Li, Jianghe Xu, Shuang Wu, Shouhong Ding, and Chao Wu. Federated learn- ing with label distribution skew via logits calibration. In International Conference on Machine Learning, pp. 2631 1-26329. PMLR, 2022a.
Xu Zhang, Yinchuan Li, Wenpeng Li, Kaiyang Guo, and Yunfeng Shao. Personalized federated learning via variational bayesian inference. In International Conference on Machine Learning, pp. 66293-26310 10. PMLR, 2022b.
Zhisheng Zhong, Jiequan Cui, Yibo Yang, Xiaoyang Wu, Xiaojuan Qi. Xiangyu Zhang, and Jiaya Jia. Understanding imbalanced semantic segmentation through neural collapse. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19550-19560, 2023.
Zhihui Zhu, Tianyu Ding, Jinxin Zhou, Xiao Li, Chong You, Jeremias Sulam, and Qing Qu. A ge- ometric analysis of neural collapse with unconstrained features. Advances in Neural Information Processing Systems, 34:29820-29834 2021.
A APPENDIX
A.1 DETAILED SETUP
Dataset, Models and Metrics: We consider image classification tasks for performance evaluation on benchmark long-tailed datasets:
ImageNet-LT, and iNaturalist- 160k.
The CIFAR-100 10/100-LT datasets sampled into long-tailed distribution employing an exponen- tial distribution governed by Imbalance Factor (IF) in Cao et al. (2019), calculated over the compre- hensive dataset D. Consistency is maintained by utilizing configurations from Liu et al. (2019) for ImageNet-LT, where the number of images per class varies between and 1280. Additionally. to evaluate performance with real- world data, the study includes experiments on iNaturalist- User- User-160kk, 160k, comprising 160k examples across 1023 species classes, sampled from INtturalist-2017 in Van Horn al. 2018).
All experiments employ non-lID data partitions, achieved through Dirichlet distributions. The con- centration parameter a serves regulate the identicalness of local data distributions among all clients. A ResNet-18 8 is trained over K = 40 clients on CIFAR-10-L3, while ResNet-34 and ResNet- 50 are implemented on CIFAR-100-LT and ImageNet-LT, respectively, with N = 20 clients. The configurations for iNaturalist- 160k align with those utilized for ImageNet-LT.
The balanced dataset is utilized for the evaluation of global model performance. The global accuracy is calculated the ratio of the number of correct predictions to the total number of predictions. We also report the accuracy across many, medium, and few classes. The categorization process unfolds follows: Initially, all classes sorted based on the number of data samples in descending order. Subsequently, two thresholds (c.g., 75%, 95%) are set to segregate the classes into head classes, middle classes, and tail classes.
14

---

Published as a conference paper at ICLR 2024
For CIFAR10/100- datasets, engaged 40 clients with full participation in each training round, setting the number of local epochs to 5. For ImageNeviNaturrliss, the experiments involved 20 clients, with 40% participation rate in each training round, and the number of local epochs set to 3.
When assessing personalized models, the test set is sampled from the global balanced dataset by local distribution. We ensure that the distribution of the local train set and the local test set is perfectly aligned. The accuracy of local models is the arithmetic mean of all clients.
Ours
FedROD
Test Accuracy
0.72
0.68
10
20
30
50
Epochs
Figure 5: Accuracy local training after GLA-FR.
Training Settings: We train the generic model with T 500 rounds via applying SGD optimizer for local training in all experiments unless otherwise stated. We use a = 1, 0.5 and IF 50, 100 in -01A--001000-T. We use a 0.1, 0.5 in ImageNet-LT and iNaturalist, respectively.
A.2 TABLE OF NOTATIONS
Please refer to Table 4 for the table of notations used throughout this paper.
A.3 DETAILES OF FEATURE DEGENERATION
Given that n, is the sample number of class c. fe is the class mean class c, and hi,e is the feature of class e on sample i, the features hi.e. will collapse to the within-class mean Pe 1 Σ hi.c- n.
This indicates that the covariance n. 1 (h4,c-Ho)(h.. me)" will converge 0. We investigate whether h4 Me tends toward zero when training with fixed ETF classifier, as shown in Fig. [ The steps to obtain the data in Fig. aree 1. Select class c. 2. During global test, compute all sample features hic. representing the number of features. Use hr [d]] denote the d feature in hi,e- 3. Compute the mean and relative variance of he, sorting them in descending order of the mean, and accordingly adjust the variance positions. It becomes evident that not all features, hi,eld pe[d], converge to zero, and smaller [d] values exhibit greater convergence noise. The process illustrated in Fig. (b) follows the same steps as Fig. [ (a), but with grey vertical shadows for the sparsely positioned features. It can be observed that sparse ETF training effectively squeezes poor-quality and noisy features into these sparsely allocated positions.
A.4 COMPUTATIONAL COST
We evaluate the computational expense of Fed-LoGe across its three stages. In the initial repre- sentation stage, the server tasked with constructing the SSE-C. Setting the learning rate 0.0001 and executing 10,000 optimization steps, we conducted the experiments three times on the Py Torch platform utilizing the NVIDIA GeForce RTX 3090; the average cost is 11 minutes and 52 seconds.
15

---

Published as a conference paper at ICLR 2024
Notation
Description
K
Number of clients in the federated learning system. Parameters of the generic model, consisting of and v. Parameters of the 1thh personalized model for client Feature extractor parameterized by 0.
wk
f(, 0)
h
Features transformed from input by feature extractor f.
g(h,
Generic classifier mapping features / to output labels. Personalized classifier for client
g(,, o&)
Personalized classifier parameters for client k.
D
Global long-tailes dataset.
DR
Local dataset of client k. Number of training rounds.
T
Class index.
Classifier vector corresponding to class Features corresponding to class
he
s
Sparse indicator matrix.
y
Predetermined l norm value for class vectors.
CSSE-C
Loss function for Static Sparse Equiangular Tight Frame Classifier (SSE-C),
Cnorm Gangle
Loss functions for optimizing the geometric structure of SSE-C. Sample number of class c.
The
He
Class mean of features for class c.
hi,
Feature of class e for sample i.
Table 4: Summary of notations used the paper.
Method
Time Cost/Round
FedAvg
3min11s
FedProx
5min24s
FedRep
4min07s
FedBABU 3min14s
FedLoGe
3min19s
Table 5: Computational cost of Fed-LoGe.
A.5 LOCAL FINETUNE FURTHER IMPROVE PERFORMANCE
After the personalized feature realignment, we proceed with local training without any aggregation. On CIFARI00-LT with IF 100 and 0.5, the accuracy consistently increases with additional local epochs, reaching stability at 15 epochs, as illustrated in Fig. 5
A.6 T-SNE VISCALIZATION OF CLASS MEANS
In this study, we compare features from models trained with SSE-C C (sparse) and ETF (dense) using I-SNE, a common method for reducing dimensions, shown Fig. 6 This helps us see how different training methods affect feature distribution. We notice that with SSE-C. the average features of each class are evenly spread out and quite similar in distance to each other. On the other hand, the model trained with ETF has many class averages very close even overlapping, making the overall spread more clustered and the model more likely to make mistakes.
A.7 NORM AND ANGLES OF SSE-C
We have individually computed the statistical measures (mean of norm, variance of norm, mean of mutual angles, and variance of mutual angles) of the classifier vectors within SSE-C under varying
16

---

Published as a conference paper at ICLR 2024
I-SNE Visualization of Class Means (ETF)
I-SNE Visualization of Class Mcans (SSE-C)
I-SNE component
I-SNE compument
I-SNE component
t-SNE component
(b)
Figure 6: I-SNE visualization of class means of ETF (dense) and SSE-C (sparse)
Sparse Ratio
Norm
Mean of Norm
Var of Norm
Mean of Angle
Var of Angle
0.6
0.5
0.50
3,55e-12
90.06
0.09
0.6
1.0
1.00
4.75e-11
90.06
0.66
0.6
1.5
1.50
1.71e-11
90.04
1.21
0,4
0.5
0.50
1.10e-11
90.06
0.03
0.4
1.0
1.00
7.14e-12
90.06
0.15
0.4
1.5
1.50
3.5e-11
90.06
0.79
Table 6: Construct measures of SSE-C
Model
PM(Avg)
PM(Macro)
PM(Weighted)
LA-LR (real distribution)
0.6200
0.2680
0.5604
LA-LR (norm)
0.7273
0.5316
0.7129
Table 7: Local adaptive feature alignment of real distribution and personalized norm
sparse ratios 3 and assigned norms y, shown in Table It observable that the constructed SSE- nearly fulfills the properties of an Equiangular Tight Frame (ETF): equal norms, and maximized, equal mutual angles. Different sparse ratios and assigned norms do not significantly influence the final construction.
A.8 LOCAL NORM OR LOCAL DISTRIBUTION FOR LA-FR
Upon the backbone 6 trained through USSE-CC we proceed with local adaptive feature realignment. In our methodology, local norm is employed to align features with personalized preference. It is intuitive to consider that real local distribution more representative of the model's preference, and moreover, the local distribution is free for utilization without any computation. Consequently, we explore whether local norm or local distribution yields superior performance.
On CIFAR 100-LT with 0.5 and IF 100, realign personalized models employing both norm and distribution, calculating the arithmetic mean, macro average, and weighted average of all clients' performance. The results be seen in Tab. 7 We discern that utilizing norm for realignment is more effective than employing the real distribution. One rationale is that the norm more aptly signifies the model's cognitive capacity A encompasses more samples than class B. B is more prone to misprediction owing to its high resemblance with other classes. In this scenario, realignment ought to adhere model cognition, fortifying class A over B.
the dataset. For instance, although class
A.9 PROJECTION LAYER AND DOT-REGRESSION Loss IS NOT NECESSARY FOR FED-LOGEE
In previous work, both the projection layer and dot-regression loss are deemed essential for training the backhone (Li al. (2023): Yang et al. (2023b) The projection layer, being dense structure,
17

---

Published as a conference paper at ICLR 2024
Method
Many
Med
Few
All
ETF
0.6904
0.4751
0.1820
0.3825
ETF+DR Loss
0.6882
0.4607
0.2059
0.3932
ETF+Proj+DR Loss
0.6909
0.5011
0,2310
0.4109
ETT+Proj+DR Loss+Sparse
0.7013
0,5142
0.2479
0,4237
FedL
0.7137
0.4989
0.2179
0.4249
FedLoGe+Proj
0.7014
0,4642
0.1887
0,4004
Table 8: Ablations of projection layer and dot-regression loss.
Initialization Method
Many
Med
Few
All
Xavier
0.6793
0,4204
0.1823
0.3784
Sparse Xavier
0.7089
0,4977
0.2189
0.4237
Gassian
0.6167
0.3858
0.1572
0.3407
Sparse Gassian
0.7119
0.4969
0.2117
0.4209
Uniform
0.6119
0.3631
0.1638
0.3366
Sparse Uniform
0.7141
0.4927
0.2174
0.4231
Orthogonal
0.6844
0,4446
0.1868
0.3882
Sparse Orthogonal
0.7022
0,4846
0,2060
0.4124
Kaiming Uniform
0.6933
0.4392
0.1881
0.3898
Sparse Kaiming Uniform
0.7130
0.4946
0.2168
0.4230
Table 9: Sparsification with different initialization of frozen classifiers
necessitates substantial computational cost. However, owing to the advanced design of SSE-C, Fed- LoGe does not depend on the projection layer and dot-regression loss. We undertake experiments investigate how the dot-regression loss and projection layer influence model performance based on ETF and Fedl LoGe, with the results documented in Tab We ascertain that solely through the sparsity design of SSE-C, performance surpassing that of employing both the projection layer and dot-regression loss can be achieved.
A.10 DIFFERENCE INIIIALIZATION METHOOSS OF FIXED CLASSIFIER
To further explore the effectiveness of sparsification for fixed classifier, we have selected various classifier initialization methods (Xavier, Gaussian, Uniform, Orthogonal, Kaiming Uniform), and subsequently applied sparsification to classifiers constructed with these initializations. The results are documented in Tab. 9 It is observable that irrespective of the initialization method employed, post-sparsification and subsequent training of the fixed classifier yield exceptionally favorable per- formance.
A.11 DISCUSSION OF PRIVACY
In the entire training process of FedLoGe, no extra information compared with FedAvg is transmit- ted: 1. During the 'Representation Learning with SSE-C" phase, FedLoGe transmits local model weights FedAvg necessitates. In the 'Feature Realignment' phase, the local models are already downloaded for norm analysis, and there no further information-sharing operations required.
As the norm analysis is performed on the client side or global side, there would still be a few privacy concerns. However, noteworthy that the potential privacy issue exists in the general FL frameworks rather than specific to our proposed Global and Local Adaptive Feature Realignment (GLA-FR). For instance, gradient inversion Huang et al. (2021) can pose a threat to almost all gradient transmission- based FL methods without any privacy-pree peuuuyyyreeervttion techniques. As the privacy issue of the FL framework is beyond the scope of this work.
18

---

Published as a conference paper at ICLR 2024
A.12 CONVERGENCE EXPERIMENTS
We have conducted experimental investigations into the convergence. Specifically, explored the number of rounds required to reach an accuracy of 0.33 on CIFARI00 with an imbalance factor (IF) of 100 and 0.5, documented in the table below. It is evident that the convergence rate of Lo almost on par with FedAvg and surpasses the current state-of-the-ar (SOTA) methods.
Methods
Convergence Round
FedETF
83
FedROD
68
Fod (Ours)
62
Table 10: Comparison of Convergence Rounds for Different Methods.
A. 13 COMPARISON OF EXISTING METHOOS
We are the first to achieve joint global and local training Federated long-tailed learning from the perspective of neural collapse. In the introduction, analyzed the improvements of FedLoGe over some existing methods, and here we further provide the discussion:
First, most existing methods in personalized federated learning (Collins al. 2021 Li et al. (202aa; Dinh et al. (2020); Zhang et al.] (2022b); Dai et al. (022)) utilize the global model regularize construct personalized models but do not evaluate them in generic setup, failing to yield strong global model without long-tail bias. This makes them unsuitable as strong incentive
for attracting new clients. In contrast, our approach learns models that excel in both personalized (GA-FR) and generic (FA-LR) setups without compromising either.
Second, existing methods in federated long-tailed learning (Shang et al. [002b)); Yang et al. 2023a); Shang et al. (2022a); Wang et al. (2022); Qian et al. (2023)), focusing only on calibrating long-tail bias for single global model, overlook the aspect representation learning (backbone) under fed- erated long-tailed settings and fail to provide personalized models. For instance, CreFF involves post-classifier retraining but does not advance in boosting the backbone and does not offer person- alized models. In contrast, Our FedLoGe achieves strong representation learning with SSE-C to support both global model and local models.
Second, existing work involving fixed classifiers has not delved deeply into the analysis of the ini- tialization of these fixed classifiers. FedBABU (Oh et al. (2022)) ). for instance, merely employs randomly initialized fixed linear layer. For FedETF (Li et al. (2023)), the Vallina fixed ETF classifier has suffered from feature degeneration, characterized by significant fluctuations between features and the class mean, while FedLoGe alleviates this issue by filtering out noisy smaller mean features (Fig. D. making features more concise and effective (Fig. 3 (a.b) and Fig. (6). Besides, Foee is less computationally expensive and easier adapt existing algorithms. DR loss is necessary for FedETF, which means some modifications are required when integrating with existing algorithms. Besides, FedETF needs projection layer for optimization. For ResNet18 with four blocks, 512x512 projection layer would be approximately IMB, almost doubling the parameter size of Block which is 0.56MB.
A. 14 IMPACT OF CLIENT NUMBERS AND PARTICIPATION RATES
To investigate the adaptability and efficiency of FedLoGGe varied federated learning environments, designed experiments focusing two main variables: the number of clients and their participa- tion rates. The experiments spanned across CIFARI0/100-LT and ImageNet/iNat datasets, tailored to simulate environments ranging from full participation to more realistic, sporadic engagement.
For CIFAR10/100-LT,, we utilized a setup involving 40 clients with 100% participation and local epochs per training round. Conversely, ImageNet/iNat experiments were conducted with 20 clients at a 40% participation rate and 3 local epochs, introducing scenario with reduced participation and increased data complexity.
19

---

Published as a conference paper at ICLR 2024
Additionally, to explore the behavior of FedLoGe under stringent conditions, conducted experiments on CIFAR100-LT with an imbalance factor of 100 and a heterogeneity coefficient of 0.5, decreasing the participation rate to 30%. This setup aimed test FedLoGe's resilience and per- sonalized performance against data imbalance and client heterogeneity. Extending our exploration cross-device FL. setting, the client count was increased to 100 with 30% participation rate, challenging FedLoGe with a highly distributed environment and testing its scalability and perfor- mance.
The experimental outcomes, detailed in Table I1 highlight FedLoGe's superior performance across different settings, particularly showcasing its strengths scenarios with varied client participation rates and numbers.
Method
40 clients (100% participate)
40 clients (30%)
100 clients (30%)
FedAvg
GM0.38 11.38888/M0002111
0.3746/0.6205
0.3718/0.6386
FedRoD
0.3919/0.6911
0.3758/0.7155
0.3727/0.7666
FedETF
0.3825/0.644 64221
0.4180/0.700
0.3847/0.788
FedLoGe
0.4233/0.7285
0.4206/0.7524
0.4159/0.800
Table 11: Impact of client diversity and participation rates FedLoGe performance compared to other FL algorithms.
These results confirm FedLoGe's robustness and adaptability, underlining its potential for broad applicability in federated learning scenarios characterized by varying degrees of client engagement and diversity.
A. EFFECTIVENESS SSE-C
We opted for a learning rate of 0.0001 with 10,000 optimization steps on ImageNet-LT with 1000 classes. The order of magnitude for the norm variance is between 10 and 10 9 For the an- gles across each pair of classifier vectors, we visualized the cosine similarity in SSE-C in Fig. 7 indicating that our optimized SSE-C is almost identical the dense ETF.
Classes
10
Classes
Figure 7: The heatmap of cosine similarity between classifier vectors in SSE-C.
20


Reviews:
[{"review_id": "LjvBCJ1Fk1", "reviewer": "Reviewer_Nmh4", "summary": "This paper addresses Federated Learning with global long-tailedness. The proposed method is inspired by the neural collapse idea and contains three steps: 1) global representation learning with SSE-C classification head, and 2) & 3) global and local feature realignment.  In step 1, a classifier head is learned with sparsity while maintaining the Equiangular Tight Frame (ETF) properties, which is then broadcast to clients for learning backbone $\\theta$ and personalized classifier head $\\psi$. In step 2 & 3, the backbone $\\theta$ and global classifier $\\psi$ and local classifier $\\phi$ is respectively updated and realigned. Empirical results on a few benchmarks show the effectiveness of the proposed approach over prior work on data heterogeneous FL or Long-tailed FL.", "questions": "How to optimize Eq (4) that contains a maximization over $j$? \n\nWhat is the purpose of introducing sparsity to the SSE-C classifier, besides its empirical effectiveness?\n\nWhy do we need two classifier heads  $\\phi_k$ and $\\psi_k$ for each of the clients $k$ instead of just one?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "\\+ This paper tackles a crucial and intriguing problem in federated learning with global long-tailedness.\n\n\\+ The key idea of using ETF to build a classifier as regularized consensus is well-motivated.\n\n\\+ Experiments and ablation studies are well-designed and comprehensively conducted.", "weaknesses": "\\- Too many technical details are missing. For example, it is vague to me why performing personal classifier realignment in Eq (7) helps in tackling local data heterogeneity. \n\n\\- Authors need to provide detailed motivation of each step in the proposed algorithm, such as the motivation of keeping two classifier heads for clients: $\\phi_k$ and $\\psi_k$, and realignment of the global classifier $\\psi$ in Eq 6.\n\n\\- Writing: Certain citation formats need to be fixed; Figure 1 is difficult to comprehend.\n\n\\- Missing discussion and comparison with related work on long-tailed FL. Authors should specify the technical contributions of the proposed method over other prior work, especially FedETF, which shows much resemblance to the proposed work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "acsSMjGE9o", "reviewer": "Reviewer_YHb3", "summary": "The paper considers federated learning with long-tailed global distribution and non-iid client distributions. the authors propose a Federated Local and Generic Model Training (FedLoGe) to train a model that can perform well both on long-tailed global distribution and local client distributions. First, the proposed Static Sparse Equiangular Tight Frame Classifier fosters the acquisition of potent data representations. Second, the Global and Local Adaptive Feature Realignment (GLA-FR) is used to align global features with client preferences.", "questions": "* Please provide clear motivation for the generic-local setting. When the personalized federated learning targets training models personalized for each client, is it necessary to consider a model adapted for global or generic distribution? In personalized federated learning, the assumption is that each client has its own data distribution and it aims to learn a model adapted for the specific distribution. If the global or generic distribution is considered, which client will be expected to use the model?\n* How do the mean and variance of features relate to the feature collapse in the quoted content?\n> However, preliminary experiments benchmarking ETF with CIFAR-100 in Fed-LT suggest that only a few features have relatively large means, while most of the small-mean features are contaminated by severe noise, as shown in Fig. 1(a). Such observations are inconsistent with the feature collapse property, and we coin it as feature degeneration.\n\n=== after rebuttal ===\nThe authors' responses have address my concerns.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "* (originality) The authors proposed a novel method by neural collapse and realignment of local and global classifiers. Though neural collapse has been used for handling data heterogeneity, the realignment is novel to address the local-generic problem.\n* (significance) The proposed method can significantly outperform baselines in the concerned settings.\n* (clarity) The figure 2 is appreciated for clarifying the method.\n* (quality) The experiments are well designed for the problem setting. Multiple baselines are compared to demonstrate the effectiveness of the proposed method both in boosting local and generic performance.", "weaknesses": "* (quality) The generic-local setting looks self-contradictive to me. When the personalized federated learning targets training models personalized for each client, is it necessary to consider a model adapted for global or generic distribution? In personalized federated learning, the assumption is that each client has its own data distribution and it aims to learn a model adapted for the specific distribution. If the global or generic distribution is considered, which client will be expected to use the model?\n* (significance) Because of my concerns about the problem setting, I am afraid that the proposed method may have a limited impact on a few specific problems.\n* (clarity) The technical motivation is not clear. The neural collapse has been used for handling data heterogeneity and classifiers should be fixed by the neural collapse theorem. It is unclear why the realignment is still needed. Why the fixed classifier cannot be used for all clients?\n* (clarity) The paper is hard to follow. Specifically, the motivation experiments in the introduction are hard to understand. Why the mean and variance of class means are evaluated in Figure 1? I cannot follow the logic in the discussion quoted below.\n> However, preliminary experiments benchmarking ETF with CIFAR-100 in Fed-LT suggest that only a few features have relatively large means, while most of the small-mean features are contaminated by severe noise, as shown in Fig. 1(a). Such observations are inconsistent with the feature collapse property, and we coin it as feature degeneration.\n\n  How do the mean and variance of features relate to the feature collapse?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yvLDlkPzKY", "reviewer": "Reviewer_tSL5", "summary": "This paper presented a model training framework that enhances the performance of both local and generic models in Fed-LT settings in the unified perspective of neural collapse. The proposed framework is comprised of SSE-C, a component developed inspired by the feature collapse phenomenon to enhance representation learning, and GLA-FR, which enables fast adaptive feature realignment for both global and local models. As a result, the proposed method attains significant performance gains over current methods in personalized and long-tail federated learning.", "questions": "Please see the Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "S1. This work focuses on addressing the data imbalance issue to simultaneously enhance the efficacy of the generic global model and its performance at the local level, which is an interesting topic.\n\nS2. This paper is well-written and has a good presentation.\n\nS3. The proposed SSE-C can address the problem of feature degeneration well, which is a promising finding.", "weaknesses": "W1. Convergence analysis is missing, which is very important for the optimization process of federated learning. Please analyze it from the experimental and theoretical point of view.\n\nW2. Lack of discussion about privacy. Federated learning is proposed to protect the privacy of the client, but the method in this paper has the risk of gradient disclosure, please add the discussion of privacy in this work.\n\nW3. The communication overhead of the model seems to be very large, which will restrict the practical application. Please increase the experiment and analysis of the communication overhead.\n\nW4. The related work is simply a list of existing methods. Please add a discussion of the differences between this work and previous work to further clarify the contribution of this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gn3jzbXr59", "reviewer": "Reviewer_YQxK", "summary": "This paper aims to address the long-tailed data in FL. The authors propose a new method called FedLoGe to improve both local and global model performance of FL under this scenario. FedLoGe introduces a Static Sparse Equiangular Tight Frame Classifier (SSE-C) to enhance data representations. It also contains feature realignment at both global and local levels.", "questions": "- How many clients participate in training each round?\n- Does the paper focus on cross-silo FL, cross-device FL, or both?\n- Why the proposed method can improve performance on FL with long-tailed problem?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The long-tailed problem in FL is an emerging problem that is worth studying.\n- The paper is generally well structured.\n- Figure 2 provides a good illustration of the proposed method.\n- The proposed method seems to work and the experiments demonstrate the effectiveness of the proposed method.\n- Evaluation are conducted on various backbones and different scales of datasets.", "weaknesses": "- The content of the paper is not very easy to follow. For example, the message from Figure 1 is a bit hard to interpret. In the first paragraph of the third page (where Figure 1 (a) is explained), it is not quite clear what large mean and small mean imply.\n- Notations seem to be inconsistent, the notation for the number of clients is K in Section 3, but N in Section 4.\n- Probably a typo: “…. personalized classifier by multiple the norm of … “ Should it be “by multiplying… “?\n- The method of fixing the classifier and backbone alternatively for training is proposed in previous papers on personalized FL.\n- Some settings are not clear: e.g., the number of clients participating in training each round and the number of local epochs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)