SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Theorem 1 (Global Convergence). If  $F_{1},\ldots ,F_{N}$  are all  $L$  -smooth,  $\mu$  -strongly convex, and the variance and norm of  $\nabla F_1,\dots,\nabla F_N$  are bounded by  $\sigma$  and  $G$  . Choose  $\kappa = L / \mu$  and  $\gamma = \max \{8\kappa ,E\}$  for all classes  $c$  and sample  $i$  , expected global representation by cross-entropy loss will converge to:
$$
\mathbb {E} \left[ \log \frac {\left(\mathbf {W} ^ {L , *}\right) ^ {T} h _ {c} ^ {i , *}}{\left(\mathbf {W} _ {g} ^ {L}\right) ^ {T} h _ {c} ^ {i}} \right] \leq \frac {\kappa}{\gamma + T - 1} \left(\frac {2 B}{\mu} + \frac {\mu \gamma}{2} \mathbb {E} | | \mathbf {W} ^ {1} - \mathbf {W} ^ {*} | | ^ {2}\right),
$$
where in FedGELA,  $B = \sum_{k=1}^{N}(p_k^2\sigma^2 + p_k||\Phi_k\mathbf{W}^L - \mathbf{W}^L||) + 6L\Gamma_1 + 8(E-1)^2G^2$ . Since  $\mathbf{W}^L = \mathbf{W}^{L,*}$  and  $(\mathbf{W}^{L,*})^T h_{c_i}^{i,*} \geq \mathbb{E}[(\mathbf{W}^L)^T h_{c_i}^i]$ ,  $h_{c_i}^i$  will converge to  $h_{c_i}^{i,*}$ .
In Theorem 1, the variable  $B$  represents the impact of algorithmic convergence  $(p_k^2\sigma^2)$ , non-iid data distribution  $(6L\Gamma_1)$ , and stochastic optimization  $(8(E - 1)^2 G^2)$ . The only difference between FedAvg, FedGE, and our FedGELA lies in the value of  $B$  while others are kept the same. FedGE and FedGELA have a smaller  $G$  compared to FedAvg because they employ a fixed ETF classifier that is predefined as optimal. FedGELA introduces a minor additional overhead  $(p_k||\Phi_k\mathbf{W}^L -\mathbf{W}^L ||)$  on the global convergence of FedGE due to the incorporation of local adaptation to ETFs. The cost might be negligible, as  $\sigma$ ,  $G$ , and  $\Gamma_{1}$  are defined on the whole model weights while  $p_k\| \Phi_k\mathbf{W}^L -\mathbf{W}^L\|$  is defined on the classifier. To verify this, we conduct experiments in Figure 3(a), and as can be seen, FedGE and FedGELA have similar quicker speeds and larger classification angles than FedAvg.
Theorem 2 (Local Convergence). If  $F_{1}, \ldots, F_{N}$  are  $L$ -smooth, variance and norm of their gradients are bounded by  $\sigma$  and  $G$ , and the heterogeneity is bounded by  $\Gamma_{1}$ , clients' expected local loss satisfies:
$$
\mathbb {E} \left[ F _ {k} ^ {(t + 1) E} \right] \leqslant F _ {k} ^ {t E} + \frac {L E \eta_ {t} ^ {2}}{2} \sigma^ {2} + \Gamma_ {1} - A,
$$
where in FedGELA,  $A = (\eta_t - \frac{L}{2}\eta_t^2)EG^2 - L\left\| \Phi_k\mathbf{W}^L - \mathbf{W}^L\right\|$ , which means if  $A - \frac{G^4}{LE(G^2 + \sigma^2)} \leq 0$ , there exist learning rate  $\eta_t$  making the expected local loss decreasing and converging.
In Theorem 2, only "A" is different on the convergence among FedAvg, FedGE, and FedGELA. Fixing the classifier as ETF and adapting the local classifier will introduce smaller G and additional cost of  $L\left\| \Phi_k\mathbf{W}^L -\mathbf{W}^L\right\|$  respectively, which might limit the speed of local convergence. However, FedGELA might reach better local optimal by adapting the feature structure. As illustrated in Figure 3 (c), the adapted structure expands the decision boundaries of existing major classes and better utilizes the feature space wasted by missing classes. To verify this, in Figure 3(b), we record the averaged angles between the existing class means during the local training. It can be seen that FedGELA converges to a much larger angle than both FedAvg and FedGE, which suits our expectations. More angle results can be seen in Figure 5.
Datasets. We adopt three popular benchmark datasets SVHN [23], CIFAR10/100 [16] in federated learning. As for data splitting, we utilize Dirichlet Distribution (Dir  $(\beta)$ ,  $\beta = \{10000, 0.5, 0.2, 0.1\}$ ) to simulate the situations of independently identical distribution and different levels of PCDD. Besides, one standard real-world PCDD dataset, Fed-ISIC2019 [4, 7, 34, 35] is used, and we follow the setting in the Flamby benchmark [34]. Please refer to Appendix C for more details.
Metrics. Denote PA as the personal accuracy, which is the mean of the accuracy computed on each client test dataset, and GA as the generic accuracy on global test dataset (mixed clients' test datasets). Since there is no global model in P-FL methods, we calculate GA of them as the averaged accuracy of all best local models on global test dataset, which is the same as FedRod [3]. Regarding PA, we record the best results of personal models for P-FL methods while for G-FL methods we fine-tune the best global model in 10 epochs and record the averaged accuracy on all client test datasets. For FedRod and FedGELA, we can directly record the GA and PA (without fine-tuning) during training.
Implementation. We compare FedGELA with FedAvg, FedRod [3], multiple state-of-the-art methods in G-FL (FedRS [21], MOON [17], FedProx [19], FedGen [54] and FedLC [47]) and in P-FL (FedRep [6], FedProto [33] and FedBABU [25]). For SVHN, CIFAR10, and CIFAR100, we adopt a commonly used ResNet18 [8, 17, 47, 48, 52] with one FC layer as the backbone, followed by a layer of classifier. FedGELA replaces the classifier as a simple ETF. We use SGD with learning rate 0.01, weight decay  $10^{-4}$ , and momentum 0.9. The batch size is set as 100 and the local updates are set as 10 epochs for all approaches. As for method-specific hyper-parameters like the proximal term in FedProx, we tune it carefully. In our method, there are  $E_{W}$  and  $E_{H}$  need to set, we normalize features with length  $1(E_{H} = 1)$  and only tune the length scaling of classifier  $(E_W)$ . All methods are implemented by PyTorch [27] with NVIDIA GeForce RTX 3090. See detailed information in Appendix C.
5.2 Performance of FedGELA
In this part, we compare FedGELA with FedAvg, FedRod, three SOTA methods of P-FL (FedRep, FedProto, and FedBABU), four SOTA methods of G-FL (FedProx, MOON, FedRS, FedLC and FedGen) on different aspects including the scale of clients, the level of PCDD, straggler situations, and real-world applications. Similar to recent studies [8, 17, 44], we split SVHN, CIFAR10, and CIFAR100 into 10 and 50 clients and each round select 10 clients to join the federated training, denoted as full participation and partial participation (straggler situation), respectively. With the help of Dirichlet distribution [11], we verify all methods on IID, Non-IID  $(\beta = 0.5)$ , and extreme Non-IID situations  $(\beta = 0.1$  or  $\beta = 0.2)$ . As the decreasing  $\beta$ , the level of PCDD increases and we show the heat map of data distribution in Appendix C. We set  $\beta = 0.2$  in partial participation to make sure each client has at least one batch of samples. The training round for SVHN and CIFAR10 is 50 in full participation and 100 in partial participation while for CIFAR100, it is set to 100 and 200. Besides, we also utilize a real federated scenario Fed-ISIC2019 to verify the ability to real-world application.
Full participation and partial participation. As shown in Table 2, with the decreasing  $\beta$  or increasing number of clients, the generic performance of FedAvg and all other methods greatly drops while the personal performance of all methods greatly increases. This means under PCDD and the straggler problem, the performance of generic performance is limited but the personal distribution is easier to capture. As for P-FL methods, they fail in global tasks especially in severe PCDD
Table 3: Personal and generic performance on a real federated application Fed-ISIC2019. More results of other realworld dataset are shown in the Appendix.
Method	FedAvg	FedProx	MOON	FedRS	FedGen	FedLC	FedRep	FedProto	FedBABU	FedRod	FedGELA
PA	77.27±0.19	77.91±0.16	77.94±0.17	78.27±0.12	78.02±0.23	77.58±0.19	76.94±0.13	77.80±0.17	78.91±0.13	78.65±0.34	79.27±0.19
GA	73.59±0.17	73.69±0.26	73.80±0.21	74.60±0.15	74.37±0.27	74.26±0.25	68.05±0.37	66.26±0.16	74.06±0.31	74.98±0.21	75.85±0.16
situations since they do not consider the global convergence during training. As for G-FL methods, the performance is better in generic tasks but limited in personalized tasks, especially in CIFAR100. They constrain the model's ability to fit personalized distributions during local training, resulting in improved consistency during global optimization. As can be seen, our FedGELA consistently exceeds all baselines for all settings with averaged performance of  $2.42\%$ ,  $5.2\%$  and  $5.7\%$  to FedAvg and  $1.35\%$ ,  $1.64\%$  and  $1.81\%$  to the best baseline on the three datasets respectively.
Performance in real-world applications. Except for the above three benchmarks, we also verify FedGELA with other methods under a real PCDD federated application: Fed-ISIC2019. As shown in Table 3, our method achieves the best improvement of  $2\%$  and  $2.26\%$  relative to FedAvg and of  $0.36\%$  and  $1.25\%$  relative to the best baseline on personal and generic tasks respectively, which demonstrates that our method is robust to practical situations in the both views. In the Appendix D.5, we provide more results on additional two real-world applications named FEMNIST and SHAKESPEARE to further show the effectiveness of our method in the real-world scenarios.
Figure 4: Bilateral performance on four datasets by tuning  $\log E_W$  (x axis) of FedGELA.
(a) Global, Full Parti.,  $\beta = 0.5$
(b) Global, Full Parti.,  $\beta = 0.1$
(c) Global, Partial Parti.,  $\beta = 0.5$
(d) Local, Full Parti.,  $\beta = 0.5$
(e) Local, Full Parti.,  $\beta = 0.1$
(f) Local, Partial Parti.,  $\beta = 0.5$
Figure 5: Illustration of the averaged angle between locally existing classes and missing classes on the local client and global server of FedAvg, FedGE, and our FedGELA on CIFAR10.
Table 4: Ablation study of FedGELA. GE and LA mean the global ETF and local adaptation.
GE LA	SVHN	CIFAR10	CIFAR100	Fed-ISIC2019
#Partition	Full Parti.	Partial Parti.	Full Parti.	Partial Parti.	Full Parti.	Partial Parti.	Real World
#Metric	PA	GA	PA	GA	PA	GA	PA	GA	PA	GA	PA	GA	PA	GA
-	-	95.02	86.36	93.15	88.43	82.50	64.88	72.52	59.19	69.09	62.80	56.46	54.28	77.27	73.59
✓	-	95.92	88.93	93.97	92.42	83.63	69.70	77.66	65.56	71.46	66.02	62.67	58.98	69.88	75.54
-	✓	95.93	74.84	93.15	89.58	83.97	63.75	77.76	61.55	71.93	60.76	58.92	51.95	54.65	62.43
✓	✓	96.54	89.07	95.69	92.15	84.61	69.46	79.95	65.21	74.23	66.05	66.33	58.81	79.27	75.85
More angle visualizations. In Figure 5, we show the effectiveness of local adaptation in FedGELA and verify the convergence of fixed classifier as ETF and local adaptation compared with FedAvg. Together with Figure 3, it can be seen that, compared with FedAvg, both FedGE and FedGELA converge faster to a larger angle between all class means in global. In the meanwhile, the angle between existing classes of FedGELA in the local is much larger, which proves FedGELA converges better than FedAvg and the adaptation brings little limits to convergence but many benefits to local performance under different levels of PCDD.
Hyper-parameter. FedGELA introduces constrain  $E_{H}$  on the features and the length  $E_{W}$  of classifier vectors. We perform  $L_{2}$  norm on all features in FedGELA, which means  $E_{H} = 1$ . For the length of the classifier, we tune it as hyper-parameter. As shown in Figure 4, from a large range from 10e3 to 10e6 of  $E_{W}$ , our method achieves bilateral improvement compared to FedAvg on all datasets.
Ablation studies. Since our method includes two parts: global ETF and local adaptation, we illustrate the average accuracy of FedGELA on all splits of SVHN, CIFAR10/100, and Fed-ISIC2019 without the global ETF or the local adaptation or both. As shown in Table 4, only adjusting the local classifier does not gain much in personal or global tasks, and compared with FedGE, FedGELA achieves similar generic performance on the four datasets but much better performance on the personal tasks.
Table 5: Performance of FedGELA compared with FedAvg and the best baseline under pure PCDD settings on CIFAR10 and SVHN datasets.  $P\varrho C_{\varsigma}$  means that the dataset is divided into  $\varrho$  clients and each client has  $\varsigma$  classes. We show the improvement in red on each baseline compared to FedGELA.
Dataset (split)	Metric	FedAvg	Best Baseline	FedGELA
CIFAR10(P10C2)	PA	92.08+3.76	94.07+1.77	95.84
GA	47.26+12.34	52.02+7.58	59.60
CIFAR10(P50C2)	PA	91.74+3.68	93.22+2.20	95.42
GA	36.22+18.56	44.74+10.04	54.78
SVHN(P10C2)	PA	95.64+3.11	97.02+1.73	98.75
GA	69.34+14.22	76.06+7.50	83.56
SVHN(P50C2)	PA	94.87+3.50	96.88+1.49	98.37
GA	66.94+10.24	72.97+4.21	77.18
Table 6: Performance of choosing different  $\Phi$ . Assuming the row vector of distribution matrix  $(\Phi_k)_c^T$  is related to class distribution  $\frac{n_{k,c}}{n_k}$  and the relationship as  $Q_{k}(\frac{n_{k,c}}{n_{k}})$ . Except for  $Q_{k}(x) = x$ , we have also considered employing alternative methods like employing an exponential  $Q_{k}(x) = e^{x}$  or power function  $Q_{k}(x) = x^{\frac{1}{2}}$  of the number of samples.
Dataset (split)	Metric	Qk(x) = ex	Qk(x) = x1/2	Qk(x) = x(ours)
SVHN(IID)	PA	95.12	95.43	94.84
GA	94.32	93.99	94.66
SVHN(β = 0.5)	PA	96.18	95.56	96.27
GA	93.28	93.22	93.66
SVHN(β = 0.1)	PA	98.33	98.21	98.52
GA	78.95	77.18	78.88
Performance under pure PCDD setting. To verify our method under pure PCDD, we decouple the PCDD setting and the ordinary heterogeneity (Non-PCDD). In Table 5, we use  $\mathrm{PxCy}$  to denote the dataset is divided in to x clients with y classes, and in each round, 10 clients are selected into federated training. The training round is 100. According to the results, FedGELA achieves significant improvement especially  $18.56\%$  to FedAvg and  $10.04\%$  to the best baseline on CIFAR10 (P50C2).
Other types of  $\Phi$ . Considering the aggregation of local classifiers should align with the global classifier, which ensures the validity of theoretical analyses from both global and local perspectives,  $\sum_{k=1}^{N} p_k \Phi_k$  should be 1 (1 is all-one matrix). Assuming the row vector of distribution matrix  $(\Phi_k)_c^T$  is related to class distribution  $\frac{n_{k,c}}{n_k}$  and the relationship as  $Q_k\left(\frac{n_{k,c}}{n_k}\right)$ . The equation can be rewrite as:  $\gamma \sum_{k=1}^{N} p_k Q_k\left(\frac{n_{k,c}}{n_k}\right) = 1$ , where  $\gamma$  is the scaling constant. In our FedGELA, to avoid sharing statistics for privacy, we only find one potential way that  $Q_k\left(\frac{n_{k,c}}{n_k}\right) = \frac{n_{k,c}}{n_k}$  and  $\gamma = \frac{1}{C}$ . In this part, we have also considered employing alternative methods like employing an exponential or power function of the number of samples. As shown in the Table 6, other methods need to share  $Q_k\left(\frac{n_{k,c}}{n_k}\right)$  but achieve the similar performance compared to FedGELA, which exhibits the merit of our choice.
In Appendix D, we provide more experiments from other perspectives like communication efficiency and the local burden of storing and computation, to show promise of FedGELA.
In this work, we study the problem of partially class-disjoint data (PCDD) in federated learning on both personalized federated learning (P-FL) and generic federated learning (G-FL), which is practical and challenging due to the angle collapse of classifier vectors for the global task and the waste of space for the personal task. We propose a novel method, FedGELA, to address the dilemma via a bilateral curation. Theoretically, we show the local and global convergence guarantee of FedGELA and verify the justification on the angle of global classifier vectors and on the angle between locally existing classes. Empirically, extensive experiments show that FedGELA achieves promising improvements on FedAvg under PCDD and outperforms state-of-the-art methods in both P-FL and G-FL.
The work is supported by the National Key R&D Program of China (No. 2022ZD0160702), STCSM (No. 22511106101, No. 22511105700, No. 21DZ1100100), 111 plan (No. BP0719010) and National Natural Science Foundation of China (No. 62306178). Ziqing Fan and Ruipeng Zhang were partially supported by Wu Wen Jun Honorary Doctoral Scholarship, AI Institute, Shanghai Jiao Tong University. Bo Han was supported by the NSFC Young Scientists Fund No. 62006202, NSFC General Program No. 62376235, Guangdong Basic and Applied Basic Research Foundation No. 2022A1515011652, and CCF-Baidu Open Fund.
[1] Manoj Ghuhan Arivazhagan, Vinay Aggarwal, Aaditya Kumar Singh, and Sunav Choudhary. Federated learning with personalization layers. arXiv preprint arXiv:1912.00818, 2019.
[2] Sebastian Caldas, Sai Meher Karthik Duddu, Peter Wu, Tian Li, Jakub Konečný, H Brendan McMahan, Virginia Smith, and Ameet Talwalkar. Leaf: A benchmark for federated settings. arXiv preprint arXiv:1812.01097, 2018.
[3] Hong-You Chen and Wei-Lun Chao. On bridging generic and personalized federated learning for image classification. In International Conference on Learning Representations, 2022.
[4] Noel CF Codella, David Gutman, et al. Skin lesion analysis toward melanoma detection: A challenge at the 2017 international symposium on biomedical imaging (isbi), hosted by the international skin imaging collaboration (isic). In ISBI, pages 168-172, 2018.
[5] Gregory Cohen, Saeed Afshar, Jonathan Tapson, and Andre Van Schaik. Emmist: Extending mnist to handwritten letters. In 2017 international joint conference on neural networks (IJCNN), pages 2921-2926. IEEE, 2017.
[6] Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai. Exploiting shared representations for personalized federated learning. In International Conference on Machine Learning, pages 2089-2099. PMLR, 2021.
[7] Marc Combalia, Noel CF Codella, Veronica Rotemberg, Brian Helba, Veronica Vilaplana, Ofer Reiter, Cristina Carrera, Alicia Barreiro, Allan C Halpern, Susana Puig, et al. Bcn20000: Dermoscopic lesions in the wild. arXiv preprint arXiv:1908.02288, 2019.
[8] Ziqing Fan, Yanfeng Wang, Jiangchao Yao, Lingjuan Lyu, Ya Zhang, and Qi Tian. Fedskip: Combating statistical heterogeneity with federated skip aggregation. In 2022 IEEE International Conference on Data Mining (ICDM), pages 131-140, 2022. doi: 10.1109/ICDM54844.2022.00023.
[9] Cong Fang, Hangfeng He, Qi Long, and Weijie J Su. Exploring deep neural networks via layer-peeled model: Minority collapse in imbalanced training. Proceedings of the National Academy of Sciences, 118(43):e2103091118, 2021.
[10] Eduardo Gaitan, Norman C Nelson, and Galen V Poole. Endemic goiter and endemic thyroid disorders. World journal of surgery, 15(2):205-215, 1991.
[11] Jonathan Huang. Maximum likelihood estimation of dirichlet distribution parameters. CMU Technique Report, pages 1-9, 2005.
[12] Wenlong Ji, Yiping Lu, Yiliang Zhang, Zhun Deng, and Weijie J Su. An unconstrained layer-peeled perspective on neural collapse. In International Conference on Learning Representations, 2021.
[13] Yilun Jin, Yang Liu, Kai Chen, and Qiang Yang. Federated learning without full labels: A survey. arXiv preprint arXiv:2303.14453, 2023.
[14] Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. Foundations and Trends® in Machine Learning, 14:1-210, 2021.
[49] Ruipeng Zhang, Ziqing Fan, Qinwei Xu, Jiangchao Yao, Ya Zhang, and Yanfeng Wang. Grace: A generalized and personalized federated learning method for medical imaging. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 14-24. Springer, 2023.
[50] Ruipeng Zhang, Qinwei Xu, Jiangchao Yao, Ya Zhang, Qi Tian, and Yanfeng Wang. Federated domain generalization with generalization adjustment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3954-3963, 2023.
[51] Yuchen Zhang, Martin J Wainwright, and John C Duchi. Communication-efficient algorithms for statistical optimization. Advances in neural information processing systems, 25, 2012.
[52] Zhihan Zhou, Jiangchao Yao, Yan-Feng Wang, Bo Han, and Ya Zhang. Contrastive learning with boosted memorization. In International Conference on Machine Learning, pages 27367-27377. PMLR, 2022.
[53] Zhihui Zhu, Tianyu Ding, Jinxin Zhou, Xiao Li, Chong You, Jeremias Sulam, and Qing Qu. A geometric analysis of neural collapse with unconstrained features. Advances in Neural Information Processing Systems, 34:29820-29834, 2021.
[54] Zhuangdi Zhu, Junyuan Hong, and Jiayu Zhou. Data-free knowledge distillation for heterogeneous federated learning. In International conference on machine learning, pages 12878-12889. PMLR, 2021.