SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The performance of Fed-GraB/SGB on diverse settings are reported in Tab. 1. Notably, Fed-GraB achieves the best overall accuracies on all settings, with a significant improvement on the tail classes while ensuring good performance of the head classes. The performance gain becomes more evident under extremely imbalanced data. Moreover, the SGB implemented in the CL setting (i.e., performance upper bound testing) surpasses existing SOTA long-tail methods, further demonstrating the universal effectiveness of Fed-GraB under different scenarios.
Setting	Method	IFG=10	IFG=50	IFG=100
Many	Med	Few	All	Many	Med	Few	All	Many	Med	Few	All
CL	Softmax	0.901	0.879	0.886	0.893±0.003	0.908	0.776	0.742	0.817±0.003	0.920	0.745	0.675	0.767±0.001
Eqlv2	0.902	0.880	0.874	0.890±0.002	0.903	0.774	0.774	0.819±0.006	0.912	0.751	0.679	0.775±0.012
LDAM	0.957	0.799	0.854	0.838±0.026	0.964	0.739	0.685	0.753±0.011	0.936	0.729	0.610	0.698±0.030
SGB-CL	0.897	0.901	0.901	0.898±0.008	0.891	0.817	0.833	0.846±0.004	0.901	0.738	0.814	0.818±0.003
α=1	FedAvg	0.896	0.858	0.846	0.877±0.001	0.888	0.771	0.693	0.792±0.005	0.922	0.716	0.616	0.737±0.001
FedProx	0.898	0.859	0.854	0.877±0.002	0.891	0.773	0.691	0.794±0.002	0.921	0.725	0.582	0.729±0.002
FedNova	0.912	0.853	0.848	0.882±0.002	0.903	0.757	0.702	0.797±0.003	0.934	0.734	0.599	0.739±0.005
FedIR	0.966	0.823	0.862	0.868±0.001	0.972	0.775	0.693	0.784±0.002	0.969	0.755	0.576	0.728±0.004
CReFF	0.911	0.850	0.887	0.884±0.002	0.896	0.769	0.664	0.791±0.003	0.935	0.723	0.574	0.726±0.002
τ-norm	0.887	0.871	0.908	0.884±0.003	0.878	0.790	0.725	0.805±0.002	0.922	0.726	0.668	0.760±0.004
Eqlv2-FL	0.896	0.852	0.857	0.878±0.003	0.886	0.786	0.690	0.790±0.009	0.919	0.704	0.597	0.729±0.003
LDAM-FL	0.901	0.845	0.825	0.863±0.004	0.881	0.739	0.662	0.768±0.005	0.891	0.638	0.495	0.679±0.025
Focal-FL	0.887	0.839	0.834	0.869±0.005	0.877	0.744	0.665	0.775±0.002	0.916	0.701	0.558	0.733±0.037
GCL-FL	0.923	0.747	0.781	0.796±0.000	0.949	0.748	0.689	0.761±0.005	0.963	0.726	0.608	0.726±0.001
Fed-GraB	0.886	0.882	0.893	0.885±0.001	0.875	0.784	0.775	0.818±0.003	0.910	0.698	0.713	0.766±0.003
α=0.5	FedAvg	0.890	0.864	0.861	0.876±0.002	0.865	0.772	0.685	0.781±0.003	0.906	0.720	0.585	0.719±0.005
FedProx	0.883	0.864	0.863	0.874±0.000	0.857	0.776	0.688	0.782±0.001	0.892	0.712	0.564	0.715±0.011
FedNova	0.903	0.856	0.834	0.877±0.002	0.888	0.773	0.679	0.788±0.003	0.924	0.739	0.609	0.739±0.004
FedIR	0.955	0.816	0.870	0.866±0.001	0.961	0.771	0.698	0.781±0.001	0.976	0.726	0.562	0.715±0.005
CReFF	0.900	0.838	0.880	0.877±0.004	0.878	0.778	0.664	0.786±0.003	0.932	0.699	0.580	0.718±0.003
τ-norm	0.883	0.863	0.880	0.877±0.004	0.867	0.759	0.728	0.795±0.003	0.962	0.726	0.611	0.731±0.011
Eqlv2-FL	0.895	0.858	0.855	0.877±0.002	0.872	0.771	0.639	0.775±0.005	0.898	0.717	0.586	0.714±0.011
LDAM-FL	0.885	0.860	0.848	0.850±0.006	0.863	0.742	0.673	0.709±0.037	0.877	0.583	0.490	0.657±0.024
Focal-FL	0.868	0.853	0.858	0.865±0.005	0.842	0.780	0.661	0.767±0.005	0.883	0.703	0.581	0.728±0.029
GCL-FL	0.902	0.731	0.813	0.798±0.004	0.938	0.725	0.725	0.768±0.001	0.954	0.730	0.623	0.713±0.016
Fed-GraB	0.864	0.891	0.897	0.881±0.002	0.836	0.790	0.783	0.806±0.001	0.910	0.698	0.713	0.761±0.008
Table 1: Test accuracies of Fed-GraB/SGB and SOTA methods on CIFAR-10-LT with diverse imbalanced and heterogeneous data settings. More results (e.g., DPA with CL re-weighting methods and IID settings) are in the Supplementary.
Method	CIFAR-100-LT	ImageNet-LT	iNaturalist-160k
Many	Med	Few	All	Many	Med	Few	All	Many	Med	Few	All
FedAvg	0.643	0.410	0.182	0.365±0.001	0.428	0.258	0.127	0.287±0.006	0.596	0.425	0.242	0.434±0.002
FedProx	0.639	0.416	0.181	0.366±0.000	0.439	0.268	0.128	0.292±0.002	0.582	0.424	0.241	0.425±0.011
FedNova	0.664	0.429	0.195	0.378±0.009	0.415	0.234	0.114	0.265±0.002	0.564	0.403	0.226	0.404±0.006
FedIR	0.634	0.410	0.182	0.364±0.000	0.388	0.206	0.074	0.236±0.012	0.579	0.387	0.191	0.396±0.003
CReFF	0.684	0.440	0.146	0.401±0.002	-	-	-	0.263±0.000	-	-	-	-
τ-norm	0.459	0.362	0.323	0.368±0.003	0.347	0.285	0.260	0.288±0.018	0.537	0.433	0.287	0.434±0.003
Eqlv2-FL	0.652	0.434	0.198	0.381±0.002	0.433	0.262	0.118	0.281±0.006	0.570	0.397	0.376	0.440±0.014
LDAM-FL	0.639	0.409	0.168	0.355±0.005	0.365	0.216	0.112	0.242±0.002	0.560	0.409	0.244	0.414±0.003
Focal-FL	0.645	0.418	0.179	0.367±0.001	0.424	0.266	0.136	0.283±0.005	0.596	0.430	0.247	0.430±0.005
GCL-FL	0.567	0.346	0.156	0.301±0.012	0.317	0.209	0.124	0.224±0.001	0.565	0.375	0.196	0.388±0.007
Fed-GraB	0.683	0.553	0.221	0.411±0.002	0.407	0.294	0.215	0.311±0.003	0.527	0.449	0.372	0.451±0.004
Table 2: Test accuracies of various methods on CIFAR-100-LT, ImageNet-LT and iNaturalist-160k with non-IID data settings.
Evaluation on CIFAR-100-LT, ImageNet-LT, and iNaturalist-160k: Tab. 2 shows that Fed-GraB consistently outperforms all baselines on test overall accuracies for CIFAR-100-LT and ImageNet-LT, with remarkable improvements on middle and tail classes. While LT-orientated methods obtain better results than the FL method in the extremely imbalanced ImageNet-LT, Fed-GraB can still surpass it $^3$ . The reason boils down to that for classification tasks with a large number of classes, local data samples for each class would be extremely small. In this case, the SGB with global prior for class-wise re-balancing is more effective in learning a better-aggregated model.
As for iNaturalist-160k, Fed-GraB significantly boosts the classification accuracy on tail classes while maintaining the best overall performance compared to FL methods. In comparison to LT-oriented methods, Fed-GraB's SGB provides a robust mechanism against data heterogeneity, leading to an outstanding performance on both 3-shot and overall metrics.
A salient advantage of Fed-GráB is that it achieves universal outperformance on three large-scale or real-world scenarios, demonstrating the robustness over other methods such as CReFF,  $\tau -\text{norm}$  or Eq1v2-FL which attain good performance on a specific dataset.
Evaluation of Communication Efficiency: Tab. 3 presents the communication efficiency of different methods trained on CIFAR-10-LT under two non-IID settings. We compare the required communica
3The result of CReFF on ImageNet-LT are from [8] and the result on iNaturalist is not included due to memory issues in large-scale datasets.
tion rounds for the tail classes to reach  $55\%$  test accuracy. We can see from this table that Fed-GraB attains the best convergence rate under both settings, confirming its effectiveness on the tail classes.
4.3 Effectiveness of DPA and SGB
SGB for consistent gradient re-weighting across clients: When various clients employ a gradient re-weighting method on a specific class, the strength of compensating divergence is because of the diverse local distributions and local dataset sizes. Therefore, although traditional re-weighting methods can mitigate the imbalance, discrepancies still exist. We demonstrate the advantage of SGB for addressing this issue. Specifically, we compute the mean and standard deviation (std) of  $\Delta_j(t)$  across all clients in CIFAR-10-LT for both Eq1v2 and Fed-GraB, as illustrated in Fig. 4 (a).
We make two observations here. First, the mean of  $\Delta_j(t)$  in Eq1v2 keeps decreasing alongside the training process while Fed-GraB can always align it to a near-zero value, suggesting that Fed-GraB can better re-weight the positive and negative gradients. Second, Eq1v2 exhibits a large variance among clients during training. In contrast, Fed-GraB maintains a very small deviation, evidently demonstrating the Fed-GraB can consistently motivate the heterogeneous clients to converge towards a better aggregated global model.
Effectiveness on global distribution and tail class estimation: We evaluate DPA under three IF settings with two metrics: (1) the L2-distance
Method	IFG=50	IFG=100
α = 1	α = 0.5	α = 1	α = 0.5
FedAvg	96	122	88	192
CReFF	285	286	309	348
Eqlv2-FL	301	210	191	205
LDAM-FL	483	427	-	-
Focal-FL	276	174	347	257
Fed-GBA	59	63	95	122
Table 3: A comparison of communication efficiency on CIFAR-10, in terms of the number of required rounds, where “-” indicate the method cannot obtain the target accuracy.
between the prior vector  $\mathcal{P}_c$  and ground truth distribution; (2) the tail identification accuracy (percentage of tail classes that are identified correctly by DPA). We plot the two metrics on a global model with FedAvg for 200 rounds in Fig. 4 (b). We can see that DPA converges after 70 rounds of training and captures more than  $90\%$  of the tail classes under different IFs. Moreover, the tail identification accuracy becomes more accurate with a higher IF, which is attributed to the extreme imbalance among classes (i.e., easier to distinguish).
Figure 4: (a): Visualization of mean and std of  $\Delta_j(t)$  during training; (b): Tail identification performance with DPA; (c): Comparison of accuracies with global/local-SGB mounting strategy.
(b)
(c)
Figure 5: (a): An illustration of effectiveness of  $\mathcal{P}_c$  with class-wise performance; (b): Mounting strategies with SGB; (c): An illustration of the trend of  $\Delta(t)$  with different values of  $z(t)$ .
Effectiveness of global  $\mathcal{P}_c$ : We investigate a local version (Local-SGB) of Fed-GraB (Global-SGB), where each client implements SGB based on their local distribution instead of the global prior vector derived by DPA. The results on CIFAR-10-LT in Fig. 4 (c) show that Global-SGB achieves a higher
test accuracy. Furthermore, we visualize the accuracies of FedAvg and Fed-GraB after local updates on the received global model in Fig. 5 (a), which demonstrate that Fed-GraB can achieve better and more balanced performance across classes, especially for tails (e.g., class 2, 3, 8, 9).
Mounting strategies with SGB: In Fed-GraB, SGB is universally applied to all classifiers through the use of DPA, rather than selectively targeting specific classes without DPA. We demonstrate that a full mounting approach with DPA can more readily attain optimal performance compared to a custom-tailored tailed class (i.e., mounting on 7 to 9 classes). As depicted in Fig. 5 (b), the model excels when SGB is employed on tail classes, while accuracy diminishes when mounting all classes or solely the exceedingly biased classes. Notably, DPA significantly bolsters overall performance.
4.4 Ablation Study and Model Analysis
The target of  $\Delta(t)$ : A critical step of SGB is to compute re-weighting coefficients  $\beta$  by  $e(t) = \Delta(t) - z(t)$ . Here we argue that for a static  $z(t)$ , different values of  $z(t)$  would not lead to significant differences in training, as long as those values are not extremely biased. We visualize the training process with two different  $z(t)$  in Fig. 5 (c) by assuming a continuous scenario. The training would consist of two phases during the closed-loop controlling. The Phase 1 could be regarded as a noisy initialization stage which is usually very short, resulting in different initial values based on  $e(t)$ . Afterward, the Phase 2 is started, and similar adjusting behaviors of  $e(t)$  could be observed, i.e.,  $\Delta(t)$  fluctuates around  $z(t)$ , leading to similar  $\beta$  that stands for re-weighting strength. As the duration  $\delta$  of Phase 1 is negligibly short and would not impose significant influence for training. The experimental results with different target values in Fig. 6 (a) further demonstrate our argument. More analysis is presented in the Supplementary.
Figure 6: (a): Performance ablation with different  $\Delta(t)$ ; (b): Ablation on different  $K_{P}$ ,  $K_{I}$  and  $K_{D}$  in Fed-GraB; (c): Performance evaluation with diverse  $\mathrm{IF}_{\mathrm{G}}$ .
Ablation on SGB hyper-parameters: To investigate the effects of hyper-parameters  $K_{P}$ ,  $K_{I}$ , and  $K_{D}$ , we observe the real-time  $\Delta(t)$  during the training process with various parameter groups, as depicted in Fig. 6 (b) using the parameter sets:  $K_{P} = 1, 10, 10, 10, K_{I} = 0, 0, 0, 0.01$ ,  $K_{D} = 0, 0, 0.1, 0.1$ . We notice that  $K_{P} = 10$  (green) more effectively constrains  $\Delta(t)$  around 0 compared to  $K_{P} = 1$  (orange). To reduce fluctuations, we incorporate  $K_{D} = 0.1$  (red). In this case,  $\Delta(t)$  remains above 0, necessitating the introduction of  $K_{I}$  (purple) to correct the static deviation. Notably, SGB demonstrates substantial robustness to the parameter group, as further evidenced in the Supplementary.
Fed-GraB on different imbalance levels: To demonstrate Fed-GraB is versatile across different global data distributions, we conduct experiments with global imbalance factors ranging from 5 to 50. As shown in Fig. 6 (c), Fed-GraB achieves better performance than others on a broad range of  $\mathrm{IF_G}$ . The results indicate that Fed-GraB could alleviate the performance degradation in moderate imbalanced cases while yielding more significant improvements on highly imbalanced data.
Computational and Storage Cost of SGB:
The additional cost from SGB are mainly attributed to the computation and storage of  $\Delta (t)$  and  $u(t)$ . As shown in Eq. (3), the main computational steps are the differential and summation of  $e(t)$  which should be linearly proportional to number of the gradients. Such extra computational cost is analogous to the additional manipulations of gradients in advanced stochastic gradient decent methods such as Momentum or Adam [60]. Overall, the extra computation which is implemented with several lines of code could be done very quickly. Besides, Fed-GraB needs some extra storage cost to store the weighted gradients which is quite cheap as well.
We proposed Fed-GraB, a self-adjusting and closed-loop gradient re-balancing framework for Fed-LT tasks. Fed-GraB comprises DPA, a federated global long-tailedness analyzer, and SGB, a local gradient balancer, addressing the discrepancies in inter-client and inter-class statistics. We carried out extensive experiments to analyze the functionality of different components in Fed-GraB and demonstrate the efficacy of Fed-GraB in various Fed-LT configurations.
Limitations. In this work, we mainly focus on aggregating a global Fed-LT model, while extensions to include clients are of high interest such as training superior heterogeneous clients with well personalized strategies. Besides, experiments with real-world medical or autonomous vehicle datasets could be examined to further demonstrate the effectiveness of Fed-GraB.
This work is supported by the National Natural Science Foundation of China (Grant No. 62106222, No. 62201504), the Natural Science Foundation of Zhejiang Province, China (Grant No. LZ23F020008, No. LGJ22F010001), Zhejiang Lab Open Research Project (No. K2022PD0AB05) and the Zhejiang University-Angelalign Inc. R&D Center for Intelligent Healthcare.
[1] Q. Yang, Y. Liu, T. Chen, and Y. Tong, "Federated machine learning: Concept and applications," ACM Transactions on Intelligent Systems and Technology (TIST), vol. 10, no. 2, pp. 1-19, 2019. 1
[2] T. Li, A. K. Sahu, A. Talwalkar, and V. Smith, "Federated learning: Challenges, methods, and future directions," IEEE Signal Processing Magazine, vol. 37, no. 3, pp. 50-60, 2020. 1
[3] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, "Communication-efficient learning of deep networks from decentralized data," in Artificial intelligence and statistics, pp. 1273-1282, PMLR, 2017. 1
[4] T.-M. H. Hsu, H. Qi, and M. Brown, “Federated visual classification with real-world data distribution,” in Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part X, pp. 76–92, 2020. 2, 3, 6
[5] Q. Li, Y. Diao, Q. Chen, and B. He, "Federated learning on non-iid data silos: An experimental study," in 2022 IEEE 38th International Conference on Data Engineering (ICDE), pp. 965-978, IEEE, 2022.
[6] L. Wang, S. Xu, X. Wang, and Q. Zhu, “Addressing class imbalance in federated learning,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, pp. 10165–10173, 2021. 3
[7] J. Xu, Z. Chen, T. Q. Quek, and K. F. E. Chong, "Fedcorr: Multi-stage federated learning for label noise correction," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10184-10193, 2022. 2
[8] X. Shang, Y. Lu, G. Huang, and H. Wang, "Federated learning on heterogeneous and long-tailed data via classifier re-training with federated features," arXiv preprint arXiv:2204.13399, 2022. 2, 3, 6, 7
[9] Z. Chen, S. Liu, H. Wang, H. H. Yang, T. Q. Quek, and Z. Liu, "Towards federated long-tailed learning," in International Workshop on Trustworthy Federated Learning in Conjunction with IJCAI 2022 (FL-IJCAI'22), 2022. 2
[10] Y. Zhang, B. Kang, B. Hooi, S. Yan, and J. Feng, "Deep long-tailed learning: A survey," arXiv preprint arXiv:2110.04596, 2021. 2, 3
[11] L. Yang, H. Jiang, Q. Song, and J. Guo, “A survey on long-tailed visual recognition,” International Journal of Computer Vision, pp. 1–36, 2022. 2
[46] H. Guo and S. Wang, “Long-tailed multi-label visual recognition by collaborative training on uniform and re-balanced samplings,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15089–15098, 2021. 3
[47] B. Li, Y. Yao, J. Tan, G. Zhang, F. Yu, J. Lu, and Y. Luo, “Equalized focal loss for dense long-tailed object detection,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6990–6999, 2022. 3
[48] T. Wu, Q. Huang, Z. Liu, Y. Wang, and D. Lin, "Distribution-balanced loss for multi-label classification in long-tailed datasets," in European Conference on Computer Vision, pp. 162-178, Springer, 2020.
[49] K. Cao, C. Wei, A. Gaidon, N. Arechiga, and T. Ma, “Learning imbalanced datasets with label-distribution-aware margin loss,” Advances in neural information processing systems, vol. 32, 2019. 3, 6
[50] T. Wang, Y. Zhu, Y. Chen, C. Zhao, B. Yu, J. Wang, and M. Tang, "C2am loss: Chasing a better decision boundary for long-tail object detection," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6980-6989, 2022. 3
[51] Y. Zhang, B. Hooi, L. Hong, and J. Feng, "Test-agnostic long-tailed recognition by test-time aggregating diverse experts with self-supervision," arXiv preprint arXiv:2107.09249, 2021. 4
[52] J. Liu, Y. Sun, C. Han, Z. Dou, and W. Li, “Deep representation learning on long-tailed data: A learnable embedding augmentation perspective,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2970–2979, 2020. 4
[53] J. Tan, C. Wang, B. Li, Q. Li, W. Ouyang, C. Yin, and J. Yan, “Equalization loss for long-tailed object recognition,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11662–11671, 2020. 4
[54] Y. Li, T. Wang, B. Kang, S. Tang, C. Wang, J. Li, and J. Feng, “Overcoming classifier imbalance for long-tail object detection with balanced group softmax,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10991–11000, 2020. 4
[55] S. Bennett, “Development of the pid controller,” IEEE Control Systems Magazine, vol. 13, no. 6, pp. 58–62, 1993. 5
[56] Y. Huang, S. Gupta, Z. Song, K. Li, and S. Arora, “Evaluating gradient inversion attacks and defenses in federated learning,” Advances in Neural Information Processing Systems, vol. 34, pp. 7232–7241, 2021. 6
[57] A. Krizhevsky, G. Hinton, et al., “Learning multiple layers of features from tiny images,” 2009. 6
[58] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, "Imagenet: A large-scale hierarchical image database," in 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255, IEEE, 2009. 6
[59] G. Van Horn, O. Mac Aodha, Y. Song, Y. Cui, C. Sun, A. Shepard, H. Adam, P. Perona, and S. Belongie, “The inaturalist species classification and detection dataset,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8769–8778, 2018. 6
[60] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in ICLR (Poster), 2015. 9