SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Additionally, to explore broader real-life data heterogeneity issues with both feature skew and label distribution skew, we conducted further experiments by exerting different levels of label distribution skew on the Digits dataset. Specifically, we partitioned each sub-dataset of Digits using the Dirichlet Distribution, with different values of  $\alpha \in \{0.3, 0.7, 1.0, 1.5, 2.0\}$ . A smaller  $\alpha$  value indicates a more severe label distribution imbalance. The experimental results are demonstrated in Fig. 3. It is evident that in every instance of label distribution imbalance, our Fed- $\mathrm{CO}_{2}$  exhibits a significant advantage over other benchmark methods in its sub-dataset as well as the overall average performance. Therefore, we can firmly conclude that our universal FL framework, Fed- $\mathrm{CO}_{2}$ , effectively addresses a wide range of real-life data heterogeneity issues.
F Additional Experiments and Analyses
F.1 Effectiveness of Cooperation Framework
The experimental results of testing accuracy on five benchmark datasets have validated that our  $\mathrm{Fed - CO_2}$  effectively combines online general knowledge and offline specialized knowledge, resulting in improved adaptation to local data distribution in FL scenarios with label imbalance and feature
Figure 3: The test accuracy of our  $\mathrm{Fed - CO_2}$  and benchmark algorithms in Federated Learning with Feature Skew and varying levels of Label Distribution Skew. Fig. 3(a) depicts the model performance on the SynthDigits sub-dataset, while Fig. 3(b) demonstrates the average model performance across all sub-datasets in Digits.
shifts. To further confirm the effectiveness of the cooperation between the online and offline models, we conducted additional experiments for FL with label imbalance and feature shifts in the following parts.
Online and Offline Models in FL with Label Distribution Skew. For FL with label imbalance, we split the CIFAR10 dataset for 20 clients with various types and degrees of label distribution skew. In the Pathological case, each client  $i$  was randomly assigned two classes following the strategies mentioned in Appendix A. In the Dirichlet case, we partitioned the dataset randomly utilizing the Dirichlet distribution with different parameters  $\alpha \in \{0.3, 0.7, 1.0\}$  to evaluate model performances under various extents of label distribution imbalance. In these experiments, we compared the model trained by our Fed-  $\mathrm{CO}_2$  with its online and offline models.
The experimental results have been exhibited in Fig. 4. It is evident that the online model outperforms the offline model when the label distribution imbalance is moderate, while the offline model excels when the imbalance is severe. Across all extensive experiments,  $\mathrm{Fed - CO_2}$  consistently surpasses both the standalone online model and the standalone offline model in terms of test accuracy statistics, exhibiting higher average and median values. At the client level,  $\mathrm{Fed - CO_2}$  improves prediction accuracy for the majority of local clients. Although  $\mathrm{Fed - CO_2}$  may not surpass the superior model between the online and offline models on a limited number of clients, it consistently outperforms the weaker model and achieves performance levels approaching that of the superior model. Therefore, we can conclude that our collaborative FL framework,  $\mathrm{Fed - CO_2}$ , which combines the strengths of online and offline models, is exceptionally effective. It successfully integrates online general knowledge and offline specialized knowledge, resulting in an improved adaptation to local data distribution under various cases.
Online and Offline Models in FL with Feature Skew. For FL with feature skew, we conducted experiments that evaluate the performance of the online and offline models on DomainNet. Based on the experimental results shown in Table 10, we have the following observations: First, the online model outperforms the offline model on most clients, except for the Quickdraw client. This phenomenon validates our hypothesis that when the feature shift is severe, the model aggregation will lead to the loss of important local offline information, resulting in the model's failure to adapt to such significant data heterogeneity. Second, prediction fusion is even inferior to the online and offline models for some clients. This result reflects that prediction fusion is not sufficient to fuse online general knowledge and offline specialized knowledge in FL with severe data heterogeneity issues. Therefore, for FL with feature skew, intra-client and inter-client knowledge transfer mechanisms are required to boost model and client cooperation under our novel framework.
(a)Pathological
(b)Dirichlet  $\alpha = 0.3$
(c)Dirichlet  $\alpha = 0.7$
(d)Dirichlet  $\alpha = 1.0$
Figure 4: Comparison results among the online model, the offline model, and our  $\mathrm{Fed - CO_2}$  with different kinds of label distribution skew. In each subgraph, we illustrate label distribution among 20 clients (Left), the classification test accuracy statistics (Middle), where the full line represents the median accuracy and the dotted line represents the average accuracy among the 20 clients, and test accuracy in each client (Right).
Table 10: Performance of Online and Offline Models in FL with Feature Skew on DomainNet
Methods	Clipart	Infograph	Painting	Quickdraw	Real	Sketch	Avg
Online Model	51.2±1.4	26.8±0.5	41.5±1.4	71.3±0.7	54.8±0.8	42.1±1.3	48.0±1.0
Offline Model	41.0±0.9	23.8±1.2	36.2±2.7	73.1±0.9	48.5±1.9	34.0±1.1	42.8±1.5
Fed-CO2(no intra and inter knowledge transfer)	48.7±0.9	26.5±2.0	42.1±1.0	72.9±0.8	57.1±1.1	40.0±0.8	47.9±0.7
Fed-CO2	55.0±1.1	28.6±1.1	44.3±0.6	75.1±0.6	62.4±0.8	45.7±1.9	51.8±0.2
F.2 Knowledge Transfer Mechanisms in Feature Skew
As mentioned in the main paper, intra-client and inter-client knowledge transfer mechanisms foster model-level and client-level collaboration to better make use of general domain-invariant and specialized domain-specific knowledge for a better adaptation to local data distribution. In this section, we did further ablation studies on the challenging DomainNet dataset to investigate two mechanisms from a fine-grained angle.
Ablation Study in Intra-client Knowledge Transfer Mechanism. In this section, we investigated the impact of the intra-client knowledge transfer mechanism on the online and offline models by excluding the inter-client knowledge transfer mechanism from  $\mathrm{Fed - CO_2}$ . Specifically, we performed separate intra-client knowledge transfers: from the offline model to the online model and from the online model to the offline model. The results are demonstrated in Table 11 and we can discover that knowledge transfer is beneficial to both online and offline models. By transferring specialized domain-specific knowledge to the online model, we can observe an improvement in the performance of the Quickdraw client, which has a more extreme non-IID data distribution and relies more on local offline information. Conversely, when transferring general domain-invariant knowledge to the offline model, we could notice enhanced performance in the remaining clients that present milder data heterogeneity and require more global information. Hence, by incorporating the full intra-client knowledge transfer mechanism, our  $\mathrm{Fed - CO_2}$  effectively utilizes both online domain-invariant and offline domain-specific knowledge, resulting in improved average test accuracy.
Table 11: Ablation Study of Intra-client Knowledge Transfer mechanism on DomainNet
Method	Clipart	Infograph	Painting	Quickdraw	Real	Sketch	Avg
No intra transfer	48.7±0.9	26.5±2.0	42.1±1.0	72.9±0.8	57.1±1.1	40.0±0.8	47.9±0.7
Intra transfer to online	48.1±1.2	26.2±1.3	42.1±1.2	74.0±0.5	57.0±1.0	40.4±1.0	48.0±0.3
Intra transfer to offline	50.9±0.7	26.6±0.7	42.1±1.1	72.8±0.5	59.1±0.3	41.3±1.8	48.8±0.2
Full intra transfer	50.4±0.7	27.0±1.1	43.9±0.7	74.1±0.6	58.1±0.8	42.0±0.8	49.3±0.5
Ablation Study in Inter-client Knowledge Transfer Mechanism. In this part, we removed the intra-client knowledge transfer mechanism from  $\mathrm{Fed - CO_2}$  and did experiments to investigate the impact of inter-client knowledge transfer mechanism on online and offline models. The experiments included exerting inter-client knowledge transfer mechanism on the online model, the offline model, and both models. Table 12 presents the experimental results. The results demonstrate that knowledge from other domains benefits both the online and offline models, with the online model showing more remarkable improvement. This observation aligns consistently with the purpose of the online model, which focuses on acquiring general domain-invariant knowledge that can be applied across various domains. Furthermore, the application of the inter-client knowledge transfer mechanism to both the online and offline models yields optimal performance, which means simultaneously enhancing domain generalization ability for online and offline models can exploit domain-invariant knowledge more effectively. In summary, our inter-client knowledge transfer mechanism enables each local client to acquire and exploit more domain-invariant knowledge resulting in a better adaptation to local data distribution.
Table 12: Ablation Study of Inter-client Knowledge Transfer Mechanism on DomainNet
Methods	Clipart	Infograph	Painting	Quickdraw	Real	Sketch	Avg
No inter transfer	48.7±0.9	26.5±2.0	42.1±1.0	72.9±0.8	57.1±1.1	40.0±0.8	47.9±0.7
Online model with inter transfer	51.3±0.7	27.1±1.0	44.2±1.3	74.1±0.7	61.4±0.7	43.5±1.5	50.3±0.5
Offline model with inter transfer	51.8±1.3	26.3±1.8	43.3±1.3	74.2±1.0	59.0±2.2	43.1±1.6	49.6±0.6
Full inter transfer	53.9±0.6	26.2±0.7	42.9±0.9	75.1±0.3	61.9±0.7	46.7±0.7	51.1±0.2
F.3 Data Heterogeneity of Noise
Noise is another common factor causing data heterogeneity issues in real FL scenarios, where each client has the same label distribution but is subject to various degrees of noise. Therefore, we implemented additional experiments on this new scenario to make the initial attempt. In detail, we divided the dataset CIFAR10 into 10 clients in a way to make them independently and identically distributed. Then we applied the increasing level of random Gaussian noise with mean  $\mu_{i} = 0$  and standard deviation  $\sigma_{i} = \frac{\sigma_{M}}{N - 1} * i$ , where  $i \in \{0, 1, \dots, N - 1\}$ . In this series of initial experiments,
we let the client number and the sample rate be 0.5 with  $\sigma_{M} \in \{0, 5, 10, 15, 20\}$ . The backbone of each method is the same as that in other experiments.
As presented in Fig. 5, the experimental results reveal that our  $\mathrm{Fed - CO_2}$  consistently exhibits better and more robust performance across nearly all experiments, particularly in scenarios with severe noise. These experiments will also serve as a source of inspiration for our future work.
Figure 5: Test accuracy on dataset CIFAR10 for  $\mathrm{Fed - CO_2}$  and some benchmark methods with different degrees of data heterogeneity caused by noise.
F.4 Effects of Data Number and Client Number
The sample size of the training set and the client number are significant factors. Therefore, in this section, we conducted a series of experiments to explore their influence.
To investigate the effect of the training sample number, we utilized dataset Digits and set  $10\%$  of its training data as the original training dataset. Then, we selected a certain ratio  $\gamma$  of the original training dataset as a new training dataset for each client, where  $\gamma \in \{0.2, 0.4, 0.6, 1.0, 2.0\}$ . The results shown in Fig. 6 demonstrate that our Fed- $\mathrm{CO}_2$  have an edge over various benchmark methods in almost every case, especially when the number of training data is very limited.
Figure 6: The effect of the training number ratio on  $\mathrm{Fed - CO_2}$  and benchmark algorithms for Federated Learning with Feature Skew. Fig. 6(a) depicts the model performance on the MNIST-M sub-dataset, while Fig. 6(b) demonstrates the average model performance across all sub-datasets in Digits.
As for the effect of client number, we employed dataset CIFAR10 with Dirichlet setting  $(\alpha = 0.3)$  to conduct a series of experiments on benchmark methods with client number  $N \in \{50, 100, 150, 300, 500\}$ . The experimental results shown in Table 13 demonstrate that our Fed- $\mathrm{CO}_{2}$  consistently outperforms a bunch of SOTA methods in every case even if the client number is substantial. The results in Table 13 and Fig. 6 prove that our Fed- $\mathrm{CO}_{2}$  owns higher scalability than benchmark methods.
Table 13: The influence of client numbers on CIFAR10
#client	50	100	150	300	500
SingleSet	71.28±0.08	68.69±0.07	70.05±0.06	64.62±0.04	60.31±0.07
FedAvg [2]	41.20±3.13	52.85±2.30	33.34±11.15	42.63±3.85	34.31±5.68
FedProx [4]	45.35±6.94	36.00±4.42	34.53±6.59	37.44±6.23	34.58±5.18
MOON [7]	40.25±3.31	52.38±2.16	34.93±9.74	49.58±3.93	37.35±5.80
FedRoD [12]	65.18±1.14	66.08±0.92	67.00±1.43	65.46±1.50	64.68±4.30
FedBN [10]	74.40±0.61	75.43±0.39	78.38±0.44	76.06±0.46	69.24±1.21
Fed-CO2	77.14±0.37	77.21±0.29	80.06±0.25	78.53±0.31	75.59±0.50
F.5 Personalized Parts in the Online model
In order to enhance the adaptation to local data distribution, it is necessary to personalize certain critical parameters in the online model to preserve the essential offline specialized knowledge. In this section, we investigated and analyzed which specific parts of our online model should be personalized for optimal performance in FL with feature skew. The most common strategy is to personalize Batch Normalization layers or the Classifier Head. Therefore in our  $\mathrm{Fed - CO_2}$  approach, we conducted three experiments with different personalized components in the online model to compare and determine the optimal strategy. The results for three datasets, namely DomainNet, Office-Caltech10, and Digits, are illustrated in Fig. 7. It is evident from the observations that personalizing the Classification Head has a negative impact on almost every sub-dataset, resulting in lower classification accuracy. Based on the results, we can conclude that personalizing Batch Normalization layers in our online model is the best strategy to adapt to feature shifts.
Figure 7: The test accuracy for our  $\mathrm{Fed - CO_2}$  with different personalized parts in the online model on DomainNet, Office-Caltech10, and Digits.