SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Assumption 4.1 For each client  $i \in \{1, \dots, N\}$ , the inputs  $\mathbf{x}_j^i$  are centered, meaning that  $\mathbb{E}[\mathbf{x}^i] = 0$  and they have a covariance matrix  $\mathbf{S}_i = \mathbb{E}[\mathbf{x}^i(\mathbf{x}^i)^\top]$ .  $\mathbf{S}_i$  is independent of the label  $y$  and varies for each client  $i$ . Not all  $\mathbf{S}_i$  matrices are identity matrices. For any index pair  $p$  and  $q$  ( $p \neq q$ ), we have  $\mathbf{x}_p \neq w \cdot \mathbf{x}_q$  for any non-zero  $w$ .
Let  $\mathbf{v}_k \in \mathbb{R}^d$  represent the parameters of the first layer, where  $k \in [m]$  and  $m$  is the width of the hidden layer. We define  $\| \mathbf{v} \|_{\mathbf{S}} \coloneqq \sqrt{\mathbf{v}^\top \mathbf{S} \mathbf{v}}$  as the induced vector norm for a positive definite matrix  $\mathbf{S}$  and a  $l_2$  vector norm  $\| \mathbf{v} \|_2$ . The projections of  $\mathbf{x}$  onto  $\mathbf{v}$  and  $\mathbf{v}^\perp$  are defined as  $\mathbf{x}^\vee \coloneqq \frac{\mathbf{v} \mathbf{v}^\top \mathbf{x}}{\|\mathbf{v}\|_2^2}$ ,  $\mathbf{x}^{\mathbf{v}^\perp} \coloneqq \left(\mathbf{I} - \frac{\mathbf{v} \mathbf{v}^\top}{\|\mathbf{v}\|_2^2}\right) \mathbf{x}$ .
Based on Assumption 4.1, for client  $i$ , the output of the first layer is normalized as  $\frac{\mathbf{v}_k^\top\mathbf{x}^i}{\|\mathbf{v}_k\|_{\mathbf{S}_i}}$ . The shift parameter of BN is omitted. Further, we denote the scaling parameter of BN  $\gamma \in \mathbb{R}^{m\times N}$  and the second layer parameter  $c\in \mathbb{R}^m$ .
With these parameters, we proceed to train the online model:
$$
\mathbf {F} ^ {\mathrm {o n}} (\mathbf {x}; \mathbf {V}, \gamma , c) = \frac {1}{\sqrt {m}} \sum_ {k = 1} ^ {m} c _ {k} ^ {\mathrm {o n}} \sum_ {i = 1} ^ {N} \sigma \left(\gamma_ {k, i} ^ {\mathrm {o n}} \cdot \frac {\mathbf {v} _ {k} ^ {\mathrm {o n}} {} ^ {\top} \mathbf {x}}{\| \mathbf {v} _ {k} ^ {\mathrm {o n}} \| \mathbf {s} _ {i}}\right) \cdot \mathbb {1} \left\{\mathbf {x} \in \mathbb {D} _ {i} \right\}, \tag {8}
$$
where  $\sigma (\cdot)$  is the ReLU activation function and  $\gamma_{k,i}^{\mathrm{on}}$  is personalized BN parameters. For the offline model with all parameters personalized, we train:
$$
\mathbf {F} ^ {\text {o f f}} (\mathbf {x}; \mathbf {V}, \gamma , c) = \frac {1}{\sqrt {m}} \sum_ {k = 1} ^ {m} \sum_ {i = 1} ^ {N} c _ {k, i} ^ {\text {o f f}} \cdot \sigma \left(\gamma_ {k, i} ^ {\text {o f f}} \cdot \frac {\mathbf {v} _ {k , i} ^ {\text {o f f}} {} ^ {\top} \mathbf {x}}{\| \mathbf {v} _ {k , i} ^ {\text {o f f}} \| \mathbf {s} _ {i}}\right) \cdot \mathbb {1} \left\{\mathbf {x} \in \mathbb {D} _ {i} \right\}. \tag {9}
$$
With online and offline models, we form our  $\mathrm{Fed - CO_2}$  as model F:
$$
\mathbf {F} (\mathbf {x}; \mathbf {V}, \gamma , c) = \frac {1}{2} \left(\mathbf {F} ^ {\text {o n}} (\mathbf {x}; \mathbf {V} ^ {\text {o n}}, \gamma^ {\text {o n}}, c ^ {\text {o n}}) + \mathbf {F} ^ {\text {o f f}} (\mathbf {x}; \mathbf {V} ^ {\text {o f f}}, \gamma^ {\text {o f f}}, c ^ {\text {o f f}})\right). \tag {10}
$$
In our analysis, we adopt one random strategy [36] to initialize parameters:
$$
\mathbf {v} _ {k, i} ^ {\text {o f f}} (0) = \mathbf {v} _ {k} ^ {\text {o n}} (0) \sim \mathcal {N} (0, \alpha^ {2} \mathbf {I}); c _ {k, i} ^ {\text {o f f}} = c _ {k} ^ {\text {o n}} \sim \operatorname {U n i f} \{- 1, 1 \}; \gamma_ {k, i} ^ {\text {o n}} = \gamma_ {k, i} ^ {\text {o f f}} = \frac {\left\| \mathbf {v} _ {k} ^ {\text {o n}} (0) \right\| _ {2}}{\alpha}, \tag {11}
$$
where  $\alpha^2$  controls the magnitude of  $\mathbf{v}_k^{\mathrm{on}}$  and  $\mathbf{v}_{k,i}^{\mathrm{off}}$  at initialization. The initialization of the BN parameters  $\gamma_{k,i}^{\mathrm{on}}$  and  $\gamma_{k,i}^{\mathrm{off}}$  are independent of  $\alpha$ . We use the MSE loss to train our model  $\mathbf{F}$ :
$$
\begin{array}{l} L (\mathbf {F}) = \frac {1}{N M} \sum_ {i = 1} ^ {N} \sum_ {j = 1} ^ {M} \left(\mathbf {F} (\mathbf {x} _ {i} ^ {j}) - y _ {j} ^ {i}\right) ^ {2} \\ = \frac {1}{N M} \sum_ {i = 1} ^ {N} \sum_ {j = 1} ^ {M} \left(\frac {\mathbf {F} ^ {\text {o n}} \left(\mathbf {x} _ {i} ^ {j}\right) + \mathbf {F} ^ {\text {o f f}} \left(\mathbf {x} _ {i} ^ {j}\right)}{2} - y _ {i} ^ {j}\right) ^ {2}. \tag {12} \\ \end{array}
$$
We employ NTK [35] to analyze the trajectory of networks  $\mathbf{F}$  learned by  $\mathrm{Fed - CO_2}$  and networks  $\mathbf{F}^{\mathrm{on}}$  learned by FedBN (the online model in  $\mathrm{Fed - CO_2}$ ). Existing studies [37, 38] have validated that the convergence rate of finite-width over-parameterized networks is controlled by the least eigenvalue of the induced kernel in the training process. Following the discovery in [38], we can decompose the NTK into a direction component  $\mathbf{V}(t) / \alpha^2$  and a magnitude component  $\mathbf{G}(t)$ :
$$
\frac {d \mathbf {F}}{d t} = - \boldsymbol {\Lambda} (t) (\mathbf {F} (t) - \mathbf {y}), \boldsymbol {\Lambda} (t) := \frac {\mathbf {V} (t)}{\alpha^ {2}} + \mathbf {G} (t). \tag {13}
$$
The specific forms of  $\mathbf{V}(t)$  and  $\mathbf{G}(t)$  are provided in the Appendix. Here, let  $\lambda_{\mathrm{min}}(H)$  denote the minimum eigenvalue of matrix  $H$ . It is worth noting that both matrices  $\mathbf{V}(t)$  and  $\mathbf{G}(t)$  are positive semi-definite as they can be interpreted as covariance matrices. Based on this, we can deduce that  $\lambda_{\mathrm{min}}(\Lambda(t)) \geq \max\{\lambda_{\mathrm{min}}(\mathbf{V}(t)/\alpha^2), \lambda_{\mathrm{min}}(\mathbf{G}(t))\}$ . From the NTK theory, the value of  $\lambda_{\mathrm{min}}(\Lambda(t))$  controls the convergence rate. Then, considering that  $\alpha$  is the pre-defined parameter, for  $\alpha \in (0,1)$ , convergence is dominated by  $\mathbf{V}(t)$ . Let  $\Lambda(t)$ ,  $\Lambda^{\mathrm{on}}(t)$  and  $\Lambda^{\mathrm{off}}(t)$  denote the evolution dynamics of Fed-  $\mathrm{CO}_2$ , the online model and the offline model, respectively. Based on the work in [10], the auxiliary version of the Gram matrices  $\mathbf{V}^{\infty}$ ,  $\mathbf{V}_{\mathrm{on}}^{\infty}$ , and  $\mathbf{V}_{\mathrm{off}}^{\infty}$  for three models are strictly positive definite. Let the least eigenvalues  $\lambda_{\mathrm{min}}(\mathbf{V}_{\mathrm{on}}^{\infty}) := \mu^{\mathrm{on}}$ ,  $\lambda_{\mathrm{min}}(\mathbf{V}_{\mathrm{off}}^{\infty}) := \mu^{\mathrm{off}}$ , and  $\lambda_{\mathrm{min}}(\mathbf{V}^{\infty}) := \mu$ , where  $\mu^{\mathrm{on}}, \mu^{\mathrm{off}}$ , and  $\mu$  are all positive values. To be specific, we define Gram matrices  $\mathbf{V}_{\mathrm{on}}^{\infty}$  and  $\mathbf{V}_{\mathrm{off}}^{\infty}$  in Definition 4.2.
Definition 4.2 Given sample points  $\{\mathbf{x}_p\}_{p=1}^{NM}$ , we define the auxiliary Gram matrices  $\mathbf{V}_{\mathrm{on}}^{\infty} \in \mathbb{R}^{NM \times NM}$  and  $\mathbf{V}_{\mathrm{off}}^{\infty} \in \mathbb{R}^{NM \times NM}$  as
$$
\mathbf {V} _ {\mathrm {o n} _ {\mathrm {p q}}} ^ {\infty} := \mathbb {E} _ {\mathbf {v} \sim \mathcal {N} (\mathbf {0}, \alpha^ {2} \mathbf {I})} \left(\alpha c\right) ^ {2} \mathbf {x} _ {p} ^ {\mathbf {v} ^ {\perp}} \mathbf {x} _ {q} ^ {\mathbf {v} ^ {\perp}}, \quad (O n l i n e M o d e l) \tag {14}
$$
$$
\mathbf {V} _ {\text {o f f} _ {\mathrm {p q}}} ^ {\infty} := \mathbb {E} _ {\mathbf {v} \sim \mathcal {N} (\mathbf {0}, \alpha^ {2} \mathbf {I})} \left(\alpha c\right) ^ {2} \mathbf {x} _ {p} ^ {\mathbf {v} ^ {\perp}} \mathbf {x} _ {q} ^ {\mathbf {v} ^ {\perp}} \mathbb {1} \left\{i _ {p} = i _ {q} \right\}, (O f f l i n e M o d e l). \tag {15}
$$
With the prerequisite provided in Appendix, the convergence performance of the online model, the offline model, and our Fed-  $\mathrm{CO}_{2}$  can be analyzed by comparing  $\lambda_{\mathrm{min}}(\mathbf{V}_{\mathrm{on}}^{\infty}),\lambda_{\mathrm{min}}(\mathbf{V}_{\mathrm{off}}^{\infty})$  , and  $\lambda_{\mathrm{min}}(\mathbf{V}^{\infty})$  . Our main theoretical result is given in Theorem 4.3.
Theorem 4.3 For the  $V$ -dominated convergence, the convergence rate of  $\text{Fed-CO}_2$  is faster than that of FedBN (the online model in  $\text{Fed-CO}_2$ ).
Proof sketch The core is to show  $\lambda_{\mathrm{min}}(\mathbf{V}_{\mathrm{on}}^{\infty}) \leq \lambda_{\mathrm{min}}(\mathbf{V}^{\infty})$ . To prove the theorem, we first show that  $\lambda_{\mathrm{min}}(\mathbf{V}_{\mathrm{on}}^{\infty}) \leq \lambda_{\mathrm{min}}(\mathbf{V}_{\mathrm{off}}^{\infty})$ . Comparing Eq. 14 and 15,  $\mathbf{V}_{\mathrm{off}}^{\infty}$  takes the  $\mathbf{M} \times \mathbf{M}$  block matrices on the diagonal of  $\mathbf{V}_{\mathrm{on}}^{\infty}$ . Let  $\mathbf{V}_i^\infty$  be the  $i$ -th  $\mathbf{M} \times \mathbf{M}$  block matrices on the diagonal of  $\mathbf{V}_{\mathrm{on}}^{\infty}$ . According to linear algebra, we have  $\lambda_{\mathrm{min}}(\mathbf{V}_i^\infty) \geq \lambda_{\mathrm{min}}(\mathbf{V}_{\mathrm{on}}^{\infty})$ , for  $i \in [N]$ . With  $\mathbf{V}_{\mathrm{off}}^{\infty} = \text{diag}(\mathbf{V}_1^{\infty}, \dots, \mathbf{V}_N^{\infty})$ , it can be obtained that  $\lambda_{\mathrm{min}}(\mathbf{V}_{\mathrm{off}}^{\infty}) \geq \lambda_{\mathrm{min}}(\mathbf{V}_{\mathrm{on}}^{\infty})$ . Then, based on Eq. 10, 12, and 13, we can obtain  $\lambda_{\mathrm{min}}(\mathbf{V}^{\infty}) = \lambda_{\mathrm{min}}(\frac{1}{2} (\mathbf{V}_{\mathrm{on}}^{\infty} + \mathbf{V}_{\mathrm{off}}^{\infty}))$ . Therefore, we have  $\lambda_{\mathrm{min}}(\mathbf{V}^{\infty}) \geq (\frac{1}{2}\lambda_{\mathrm{min}}(\mathbf{V}_{\mathrm{on}}^{\infty}) + \frac{1}{2}\lambda_{\mathrm{min}}(\mathbf{V}_{\mathrm{off}}^{\infty}))$ . Since we have proven that  $\lambda_{\mathrm{min}}(\mathbf{V}_{\mathrm{off}}^{\infty}) \geq \lambda_{\mathrm{min}}(\mathbf{V}_{\mathrm{on}}^{\infty})$ , the result  $\lambda_{\mathrm{min}}(\mathbf{V}^{\infty}) \geq \lambda_{\mathrm{min}}(\mathbf{V}_{\mathrm{on}}^{\infty})$  can be achieved.
5 EXPERIMENTS
Dataset and Data Heterogeneity. We followed the footstep of prior research [39, 7, 5, 13] to study the label distribution skew heterogeneity with image datasets: CIFAR10 and CIFAR100 [40]. Specifically, we considered two different label distributions among participants: 1) Pathological Distribution, each client is randomly assigned 2 classes per client in CIFAR10 (10 classes per client in CIFAR100); 2) Dirichlet Distribution, each client gets its private data through partitioning of the datasets using a symmetric Dirichlet distribution with a default parameter  $\alpha = 0.3$ .
For feature skew heterogeneity, we conducted extensive experiments on three datasets: Digits, Office-Caltech10 [41], and DomainNet [42]. In this non-IID data setting, each domain serves as a client, with each client having access to all the data from its respective domain. Digits is composed of 5 different digit datasets with feature shift: SVHN [43], USPS [44], SynthDigits [45], MNIST-M [45] and MNIST [46]. Office-Caltech10 [41] owns data from four different domains including Amazon, DSLR, WebCam, and Caltech-256. DomainNet [42] is a challenging dataset that comprises six distinct domains, namely Clipart, Infograph, Painting, Quickdraw, Real, and Sketch. To explore the more realistic FL scenarios with both label distribution skew and feature skew, we exert a Dirichlet Distribution to datasets with feature shift. More details are supplemented in Appendix.
Compared Benchmarks. We compared our universal framework Fed-  $\mathbf{CO}_2$  with fundamental algorithms including SingleSet, FedAvg [2] and FedProx [4], together with some refined FL algorithms: FedPer [11], MOON [7], FedBN [10], and FedRoD [12]. Here, SingleSet means each client trains their own model on their private data. We applied the linear version for FedRoD. For FL with feature skew issues, we further compared COPA [34], which was originally proposed to solve FDG and FDA problems. We now train the model on all domains in the dataset and evaluate its performance on each individual training client using the learned universal model.
5.2 Performance Evaluation
Model Evaluation on Feature Skew. Experiments are conducted on datasets Digits, Office-Caltech10, and DomainNet. Here, we present the results of experiments on Office-Caltech10, and DomainNet in Table 1 and Table 2. Full experiments are shown in Appendix. The performance of our  $\mathrm{Fed - CO_2}$  exhibits a dominant edge over state-of-the-art algorithms in all experiments, where the average accuracy on each domain increases by nearly  $4\%$ . The results confirm that our novel collaboration framework between online and offline models effectively addresses FL challenges associated with severe feature skew data heterogeneity.
Remarkably, when compared to algorithms FedBN [10] and COPA [34], which are explicitly designed to handle feature shift issues, our method consistently outperforms them across all sub-datasets. This phenomenon provides compelling evidence that our cooperation mechanism adapts to local feature shifts more effectively by leveraging both general domain-invariant and specialized domain-specific knowledge.
Model Evaluation on Label Distribution Skew. As shown in Table 3, our  $\mathrm{Fed - CO_2}$  outperforms a variety of state-of-the-art PFL algorithms in terms of average test accuracy across almost all cases in experiments with label distribution skew. Our state-of-the-art results across almost all cases
Table 1: Experiment results for FL with Feature Skew on Office-Caltech10.
Methods	Office-Caltech10
Amazon	Caltech	DSLR	WebCam	Avg
SingleSet	54.9±1.5	40.2±1.6	78.7±1.3	86.4±2.4	65.1±1.7
FedAvg [2]	54.1±1.1	44.8±1.0	66.9±1.5	85.1±2.9	62.7±1.6
FedProx [4]	54.2±2.5	44.5±0.5	65.0±3.6	84.4±1.7	62.0±2.1
FedPer [11]	49.0±1.2	37.1±2.4	57.7±3.7	79.7±2.1	56.0±1.1
MOON [7]	57.3±0.7	44.4±0.5	76.2±2.5	83.1±1.1	65.2±0.5
FedRoD [12]	60.4±2.3	45.3±0.9	73.7±2.5	83.7±2.3	65.8±1.4
COPA [34]	51.9±2.5	46.7±0.8	65.6±2.0	85.0±1.3	62.3±0.9
FedBN [10]	63.0±1.6	45.3±1.5	83.1±2.5	90.5±2.3	70.5±2.0
Fed-CO2	63.0±1.6	49.1±0.7	89.4±2.5	96.6±1.5	74.5±0.3
Table 2: Experiment results for FL with Feature Skew on DomainNet.
Methods	DomainNet
Clipart	Infograph	Painting	Quickdraw	Real	Sketch	Avg
SingleSet	41.0±0.9	23.8±1.2	36.2±2.7	73.1±0.9	48.5±1.9	34.0±1.1	42.8±1.5
FedAvg [2]	48.8±1.9	24.9±0.7	36.5±1.1	56.1±1.6	46.3±1.4	36.6±2.5	41.5±1.5
FedProx [4]	48.9±0.8	24.9±1.0	36.6±1.8	54.4±3.1	47.8±0.8	36.9±2.1	41.6±1.6
FedPer [11]	40.4±0.8	25.7±0.6	37.3±0.6	62.5±1.2	47.4±0.5	32.8±0.8	41.0±0.3
MOON [7]	52.5±1.1	25.7±0.6	39.4±1.7	50.8±4.7	48.8±0.8	40.1±4.1	42.9±1.5
FedRoD [12]	50.8±1.6	26.3±0.2	40.1±1.8	66.8±1.8	51.5±1.1	39.1±2.0	45.7±0.7
COPA [34]	51.1±1.0	24.7±1.2	36.8±0.8	54.8±1.6	47.1±1.8	41.0±1.4	42.6±0.4
FedBN [10]	51.2±1.4	26.8±0.5	41.5±1.4	71.3±0.7	54.8±0.8	42.1±1.3	48.0±1.0
Fed-CO2	55.0±1.1	28.6±1.1	44.3±0.6	75.1±0.6	62.4±0.8	45.7±1.9	51.8±0.2
prove that our method can more effectively excavate and utilize local information to overcome data heterogeneity caused by label distribution imbalance than previous methods. It is worth highlighting that our  $\mathrm{Fed - CO_2}$  outperforms both SingleSet and FedBN, regardless of which one performs better individually. This observation confirms that  $\mathrm{Fed - CO_2}$  effectively integrates the general knowledge obtained by the online model and the specialized knowledge acquired by the offline model, resulting in enhanced performance.
Table 3: Experiment results for FL with Label Distribution Skew on CIFAR10 and CIFAR100. Experiments are conducted with two kinds of label distribution data heterogeneity: Pathological setting and Dirichlet setting.
Methods	CIFAR10	CIFAR100
Pathological	Dirichlet	Pathological	Dirichlet
SingleSet	85.85±0.05	68.38±0.06	49.54±0.05	21.39±0.05
FedAvg [2]	44.12±3.10	57.52±1.01	14.59±0.40	20.34±1.34
FedProx [4]	57.38±1.08	56.46±0.66	21.32±0.71	19.40±1.76
FedPer [11]	80.99±0.71	74.21±0.07	42.08±0.18	20.06±0.26
MOON [7]	48.43±3.18	54.49±1.87	17.89±0.76	19.73±0.71
FedRoD [12]	89.05±0.04	73.99±0.09	54.96±1.30	28.29±1.53
FedBN [10]	86.71±0.56	75.41±0.37	48.37±0.56	28.70±0.46
Fed-CO2	88.79±0.25	77.45±0.30	58.50±0.43	32.43±0.37
Model Evaluation on Label Distribution Skew and Feature Skew. Here, we evaluate our  $\mathrm{Fed - CO_2}$  and benchmark algorithms in FL scenarios with label distribution skew and feature skew. We exhibit the experiment results on Digits in Table 4, where we exerted a Dirichlet Distribution on each client with  $\alpha = 0.3$ . From the results, it is evident that, even in the presence of two types of data heterogeneity, our  $\mathrm{Fed - CO_2}$  outperforms various state-of-the-art algorithms with a significant margin in every sub-dataset. Therefore, we can conclude that under our universal cooperation framework, local clients not only make the most use of their local offline information but also benefit from the general knowledge shared by other clients for a better adaptation to local data, resulting in state-of-the-art performance in FL with both label distribution skew and feature skew. More experiments are provided in Appendix.
Table 4: Experiment results for FL with both Label Skew and Feature Skew on Digits.
Methods	Digits
MNIST	SVHN	USPS	SynthDigits	MNIST-M	Avg
SingleSet	83.75±7.58	74.63±0.31	97.14±0.06	87.95±0.46	80.55±0.26	84.80±1.44
FedAvg [2]	89.27±6.39	57.23±5.43	94.60±1.05	81.30±2.51	71.71±7.59	78.82±4.03
FedProx [4]	87.09±7.83	53.40±7.38	90.55±7.47	78.40±7.05	69.98±6.55	75.88±6.90
FedPer [11]	96.92±0.02	72.86±0.11	97.09±0.12	87.82±0.02	83.69±0.07	87.68±0.02
MOON [7]	96.58±0.07	74.10±0.22	96.19±0.10	88.15±0.04	85.05±0.14	88.01±0.07
FedRoD [12]	96.65±0.06	77.05±0.16	96.88±0.13	89.59±0.06	86.18±0.11	89.27±0.05
COPA [34]	96.82±0.08	78.32±0.12	96.54±0.15	89.36±0.04	87.46±0.11	89.70±0.06
FedBN [10]	92.68±3.45	70.26±4.38	91.40±8.72	83.17±4.95	77.98±3.84	83.10±4.96
Fed-CO2	97.17±0.67	83.16±0.23	98.12±0.13	93.04±0.11	91.45±0.14	92.59±0.15
5.3 Component Analysis
In the presence of feature skew, we add intra-client and inter-client knowledge transfer mechanisms to foster the collaboration between the online and offline models for better local adaptation. Here, we evaluate the efficacy of these two knowledge transfer mechanisms on the challenging DomainNet dataset through three additional experiments, as presented in Table 5. In these experiments, we selectively removed specific mechanisms from  $\mathrm{Fed - CO_2}$  to investigate their individual contributions and functionalities.
According to the experimental results, we can observe that: 1) In the absence of intra-client and inter-client knowledge transfer mechanisms,  $\mathrm{Fed - CO_2}$  exhibits a significant decline in performance, resulting in a performance level comparable to that of benchmark methods. Therefore, cooperation relying solely on prediction fusion proves to be inadequate for FL with feature skew. 2) The removal of either the intra-client or the inter-client knowledge transfer mechanisms will also result in an average decline in performance. This phenomenon validates the effectiveness of our intra-client and inter-client knowledge transfer mechanisms, both of which contribute to enhancing the models' capability to adapt to local data distributions. 3) Compared with all three ablation experiments, our  $\mathrm{Fed - CO_2}$  consistently achieves the highest performance on average and across the majority of sub-datasets. These results firmly demonstrate that our cooperation framework effectively leverages both model-level and client-level collaboration through the intra-client knowledge transfer and inter-client knowledge transfer mechanisms, respectively, to adapt to feature shifts resulting from domain gaps. Further analysis of these two cooperation mechanisms is supplemented in the Appendix.
Table 5: Ablation study of Intra-client and Inter-client knowledge transfer mechanisms for FL with Feature Skew Data Heterogeneity on DomainNet.
Methods	Clipart	Infograph	Painting	Quickdraw	Real	Sketch	Avg
Fed-CO2 w/o Intra and Inter Transfer	48.75±0.94	26.49±2.05	42.10±1.05	72.86±0.80	57.12±1.08	39.96±0.79	47.88±0.70
Fed-CO2 w/o Intra Transfer	53.88±0.63	26.18±0.71	42.94±0.92	75.10±0.28	61.94±0.70	46.68±0.75	51.12±0.19
Fed-CO2 w/o Inter Transfer	50.42±0.72	26.97±1.07	43.94±0.69	74.14±0.64	58.14±0.85	42.02±0.80	49.27±0.46
Fed-CO2	55.02±1.13	28.58±1.10	44.27±0.62	75.08±0.62	62.37±0.76	45.67±1.95	51.83±0.25
6 CONCLUSIONS
In this work, we proposed a new universal FL framework called Fed-  $\mathrm{CO}_{2}$  that is capable of handling label distribution skew and feature skew even when both are present in the same data. The core of our approach is to foster cooperation between the online and offline models, leveraging the benefits of online general knowledge and offline specialized knowledge to effectively adapt to local data distribution. To improve model collaboration in FL with feature shifts, we designed two novel knowledge transfer mechanisms: one intra-client and the other inter-client. These mechanisms facilitate mutual learning between the online and offline models, concurrently enhancing the model's capacity to generalize across different domains. Comparisons with a wide range of state-of-the-art methods on five benchmark datasets consistently show that Fed-  $\mathrm{CO}_{2}$  yields superior performance in addressing both label distribution skew and feature skew challenges, both individually and in combination. Extending Fed-  $\mathrm{CO}_{2}$  to scenarios with noisy training data is under consideration in our future work.
[1] Qiang Yang, Yang Liu, Tianjian Chen, and Yongxin Tong. Federated machine learning: Concept and applications. ACM Transactions on Intelligent Systems and Technology (TIST), 10(2):1-19, 2019.
[2] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273-1282. PMLR, 2017.
[3] Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. Foundations and Trends® in Machine Learning, 14(1-2):1-210, 2021.
[4] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems, 2:429-450, 2020.
[5] Othmane Marfoq, Giovanni Neglia, Richard Vidal, and Laetitia Kameni. Personalized federated learning through local memorization. In International Conference on Machine Learning, pages 15070-15092. PMLR, 2022.
[6] Aviv Shamsian, Aviv Navon, Ethan Fetaya, and Gal Chechik. Personalized federated learning using hypernetworks. In International Conference on Machine Learning, pages 9489-9502. PMLR, 2021.
[7] Qinbin Li, Bingsheng He, and Dawn Song. Model-contrastive federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10713–10722, 2021.
[8] Benyuan Sun, Hongxing Huo, Yi Yang, and Bo Bai. Partialfed: Cross-domain personalized federated learning via partial initialization. Advances in Neural Information Processing Systems, 34:23309-23320, 2021.
[9] Wang Lu, Jindong Wang, Yiqiang Chen, Xin Qin, Renjun Xu, Dimitrios Dimitriadis, and Tao Qin. Personalized federated learning with adaptive batchnorm for healthcare. IEEE Transactions on Big Data, 2022.
[10] Xiaoxiao Li, Meirui Jiang, Xiaofei Zhang, Michael Kamp, and Qi Dou. Fedbn: Federated learning on non-iid features via local batch normalization. International Conference on Learning Representations, 2021.
[11] Manoj Ghuhan Arivazhagan, Vinay Aggarwal, Aaditya Kumar Singh, and Sunav Choudhary. Federated learning with personalization layers. arXiv preprint arXiv:1912.00818, 2019.
[12] Hong-You Chen and Wei-Lun Chao. On bridging generic and personalized federated learning for image classification. International Conference on Learning Representations, 2022.
[13] Hongxia Li, Zhongyi Cai, Jingya Wang, Jiangnan Tang, Weiping Ding, Chin-Teng Lin, and Ye Shi. FedTP: Federated Learning by Transformer Personalization. IEEE Transactions on Neural Networks and Learning Systems, pages 1-15, 2023.