SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[49] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 60(6):84-90, 2017.
[50] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
A Dataset Setup Details
The details of non-IID settings of mentioned datasets are presented in the section. Our two settings of label distribution skew, namely the Pathological setting and the Dirichlet setting are similar to preceding works [5, 13]. In the Pathological setting, where each client  $i$  is randomly assigned two (ten) classes from CIFAR10 (CIFAR100), we first sampled  $q_{c}^{i} \sim U(0.4, 0.6)$  for the selected class  $c$  and then calculated the sample rate for this class  $c$  on client  $i$  with  $sr_{c}^{i} = \frac{q_{c}^{i}}{\sum_{j} q_{c}^{j}}$ . In the Dirichlet setting, we partitioned the datasets randomly utilizing a symmetric Dirichlet distribution with default parameter  $\alpha = 0.3$ . Specifically, for CIFAR100, we leveraged its coarse labels following previous works [5, 39]. In detail, a two-stage Pachinko allocation method was adopted to partition samples in CIFAR100. For each client, this method first generates a Dirichlet distribution with default parameter  $\alpha = 0.3$  over the coarse labels and then generates another Dirichlet distribution with parameter  $\beta = 10$  over the corresponding fine labels. The local training batch size was set to 64. In both non-IID partitions, the classes and their distribution in each client's training and test sets remain consistent.
We adopted the setting used in FedBN [10] for feature skew heterogeneity, which involved adjusting the local training batch size to 32 to maintain consistency. Additionally, we followed the implementation of selecting only the top 10 classes based on data amount from DomainNet to create the dataset for our experiments. For experiments with label distribution skew and feature skew, we partitioned each sub-dataset into several parts with a Dirichlet Distribution and selected one part as the new training set. Then we adjusted the label distribution in the test set to the same distribution in the new training set. Specifically, based on the number of training images in each dataset, we divided each sub-dataset in Digits into three parts, each sub-dataset in Office-Caltech10 into two parts, and each sub-dataset in DomainNet into five parts. Table 6 summarizes the datasets and the number of clients.
Table 6: Datasets Statistics.
Dataset	Label Distribution Skew	Feature Skew	Client Number
CIFAR10	✓	100
CIFAR100	✓	100
Digits	✓	5
Office-Caltech10	✓	4
DomainNet	✓	6
Network Modules. Similar to prior works [47, 48], we adopted a ConvNet [46] with two convolutional layers and three fully-connected layers for methods in experiments conducted under label distribution skew setting. With regards to feature skew setting, experiments were performed with a backbone similar to FedBN [10]. In other words, for the Digits dataset, we applied a ConvNet [46] with four convolutional layers and three fc layers. For Office-Caltech10 and DomainNet dataset, we applied one Alexnet [49] as the default model backbone.
In our experimental evaluations of model performance under label distribution skew heterogeneity, we added an additional BN layer after each convolutional layer and fully connected (fc) layer, except for the last layer to algorithms: FedBN [10] and  $\mathrm{Fed - CO_2}$ . Meanwhile, in experiments pertaining to feature skew heterogeneity, as our default network incorporated a BN layer after each convolutional and fc layer (except for the last layer), we omitted the BNs in our classifiers for methods other than FedBN [10] and  $\mathrm{Fed - CO_2}$ . All other ConvNet network architectures used for the Digits dataset, as well as the Alexnet architecture employed for Office-Caltech10 and DomainNet, are identical to those used in FedBN [10].
Concrete Implementation. For experiments evaluating model performance on label distribution skew, we set up 100 local clients and randomly sampled  $5\%$  of them to participate in each communication round. We trained every algorithm with 1500 communication rounds. Meanwhile, in experiments about feature skew non-iid heterogeneity, all clients participated in each communication round and every algorithm was trained with 300 communication rounds. All training tasks were optimized with an SGD optimizer with the default learning rate 0.01. We used PyTorch [50] to implement our algorithms with an RTX 2080 Ti GPU.
For experiments on CIFAR10 and CIFAR100, we tested the model performance every 5 rounds during its last 200 communication rounds. The average and variance of test accuracy were reported. For all experiments with feature skew, we run 5 times and reported its mean and variance.
Evaluation Metrics. We evaluated the learned personal models on the private test data of each client. We examined the model's performance on each domain using the same method as FedBN [10] in FL with feature skew. Furthermore, for experiments on label distribution skew, we calculated the accuracy using the same protocol as FedTP [13].
C Convergence
In this section, we first provide the preliminary and complete theorem proof. Then, we empirically demonstrate that our  $\mathrm{Fed - CO_2}$  shows faster and more robust convergence performance than FedBN and other benchmark methods on FL issues with Feature Skew.
C.1 Preliminary
Following the V-dominated convergence rate for FedAvg (Theorem C.1) in [38], we can derive the convergence rate for the online model, the offline model, and Fed-  $\mathrm{CO}_{2}$  Corollary C.2.
Theorem C.1 (V-dominated convergence for FedAvg [38]) Suppose the network is initialized as in Eq. 11 with  $\alpha \leq 1$ , trained using gradient descent and Assumption 4.1 holds. Assuming the loss function used for training the neural network is the square loss, and the target values  $\mathbf{y}$  satisfy  $\| \mathbf{y}\|_{\infty} = O(1)$ . If  $m = \Omega (N^4 M^4\log (NM / \delta) / \mu_0^4)$ , then with probability  $1 - \delta$
1. for iterations  $t = 0,1,\dots$ , the evolution matrix  $\Lambda (t)$  satisfies  $\lambda_{\mathrm{min}}(\Lambda (t))\geq \frac{\mu_0}{2\alpha^2}$
2. training with gradient descent of step-size  $\eta = O\left(\frac{\alpha^2}{\|\mathbf{V}^\infty\|_2}\right)$  converges linearly as
$$
\| \mathbf {f} (t) - \mathbf {y} \| _ {2} ^ {2} \leq \left(1 - \frac {\eta \mu_ {0}}{2 \alpha^ {2}}\right) ^ {t} \| \mathbf {f} (0) - \mathbf {y} \| _ {2} ^ {2}.
$$
Corollary C.2 Under the same assumption in Theorem C.1, with probability  $1 - \delta$ , for iterations  $t = 0, 1, \dots$ , we have
1. The evolution matrix  $\Lambda^{\mathrm{on}}(t)$  satisfies  $\lambda_{\mathrm{min}}(\Lambda^{\mathrm{on}}(t)) \geq \frac{\mu^{\mathrm{on}}}{2\alpha^2}$  and training with gradient descent of step-size  $\eta = O\left(\frac{\alpha^2}{\|\mathbf{V}_{\mathrm{on}}^{\infty}\|_2}\right)$  converges linearly as  $\| \mathbf{F}^{\mathrm{on}}(t) - \mathbf{y}\| _2^2 \leq \left(1 - \frac{\eta\mu^{\mathrm{on}}}{2\alpha^2}\right)^t\| \mathbf{F}^{\mathrm{on}}(0) - \mathbf{y}\| _2^2$ .
2. The evolution matrix  $\Lambda^{\mathrm{off}}(t)$  satisfies  $\lambda_{\mathrm{min}}(\Lambda^{\mathrm{off}}(t)) \geq \frac{\mu^{\mathrm{off}}}{2\alpha^2}$  and training with gradient descent of step-size  $\eta = O\left(\frac{\alpha^2}{\|\mathbf{V}_{\mathrm{off}}^{\infty}\|_2}\right)$  converges linearly as  $\| \mathbf{F}^{\mathrm{off}}(t) - \mathbf{y}\| _2^2 \leq \left(1 - \frac{\eta\mu^{\mathrm{off}}}{2\alpha^2}\right)^t\| \mathbf{F}^{\mathrm{off}}(0) - \mathbf{y}\| _2^2$ .
3. The evolution matrix  $\Lambda(t)$  satisfies  $\lambda_{\min}(\Lambda(t)) \geq \frac{\mu}{2\alpha^2}$  and training with gradient descent of step-size  $\eta = O\left(\frac{\alpha^2}{\|\mathbf{V}^\infty\|_2}\right)$  converges linearly as  $\|\mathbf{F}(t) - \mathbf{y}\|_2^2 \leq \left(1 - \frac{\eta\mu}{2\alpha^2}\right)^t \|\mathbf{F}(0) - \mathbf{y}\|_2^2$ .
Therefore, the exponential factor of convergence for the online model, the offline model, and  $\mathrm{Fed - CO_2}$  are controlled by the smallest eigenvalue of  $\mathbf{V}^{\mathrm{on}}(t)$ ,  $\mathbf{V}^{\mathrm{off}}(t)$ , and  $\mathbf{V}(t)$ , respectively. The convergence performance of the online model, the offline model, and our  $\mathrm{Fed - CO_2}$  can be analyzed by comparing  $\lambda_{\min}(\mathbf{V}_{\mathrm{on}}^{\infty})$ ,  $\lambda_{\min}(\mathbf{V}_{\mathrm{off}}^{\infty})$ , and  $\lambda_{\min}(\mathbf{V}^{\infty})$ .
C.2 Theorem Proof
To prove Theorem 4.3, we will first prove SingleSet (the offline model in  $\mathrm{Fed - CO_2}$ ) converges faster than FedBN (the online model in  $\mathrm{Fed - CO_2}$ ). Gram matrix  $\mathbf{V}^{\mathrm{on}}(t)$  and  $\mathbf{G}^{\mathrm{on}}(t)$  of FedBN can be
obtained from work [10] as:
$$
\begin{array}{l} \mathbf {V} _ {p q} ^ {\mathrm {o n}} (t) = \frac {1}{m} \sum_ {k = 1} ^ {m} \left(\alpha c _ {k} ^ {\mathrm {o n}}\right) ^ {2} \gamma_ {k, i _ {p}} ^ {\mathrm {o n}} (t) \gamma_ {k, i _ {q}} ^ {\mathrm {o n}} (t) \| \mathbf {v} _ {k} ^ {\mathrm {o n}} (t) \| _ {\mathbf {S} _ {i _ {p}}} ^ {- 1} \| \mathbf {v} _ {k} ^ {\mathrm {o n}} (t) \| _ {\mathbf {S} _ {i _ {q}}} ^ {- 1} \left\langle \mathbf {x} _ {p} ^ {\mathbf {v} _ {k} ^ {\mathrm {o n} ^ {\mathrm {i p}}} (t) ^ {\perp}}, \mathbf {x} _ {q} ^ {\mathbf {v} _ {k} ^ {\mathrm {o n} ^ {\mathrm {i q}}} (t) ^ {\perp}} \right\rangle \tag {16} \\ \mathbb {1} _ {p k} (t) \mathbb {1} _ {q k} (t), \\ \end{array}
$$
$$
\mathbf {G} _ {p q} ^ {\mathrm {o n}} (t) = \frac {1}{m} \sum_ {k = 1} ^ {m} c _ {k} ^ {\mathrm {o n} ^ {2}} \| \mathbf {v} _ {k} ^ {\mathrm {o n}} (t) \| _ {\mathbf {S} _ {i _ {p}}} ^ {- 1} \| \mathbf {v} _ {k} ^ {\mathrm {o n}} (t) \| _ {\mathbf {S} _ {i _ {q}}} ^ {- 1} \sigma \left(\mathbf {v} _ {k} ^ {\mathrm {o n}} (t) ^ {\top} \mathbf {x} _ {p}\right) \sigma \left(\mathbf {v} _ {k} ^ {\mathrm {o n}} (t) ^ {\top} \mathbf {x} _ {q}\right) \mathbb {1} \left\{i _ {p} = i _ {q} \right\}, \tag {17}
$$
where  $\mathbb{1}_{pk}(t)\coloneqq \mathbb{1}_{\{\mathbf{v}_k(t)^\top \mathbf{x}_p\geq 0\}}$
Similarly, the Gram matrix  $\mathbf{V}^{\mathrm{off}}(t)$  for method SingleSet with  $\mathbf{F}^{\mathrm{off}}$  can be computed as:
$$
\begin{array}{l} \mathbf {V} _ {p q} ^ {\text {o f f}} (t) = \frac {1}{m} \sum_ {k = 1} ^ {m} \alpha^ {2} c _ {k, i _ {p}} ^ {\text {o f f}} c _ {k, i _ {q}} ^ {\text {o f f}} \gamma_ {k, i _ {p}} ^ {\text {o f f}} (t) \gamma_ {k, i _ {q}} ^ {\text {o f f}} (t) \left\| \mathbf {v} _ {k, i _ {p}} ^ {\text {o f f}} (t) \right\| _ {\mathbf {S} _ {i _ {p}}} ^ {- 1} \left\| \mathbf {v} _ {k, i _ {q}} ^ {\text {o f f}} (t) \right\| _ {\mathbf {S} _ {i _ {q}}} ^ {- 1} \tag {18} \\ \left\langle \mathbf {x} _ {p} ^ {\mathbf {v} _ {k, i _ {p}} ^ {\mathrm {o f f} ^ {i _ {p}}} (t) ^ {\perp}}, \mathbf {x} _ {q} ^ {\mathbf {v} _ {k, i _ {q}} ^ {\mathrm {o f f} ^ {i _ {q}}} (t) ^ {\perp}} \right\rangle \mathbb {1} _ {p k} (t) \mathbb {1} _ {q k} (t) \mathbb {1} \{i _ {p} = i _ {q} \}. \\ \end{array}
$$
To compare the convergence rates of SingleSet and FedBN, we compare the exponential factor in the convergence rates, which are  $(1 - \frac{\eta\mu^{\mathrm{off}}}{2\alpha^{2}})$  and  $(1 - \frac{\eta\mu^{\mathrm{on}}}{2\alpha^{2}})$ , respectively. Then, considering that  $\alpha$  is the pre-defined parameter, it reduces to comparing  $\mu^{\mathrm{on}} = \lambda_{min}(\mathbf{V}_{\mathrm{on}}^{\infty})$  and  $\mu^{\mathrm{off}} = \lambda_{min}(\mathbf{V}_{\mathrm{off}}^{\infty})$ . Comparing Eq. 16 and 18,  $\mathbf{V}_{\mathrm{off}}^{\infty}$  takes the  $\mathbf{M} \times \mathbf{M}$  block matrices on the diagonal of  $\mathbf{V}_{\mathrm{on}}^{\infty}$ :
$$
\mathbf {V} _ {\mathrm {o n}} ^ {\infty} = \left[ \begin{array}{c c c c} \mathbf {V} _ {1} ^ {\infty} & \mathbf {V} _ {1, 2} ^ {\infty} & \dots & \mathbf {V} _ {1, N} ^ {\infty} \\ \mathbf {V} _ {1, 2} ^ {\infty} & \mathbf {V} _ {2} ^ {\infty} & \dots & \mathbf {V} _ {2, N} ^ {\infty} \\ \vdots & \vdots & \ddots & \vdots \\ \mathbf {V} _ {1, N} ^ {\infty} & \mathbf {V} _ {2, N} ^ {\infty} & \dots & \mathbf {V} _ {N} ^ {\infty} \end{array} \right], \quad \mathbf {V} _ {\mathrm {o f f}} ^ {\infty} = \left[ \begin{array}{c c c c} \mathbf {V} _ {1} ^ {\infty} & 0 & \dots & 0 \\ 0 & \mathbf {V} _ {2} ^ {\infty} & \dots & 0 \\ \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & \dots & \mathbf {V} _ {N} ^ {\infty} \end{array} \right],
$$
where  $\mathbf{V}_i^\infty$  is the i-th  $M\times M$  block matrices on the diagonal of  $\mathbf{V}_{\mathrm{on}}^{\infty}$ . Therefore, we have:
$$
\lambda_ {\min } \left(\mathbf {V} _ {i} ^ {\infty}\right) \geq \lambda_ {\min } \left(\mathbf {V} _ {\mathrm {o n}} ^ {\infty}\right), \quad \forall i \in [ N ].
$$
Since the eigenvalues of  $\mathbf{V}_{\mathrm{off}}^{\infty}$  are exactly the union of eigenvalues of  $\mathbf{V}_i^\infty$ , we get:
$$
\begin{array}{l} \lambda_ {\min } \left(\mathbf {V} _ {\text {o f f}} ^ {\infty}\right) = \min  _ {i \in [ N ]} \left\{\lambda_ {\min } \left(\mathbf {V} _ {i} ^ {\infty}\right) \right\}, \\ \geq \lambda_ {\min } (\mathbf {V} _ {\mathrm {o n}} ^ {\infty}). \\ \end{array}
$$
Hence,  $\left(1 - \frac{\eta\mu^{\mathrm{on}}}{2\alpha^{2}}\right)\geq \left(1 - \frac{\eta\mu^{\mathrm{off}}}{2\alpha^{2}}\right)$  and we get the first-stage conclusion that the convergence rate of method SingleSet is faster than that in FedBN.
Then we go to prove  $\mathrm{Fed - CO_2}$  is faster than FedBN, as well. With Eq. 10, 12, and 13, we get:
$$
\begin{array}{l} \frac {d \mathbf {F}}{d t} = \frac {1}{2} \left(\frac {d \mathbf {F} ^ {\mathrm {o n}}}{d t} + \frac {d \mathbf {F} ^ {\mathrm {o f f}}}{d t}\right) \\ = - \left(\frac {1}{2} \boldsymbol {\Lambda} ^ {\text {o n}} (t) + \frac {1}{2} \boldsymbol {\Lambda} ^ {\text {o f f}} (t)\right) \left(\frac {1}{2} \left(\mathbf {F} ^ {\text {o n}} (t) + \mathbf {F} ^ {\text {o f f}} (t)\right) - \mathbf {y}\right) \tag {19} \\ = - \left(\frac {1}{2} \boldsymbol {\Lambda} ^ {\mathrm {o n}} (t) + \frac {1}{2} \boldsymbol {\Lambda} ^ {\mathrm {o f f}} (t)\right) (\mathbf {F} (t) - \mathbf {y}). \\ \end{array}
$$
Based on Eq. 19, we get the following relationships among three models:
$$
\begin{array}{l} \boldsymbol {\Lambda} (t) := \frac {1}{2} \left(\boldsymbol {\Lambda} ^ {\mathrm {o n}} (t) + \boldsymbol {\Lambda} ^ {\mathrm {o f f}} (t)\right) \\ = \frac {1}{2} \left(\frac {\mathbf {V} ^ {\text {o n}} (t)}{\alpha^ {2}} + \mathbf {G} ^ {\text {o n}} (t)\right) + \frac {1}{2} \left(\frac {\mathbf {V} ^ {\text {o f f}} (t)}{\alpha^ {2}} + \mathbf {G} ^ {\text {o f f}} (t)\right) \tag {20} \\ = \frac {1}{2} \left(\frac {\mathbf {V} ^ {\mathrm {o n}} (t) + \mathbf {V} ^ {\mathrm {o f f}} (t)}{\alpha^ {2}}\right) + \frac {1}{2} \left(\mathbf {G} ^ {\mathrm {o n}} (t) + \mathbf {G} ^ {\mathrm {o f f}} (t)\right). \\ \end{array}
$$
To compare convergence rates of  $\mathrm{Fed - CO_2}$  and FedBN, we need to compare their exponential factor  $(1 - \frac{\eta\mu}{2\alpha^2})$  and  $(1 - \frac{\eta\mu^{\mathrm{on}}}{2\alpha^2})$ , as well. Similar to the former proof, the problem reduces to compare  $\mu = \lambda_{\min} \left( \frac{1}{2} (\mathbf{V}_{\mathrm{on}}^{\infty} + \mathbf{V}_{\mathrm{off}}^{\infty}) \right)$  with  $\mu^{\mathrm{on}} = \lambda_{\min} (\mathbf{V}_{\mathrm{on}}^{\infty})$ . With previous proof result  $\lambda_{\min} (\mathbf{V}_{\mathrm{off}}^{\infty}) \geq \lambda_{\min} (\mathbf{V}_{\mathrm{on}}^{\infty})$ , we get
$$
\begin{array}{l} \lambda_ {\min } \left(\frac {1}{2} \left(\mathbf {V} _ {\text {o n}} ^ {\infty} + \mathbf {V} _ {\text {o f f}} ^ {\infty}\right)\right) \geq \frac {1}{2} \lambda_ {\min } \left(\mathbf {V} _ {\text {o n}} ^ {\infty}\right) + \frac {1}{2} \lambda_ {\min } \left(\mathbf {V} _ {\text {o f f}} ^ {\infty}\right) \\ \geq \frac {1}{2} \lambda_ {\min } \left(\mathbf {V} _ {\text {o n}} ^ {\infty}\right) + \frac {1}{2} \lambda_ {\min } \left(\mathbf {V} _ {\text {o n}} ^ {\infty}\right) \\ = \lambda_ {\mathrm {m i n}} (\mathbf {V} _ {\mathrm {o n}} ^ {\infty}). \\ \end{array}
$$
Therefore, we have  $(1 - \frac{\eta\mu^{\mathrm{on}}}{2\alpha^{2}})\geq (1 - \frac{\eta\mu}{2\alpha^{2}})$ . This theoretical conclusion guarantees a faster convergence in our Fed-  $\mathrm{CO}_{2}$  than FedBN.
C.3 Experimental Performance
Here, we empirically analyze the convergence of test accuracy for various algorithms, as well as the convergence behavior of both the online and offline models. The results for sub-dataset WebCam and SVHN are illustrated in Fig. 2 as examples. As Fig. 2 exhibits, among various FL algorithms, our Fed- $\mathrm{CO}_{2}$  achieves the highest accuracy with significantly faster convergence speed and demonstrates much more robust behavior. Compared with its online and offline models, Fed- $\mathrm{CO}_{2}$  consistently surpasses both models, exhibiting a smoother curve, which proves the success in fusing domain-invariant and domain-specific knowledge.
WebCam
Figure 2: Convergence of test accuracy on sub-datasets. Fig. 2(a) and Fig. 2(b) illustrate convergence behavior on WebCam from Office-Caltech10 and SVHN from Digits, respectively. In each sub-graph, we exhibit convergence performance comparison among different algorithms (Top) and comparison with online and offline models (Bottom). Fed-  $\mathrm{CO}_{2}$  exhibits faster and more robust convergence.
D Fed-  $\mathbf{CO}_2$  Algorithm
Here, we provide a detailed description of the algorithm for our universal FL framework,  $\mathrm{Fed - CO_2}$ . The learning process for FL with and without Feature Skew is presented in Algorithm 1 and Algorithm 2, respectively.
Algorithm 1: Fed-  $\mathrm{CO}_{2}$  for FL with Feature Skew
Input:  $T-$  number of communication rounds,  $N-$  number of clients.  
1 Set parameters  $\theta_{g,0}^{\mathrm{on}},\theta_{p,i,0}^{\mathrm{on}},\theta_{i,0}^{\mathrm{off}};$   
2 for each communication rounds  $t\in \{1,\dots,T\}$  do  
3 Communicate  $\{\overline{\phi}_j^{\mathrm{off}}\}_{j = 1}^N$ $\widetilde{\theta}_{g,t - 1}^{\mathrm{on}}$  to all clients;  
4 for each client  $i\in [1,N]$  do  
5 Initialize  $\theta_{i,t}^{\mathrm{on}} = \{\widetilde{\theta}_{g,t - 1}^{\mathrm{on}},\theta_{p,i,t - 1}^{\mathrm{on}}\}$ $\theta_{i,t}^{\mathrm{off}} = \theta_{i,t - 1}^{\mathrm{off}},\bar{\theta}_{i}^{\mathrm{on}} = \theta_{i,t}^{\mathrm{on}},\bar{\theta}_{i}^{\mathrm{off}} = \theta_{i,t}^{\mathrm{off}};$   
6  $(\theta_{i,t}^{\mathrm{on}},\theta_{i,t}^{\mathrm{off}})\gets$  Intra-client Knowledge Transfer  $(\bar{\theta}_i^{\mathrm{off}},\theta_{i,t}^{\mathrm{on}},\bar{\theta}_i^{\mathrm{on}},\theta_{i,t}^{\mathrm{off}})$  [Eq. 4 and Eq. 5];  
7  $(\theta_{i,t}^{\mathrm{on}},\theta_{i,t}^{\mathrm{off}})\gets$  Inter-client Knowledge Transfer  $(\theta_{i,t}^{\mathrm{on}},\theta_{i,t}^{\mathrm{off}},\{\overline{\phi}_j^{\mathrm{off}}\}_{j = 1}^N)$  [Eq. 7];  
8 Communicate  $\theta_{g,i,t}^{\mathrm{on}},\phi_{i,t}^{\mathrm{off}}$  to the server  
9 end  
10 Aggregate  $\widetilde{\theta}_{g,t}^{\mathrm{on}} = \frac{1}{N}\sum_{i = 1}^{N}\theta_{g,i,t}^{\mathrm{on}}$  [Eq. 2];  
11 Construct  $\{\overline{\phi}_j^{\mathrm{off}}\}_{j = 1}^N = \{\phi_{i,t}^{\mathrm{off}}\}_{i = 1}^N;$   
12 end  
13 return  $\widetilde{\theta}_{g,t}^{\mathrm{on}},\theta_{p,i,t}^{\mathrm{on}}$  and  $\theta_{i,t}^{\mathrm{off}}$
Algorithm 2: Fed-  $\mathrm{CO}_{2}$  for FL without Feature Skew
Input:  $T-$  number of communication rounds,  $N-$  number of clients.  
1 Set parameters  $\theta_{g,0}^{\mathrm{on}},\theta_{p,i,0}^{\mathrm{on}},\theta_{i,0}^{\mathrm{off}};$   
2 for each communication rounds  $t\in \{1,\dots,T\}$  do  
3 Communicate  $\widetilde{\theta}_{g,t - 1}^{\mathrm{on}}$  to all clients;  
4 for each client  $i\in [1,N]$  do  
5 Initialize  $\theta_{i,t}^{\mathrm{on}} = \{\widetilde{\theta}_{g,t - 1}^{\mathrm{on}},\theta_{p,i,t - 1}^{\mathrm{on}},\theta_{i,t}^{\mathrm{off}} = \theta_{i,t - 1}^{\mathrm{off}};$   
6  $(\theta_{i,t}^{\mathrm{on}},\theta_{i,t}^{\mathrm{off}})\gets \text{Local Training} (\theta_{i,t}^{\mathrm{on}},\theta_{i,t}^{\mathrm{off}});$   
7 Communicate  $\theta_{g,i,t}^{\mathrm{on}}$  to the server  
8 end  
9 Aggregate  $\widetilde{\theta}_{g,t}^{\mathrm{on}} = \frac{1}{N}\sum_{i = 1}^{N}\theta_{g,i,t}^{\mathrm{on}}$  [Eq. 2];  
10 end  
11 return  $\widetilde{\theta}_{g,t}^{\mathrm{on}},\theta_{p,i,t}^{\mathrm{on}}$  and  $\theta_{i,t}^{\mathrm{off}}$
Table 7: Experiment results for FL with Feature Skew on Digits.
Methods	Digits
MNIST	SVHN	USPS	SynthDigits	MNIST-M	Avg
SingleSet	94.38±0.07	65.25±1.07	95.16±0.12	80.31±0.38	77.77±0.47	82.00±0.40
FedAvg [2]	95.87±0.20	62.86±1.49	95.56±0.27	82.27±0.44	76.85±0.54	82.70±0.60
FedProx [4]	95.75±0.21	63.08±1.62	95.58±0.31	82.34±0.37	76.64±0.55	82.70±0.60
FedPer [11]	96.21±0.02	67.61±0.04	96.53±0.02	83.88±0.02	81.89±0.03	85.22±0.01
MOON [7]	96.25±0.04	65.48±0.49	95.05±0.12	82.89±0.15	80.57±0.23	84.05±0.16
FedRoD [12]	96.09±0.08	71.50±0.25	96.42±0.08	85.51±0.04	81.79±0.09	86.26±0.07
COPA [34]	96.27±0.05	72.90±0.17	95.99±0.06	85.52±0.04	83.08±0.22	86.75±0.02
FedBN [10]	96.57±0.13	71.04±0.31	96.97±0.32	83.19±0.42	78.33±0.66	85.20±0.40
Fed-CO2	97.66±0.07	77.56±0.60	97.78±0.13	88.68±0.08	87.84±0.12	89.91±0.11
E More Experimental Results On Benchmark Datasets
Here, we supplement more experimental results on benchmark datasets. Firstly, we present the experimental results on Digits dataset for FL with feature skew in Table 7. As the table shows, our Fed-  $\mathrm{CO}_{2}$  surpasses other various state-of-the-art algorithms with a substantial margin, which further proves the effectiveness of our framework in adapting local feature shifts.
Then, we show experimental results in FL with both label distribution skew and feature skew on DomainNet and Office-Caltech10 in Table 8 and Table 9. We applied a Dirichlet Distribution with  $\alpha = 2.0$  to DomainNet and another Dirichlet Distribution with  $\alpha = 1.0$  to Office-Caltech10. It can be observed that when facing two severe forms of data heterogeneity, algorithms demonstrated significantly lower performance on Office-Caltech10 and DomainNet compared to FL scenarios with feature skew alone. Even in such a challenging task, our  $\mathrm{Fed - CO_2}$  still achieves the highest average accuracy and outperforms locally-trained models. This result proves that  $\mathrm{Fed - CO_2}$  succeeds in making use of both general domain-invariant and specialized domain-specific knowledge to adapt to extreme local data heterogeneity.
Table 8: Experiment results for FL with both Label Skew and Feature Skew on DomainNet.
Methods	DomainNet
Clipart	Infograph	Painting	Quickdraw	Real	Sketch	Avg
SingleSet	16.3±2.5	17.4±1.5	18.8±3.7	12.6±1.1	12.3±1.6	12.4±4.3	15.0±0.6
FedAvg [2]	9.4±1.7	9.2±1.4	15.2±1.3	10.0±1.7	9.1±0.7	10.2±1.4	10.5±0.3
FedProx [4]	9.6±1.4	10.8±1.6	12.2±1.2	10.1±0.7	10.3±1.5	11.0±2.7	10.7±1.1
FedPer [11]	16.6±1.2	15.3±1.3	18.6±2.4	14.4±1.0	11.9±1.7	15.3±2.9	15.4±0.5
FedRoD [12]	13.6±1.1	15.4±2.1	15.9±1.8	10.6±2.3	11.8±1.0	11.4±1.4	13.1±0.6
COPA [34]	8.4±0.9	10.2±2.5	13.2±2.5	10.2±1.9	11.1±1.3	10.9±1.3	10.7±0.9
FedBN [10]	11.0±1.1	12.9±1.3	13.0±2.0	11.8±0.6	10.4±1.4	11.0±2.2	11.7±0.4
Fed-CO2	18.8±2.4	17.9±1.8	19.5±2.5	14.1±1.1	12.2±2.2	14.6±4.4	16.2±0.5
Table 9: Experiment results for FL with both Label Skew and Feature Skew on Office-Caltech10.
Methods	Office-Caltech10
Amazon	Caltech	DSLR	WebCam	Avg
SingleSet	14.6±2.3	15.2±0.3	10.0±1.2	3.4±0.0	10.8±0.6
FedAvg [2]	15.9±1.7	15.0±2.4	11.2±1.5	3.0±2.7	11.3±0.7
FedProx [4]	14.0±1.3	13.0±1.0	9.4±2.0	5.1±2.4	10.4±0.8
FedPer [11]	16.5±0.6	16.1±0.3	11.9±1.2	1.7±0.0	11.5±0.4
FedRoD [12]	14.7±0.9	12.1±1.0	6.25±0.0	6.4±0.7	9.9±0.5
COPA [34]	17.5±1.7	11.5±0.7	8.1±2.5	4.7±2.7	10.4±1.0
FedBN [10]	19.5±2.5	12.6±1.9	11.9±2.3	2.4±0.8	11.6±0.4
Fed-CO2	17.5±0.9	16.6±1.2	12.5±0.0	3.7±1.3	12.6±0.5