SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jianqing Zhang<sup>1</sup>, Yang Hua<sup>2</sup>, Jian Cao<sup>1*</sup>, Hao Wang<sup>3</sup>, Tao Song<sup>1</sup>, Zhengui Xue<sup>1</sup>, Ruhui Ma<sup>1*</sup>, Haibing Guan<sup>1</sup>
\(^{1}\)Shanghai Jiao Tong University \(^{2}\)Queen's University Belfast \(^{3}\)Louisiana State University \(\{\text { tsingz, cao-jian, songt333, zhenguixue, ruhuima, hbguan }\} @sjtu.edu.cn Y.Hua@qub.ac.uk, haowang@lsu.edu
Recently, federated learning (FL) is popular for its privacy-preserving and collaborative learning abilities. However, under statistically heterogeneous scenarios, we observe that biased data domains on clients cause a representation bias phenomenon and further degenerate generic representations during local training, i.e., the representation degeneration phenomenon. To address these issues, we propose a general framework Domain Bias Eliminator (DBE) for FL. Our theoretical analysis reveals that DBE can promote bi-directional knowledge transfer between server and client, as it reduces the domain discrepancy between server and client in representation space. Besides, extensive experiments on four datasets show that DBE can greatly improve existing FL methods in both generalization and personalization abilities. The DBE-equipped FL method can outperform ten state-of-the-art personalized FL methods by a large margin. Our code is public at https://github.com/TsingZ0/DBE.
As a popular distributed machine learning paradigm with excellent privacy-preserving and collaborative learning abilities, federated learning (FL) trains models among clients with their private data kept locally [37, 56, 79]. Traditional FL (e.g., the famous FedAvg [56]) learns one single global model in an iterative manner by locally training models on clients and aggregating client models on the server. However, it suffers an accuracy decrease under statistically heterogeneous scenarios, which are common scenarios in practice [47, 56, 67, 86].
Due to statistical heterogeneity, the data domain on each client is biased, which does not contain the data of all labels [37, 45, 67, 69, 79, 84]. As the received global model is locally trained on individual clients' biased data domain, we observe that this model extracts biased (i.e., forming client-specific clusters) representations during local training. We call this phenomenon "representation bias" and visualize it in Figure 1. Meanwhile, by training the received global model with missing labels, the generic representation quality over all labels also decreases during local training [45]. Furthermore, we ob
(a) Before.
(b) After.
(c) MDL change.
Figure 1: t-SNE [73] visualization and per-layer MDL (bits) for representations before/after local training in FedAvg. We use color and shape to distinguish labels and clients respectively for t-SNE. A large MDL means low representation quality. Best viewed in color and zoom-in.
serve that this "representation degeneration" phenomenon happens at every layer, as shown in Figure 1(c). We estimate the representation quality via minimum description length (MDL) [62, 65, 74], a metric independent of data and models, measuring the difficulty of classifying target labels according to given representations.
To tackle the statistical heterogeneity, unlike traditional FL that learns a single global model, personalized FL (pFL) comes along by learning personalized models (or modules) for each client besides learning a global model among clients [20, 22, 69]. Typically, most of the existing pFL methods train a personalized classifier $^{2}$  for each client [3, 14, 20, 61], but the feature extractor still extracts all the information from the biased local data domain, leading to representation bias and representation degeneration during local training.
To address the representation bias and representation degeneration issues in FL, we propose a general framework Domain Bias Eliminator (DBE) for FL including two modules introduced as follows. Firstly, we detach the representation bias from original representations and preserve it in a Personalized Representation Bias Memory (PRBM) on each client. Secondly, we devise a Mean Regularization (MR) that explicitly guides local feature extractors to extract representations with a consensual global mean during local training to let the local feature extractor focus on the remaining unbiased information and improve the generic representation quality. In this way, we turn one level of representation between the feature extractor and the classifier on each client into two levels of representation with a client-specific bias and a client-invariant mean, respectively. Thus, we can eliminate the conflict of extracting representations with client-specific biases for clients' requirements while extracting representations with client-invariant features for the server's requirements in the same representation space. Our theoretical analysis shows that DBE can promote the bi-directional knowledge transfer between server and client with lower generalization bounds.
We conduct extensive experiments in computer vision (CV) and natural language processing (NLP) fields on various aspects to study the characteristics and effectiveness of DBE. In both generalization ability (measured by MDL) and personalization ability (measured by accuracy), DBE can promote the fundamental FedAvg as well as other representative FL methods. Furthermore, we compare the representative FedAvg+DBE with ten state-of-the-art (SOTA) pFL methods in various scenarios and show its superiority over these pFL methods. To sum up, our contributions are:
- We observe the representation bias and per-layer representation degeneration phenomena during local training in the representative FL method FedAvg.
- We propose a framework DBE to memorize representation bias on each client to address the representation bias issue and explicitly guide local feature extractors to generate representations with a universal mean for higher generic representation quality.
- We provide theoretical analysis and derive lower generalization bounds of the global and local feature extractors to show that DBE can facilitate bi-directional knowledge transfer between server and client in each iteration.
- We show that DBE can improve other representative traditional FL methods including FedAvg at most  $-22.35\%$  in MDL (bits) and  $+32.30$  in accuracy (\%), respectively. Furthermore, FedAvg+DBE outperforms SOTA pFL methods by up to  $+11.36$  in accuracy (\%).
Traditional FL methods that focus on improving accuracy under statistically heterogeneous scenarios based on FedAvg including four categories: update-correction-based FL [25, 38, 60], regularization-based FL [1, 17, 40, 46], model-split-based FL [35, 45], and knowledge-distillation-based FL [27, 33, 88, 96]. For pFL methods, we consider four categories: meta-learning-based pFL [13, 22], regularization-based pFL [47, 67], personalized-aggregation-based pFL [21, 52, 87, 89], and model-split-based pFL [3, 14, 20, 61, 85]. Due to limited space, we only introduce the FL methods that are close to ours and leave the extended version of this section to Appendix A.
Traditional FL methods. MOON [45] utilizes contrastive learning to correct the local training of each client, but this input-wise contrastive learning still relies on the biased local data domain, so it still suffers from representation skew. Although FedGen [96] learns a shared generator on the server
${}^{2}$  A model is split into a feature extractor and a classifier. They are sequentially jointed.
and reduces the heterogeneity among clients with the generated representations through knowledge distillation, it only considers the local-to-global knowledge for the single global model learning. On the other hand, FedGen additionally brings non-negligible communication and computation overhead for learning and transmitting the generator.
pFL methods. FedPer [3] and FedRep [20] keep the classifier locally, but the feature extractor still learns biased features without explicit guidance. Besides, their local feature extractors are trained to cater to personalized classifiers thus losing generality. FedRoD [14] reduces the discrepancy of local training tasks among clients by using a balanced softmax (BSM) loss function [64], but the BSM is useless for missing labels on each client while label missing is a common situation in statistically heterogeneous scenarios [50, 86, 89]. Moreover, the uniform label distribution modified by the BSM cannot reflect the original distribution. Differently, FedBABU [61] trains a global feature extractor with a naive and frozen classifier, then it fine-tunes the classifier for each client to finally obtain personalized models. However, the post-FL fine-tuning study is beyond the FL scope, as almost all the FL methods have multiple fine-tuning variants, e.g., fine-tuning the whole model or only a part of the model. Like FedAvg, FedBABU still locally extracts biased features during the FL process.
3 Notations and Preliminaries
In this work, we discuss the statistically heterogeneous scenario in typical multi-class classification tasks for FL, where  $N$  clients share the same model structure. Here, we denote notations following FedGen [96] and FedRep [20]. The client  $i$ ,  $i \in [N]$ , has its own private data domain  $\mathcal{D}_i$ , where the data are sampled from  $\mathcal{D}_i$ . All the clients collaborate to train a global model  $g$  parameterized by  $\theta$  without sharing their private local data.
Since we focus on representation learning in FL, we regard  $g$  as the sequential combination of a feature extractor  $f$  that maps from the input space  $\mathcal{X}$  to a representation space  $\mathcal{Z}$ , i.e.,  $f:\mathcal{X}\mapsto \mathcal{Z}$  parameterized by  $\pmb{\theta}^{f}$  and a classifier  $h$  that maps from the representation space to the output space  $\triangle^{\mathcal{V}}$ , i.e.,  $h:\mathcal{Z}\mapsto \triangle^{\mathcal{V}}$  parameterized by  $\pmb{\theta}^{h}$ . Formally, we have  $g\coloneqq h\circ f$ ,  $\pmb {\theta}\coloneqq [\pmb{\theta}^{f};\pmb{\theta}^{h}]$ ,  $\mathcal{X}\subset \mathbb{R}^{D}$  and  $\mathcal{Z}\subset \mathbb{R}^{K}$ .  $\triangle^{\mathcal{V}}$  is the simplex over label space  $\mathcal{Y}\subset \mathbb{R}$ . With any input  $\pmb {x}\in \mathcal{X}$ , we obtain the feature representation by  $z = f(x;\pmb{\theta}^{f})\in \mathcal{Z}$ .
3.2 Traditional Federated Learning
With the collaboration of  $N$  clients, the objective of traditional FL, e.g., FedAvg [56], is to iteratively learn a global model that minimizes its loss on each local data domain:
$$
\min  _ {\boldsymbol {\theta}} \mathbb {E} _ {i \in [ N ]} \left[ \mathcal {L} _ {\mathcal {D} _ {i}} (\boldsymbol {\theta}) \right], \tag {1}
$$
$$
\mathcal {L} _ {\mathcal {D} _ {i}} (\boldsymbol {\theta}) := \mathbb {E} _ {(\boldsymbol {x} _ {i}, y _ {i}) \sim \mathcal {D} _ {i}} [ \ell (g (\boldsymbol {x} _ {i}; \boldsymbol {\theta}), y _ {i}) ] = \mathbb {E} _ {(\boldsymbol {x} _ {i}, y _ {i}) \sim \mathcal {D} _ {i}} [ \ell (h (f (\boldsymbol {x} _ {i}; \boldsymbol {\theta} ^ {f}); \boldsymbol {\theta} ^ {h}), y _ {i}) ], \tag {2}
$$
where  $\ell : \triangle^{\mathcal{Y}} \times \mathcal{Y} \mapsto \mathbb{R}$  is a non-negative and convex loss function. Following FedGen, we assume that all clients share an identical loss function  $\ell$  and a virtual global data domain  $\mathcal{D}$ , which is the union of all local domains:  $\mathcal{D} := \bigcup_{i=1}^{N} \mathcal{D}_i$ . In practice, traditional FL methods [45, 46, 56] optimize Eq. (1) by  $\min_{\boldsymbol{\theta}} \sum_{i=1}^{N} \frac{n_i}{n} \mathcal{L}_{\hat{\mathcal{D}}_i}(\boldsymbol{\theta})$ , where  $\hat{\mathcal{D}}_i$  is an observable dataset,  $n_i = |\hat{\mathcal{D}}_i|$  is its size, and  $n = \sum_{i=1}^{N} n_i$ .
In each communication iteration, clients conduct local updates on their private data to train the global model  $\pmb{\theta}$  by minimizing their local loss. Formally, for client  $i$ , the objective during local training is  $\min_{\pmb{\theta}}\mathcal{L}_{\mathcal{D}_i}(\pmb{\theta})$ . The empirical version of  $\mathcal{L}_{\mathcal{D}_i}(\pmb{\theta})$  is  $\mathcal{L}_{\hat{\mathcal{D}}_i}(\pmb{\theta}) \coloneqq \frac{1}{n_i}\sum_{j=1}^{n_i}\ell(h(f(\pmb{x}_{ij};\pmb{\theta}^f);\pmb{\theta}^h),y_{ij})$ , which is optimized by stochastic gradient descent (SGD) [56, 90] in FedAvg.
4.1 Problem Statement
pFL iteratively learns a personalized model or module for each client with the assistance of the global model parameters from the server. Our objective is (with a slight reuse of the notation  $\mathcal{L}_{\mathcal{D}_i}$ )
$$
\min  _ {\boldsymbol {\theta} _ {1}, \dots , \boldsymbol {\theta} _ {N}} \mathbb {E} _ {i \in [ N ]} \left[ \mathcal {L} _ {\mathcal {D} _ {i}} \left(\boldsymbol {\theta} _ {i}\right) \right], \tag {3}
$$
where  $\theta_{i}$  is a model consisting of global and personalized modules. The global modules are locally trained on clients and shared with the server for aggregation like traditional FL, but the personalized modules are preserved locally on clients. Following traditional FL, we empirically optimize Eq. (3) by  $\min_{\theta_1,\dots,\theta_N}\sum_{i = 1}^{N}\frac{n_i}{n}\mathcal{L}_{\hat{\mathcal{D}}_i}(\pmb {\theta}_i)$ .
4.2 Personalized Representation Bias Memory (PRBM)
Due to the existence of statistical heterogeneity in FL, the local feature extractor intends to learn biased representations after being overwritten by the received global model parameters. To detach and store the representation bias locally, we propose a personalized module PRBM that memorizes representation bias for client  $i$ . Originally, the feature representation  $\mathbf{z}_i \in \mathbb{R}^K$  is directly fed into the predictor in Eq. (2). Instead, we consider  $\mathbf{z}_i$  as the combination of a global  $\mathbf{z}_i^g \in \mathbb{R}^K$  and a personalized  $\bar{\mathbf{z}}_i^p \in \mathbb{R}^K$ , i.e.,
$$
\boldsymbol {z} _ {i} := \boldsymbol {z} _ {i} ^ {g} + \bar {\boldsymbol {z}} _ {i} ^ {p}. \tag {4}
$$
We let the feature extractor output  $\boldsymbol{z}_i^g$  instead of the original  $\boldsymbol{z}_i$ , i.e.,  $\boldsymbol{z}_i^g \coloneqq f(\boldsymbol{x}_i; \boldsymbol{\theta}^f)$  and keep the trainable vector  $\bar{\boldsymbol{z}}_i^p$  locally.  $\bar{\boldsymbol{z}}_i^p$  is specific
among clients but identical for all the local data on one client, so it memorizes client-specific mean. The original feature extractor is trained to capture the biased features for  $\mathbf{z}_i$ . Instead, with the personalized mean stored in  $\bar{\mathbf{z}}_i^p$ , the feature extractor turns to capture  $\mathbf{z}_i^g$  with less biased feature information. We illustrate the difference between the original approach and our method in Figure 2 (PRBM). Then, we define the local objective as  $\min_{\theta_i} \mathcal{L}_{\mathcal{D}_i}(\theta_i)$ , where  $\theta_i := [\theta^f; \bar{\mathbf{z}}_i^p; \theta^h]$ ,
$$
\mathcal {L} _ {\mathcal {D} _ {i}} \left(\boldsymbol {\theta} _ {i}\right) := \mathbb {E} _ {\left(\boldsymbol {x} _ {i}, y _ {i}\right) \sim \mathcal {D} _ {i}} \left[ \ell \left(h \left(f \left(\boldsymbol {x} _ {i}; \boldsymbol {\theta} ^ {f}\right) + \bar {\boldsymbol {z}} _ {i} ^ {p}; \boldsymbol {\theta} ^ {h}\right), y _ {i}\right) \right]. \tag {5}
$$
From the view of transformation, we rewrite Eq. (5) to
$$
\mathcal {L} _ {\mathcal {D} _ {i}} \left(\boldsymbol {\theta} _ {i}\right) := \mathbb {E} _ {\left(\boldsymbol {x} _ {i}, y _ {i}\right) \sim \mathcal {D} _ {i}} \left[ \ell \left(h \left(\operatorname {P R B M} \left(f \left(\boldsymbol {x} _ {i}; \boldsymbol {\theta} ^ {f}\right); \bar {\boldsymbol {z}} _ {i} ^ {p}\right); \boldsymbol {\theta} ^ {h}\right), y _ {i}\right) \right], \tag {6}
$$
where  $\mathrm{PRBM}:\mathcal{Z}\mapsto \mathcal{Z}$  a personalized translation transformation [78] parameterized by  $\bar{z}_i^p$ . Formally,  $\mathrm{PRBM}(z_i^g;\bar{z}_i^p) = z_i^g +\bar{z}_i^p,\forall z_i^g\in \mathcal{Z}$ . With PRBM, we create an additional level of representation  $z_{i}^{g}$  besides the original level of representation  $z_{i}$ . We call  $z_{i}^{g}$  and  $z_{i}$  as the first and second levels of representation, respectively. For the original local model (Figure 2(a)), we have  $z_{i}^{g}\equiv z_{i}$ .
4.3 Mean Regularization (MR)
Without explicit guidance, it is hard for the feature extractor and the trainable PRBM to distinguish between unbiased and biased information in representations automatically. Therefore, to let the feature extractor focus on the unbiased information and further separate  $z_{i}^{g}$  and  $\bar{z}_{i}^{p}$ , we propose an MR that explicitly guides the local feature extractor to generate  $z_{i}^{g}$  with the help of a client-invariant mean, which is opposite to the client-specific mean memorized in  $\bar{z}_{i}^{p}$ , as shown in Figure 2 (MR). Specifically, we regularize the mean of  $z_{i}^{g}$  to the consensual global mean  $\bar{z}^{g}$  at each feature dimension independently. We then modify Eq. (6) as
$$
\mathcal {L} _ {\mathcal {D} _ {i}} \left(\boldsymbol {\theta} _ {i}\right) := \mathbb {E} _ {\left(\boldsymbol {x} _ {i}, y _ {i}\right) \sim \mathcal {D} _ {i}} \left[ \ell \left(h \left(\operatorname {P R B M} \left(f \left(\boldsymbol {x} _ {i}; \boldsymbol {\theta} ^ {f}\right); \bar {\boldsymbol {z}} _ {i} ^ {p}\right); \boldsymbol {\theta} ^ {h}\right), y _ {i}\right) \right] + \kappa \cdot \mathrm {M R} \left(\bar {\boldsymbol {z}} _ {i} ^ {g}, \bar {\boldsymbol {z}} ^ {g}\right), \tag {7}
$$
where  $\bar{z}_i^g = \mathbb{E}_{(\pmb{x}_i,y_i)\sim \mathcal{D}_i}[f(\pmb{x}_i;\pmb{\theta}^f)]$ . We obtain the consensus  $\bar{z}^{g} = \sum_{i = 1}^{N}\bar{z}_{i}^{g}$  during the initialization period before FL (see Algorithm 1). We measure the distance of  $\bar{z}_i^g$  and  $\bar{z}^{g}$  by mean squared error (MSE) [72], and  $\kappa$  is a hyperparameter to control the importance of MR for different tasks. Empirically,
$$
\mathcal {L} _ {\hat {\mathcal {D}} _ {i}} \left(\boldsymbol {\theta} _ {i}\right) := \frac {1}{n _ {i}} \sum_ {j = 1} ^ {n _ {i}} \ell \left(h \left(\operatorname {P R B M} \left(f \left(\boldsymbol {x} _ {i j}; \boldsymbol {\theta} ^ {f}\right); \bar {\boldsymbol {z}} _ {i} ^ {p}\right); \boldsymbol {\theta} ^ {h}\right), y _ {i j}\right) + \kappa \cdot \mathrm {M R} \left(\frac {1}{n _ {i}} \sum_ {j = 1} ^ {n _ {i}} f \left(\boldsymbol {x} _ {i j}; \boldsymbol {\theta} ^ {f}\right), \bar {\boldsymbol {z}} ^ {g}\right), \tag {8}
$$
which is also optimized by SGD following FedAvg.
(a) Local model (original).
(b) Local model (ours).
Figure 2: The illustration of the local model. We emphasize the parts that correspond to PRBM and MR with red and green, respectively.
In Eq. (8), the value of the MR term is obtained after calculating the empirical version of  $\bar{z}_i^g$ :  $\hat{\bar{z}}_i^g = \frac{1}{n_i}\sum_{j=1}^{n_i}f(\pmb{x}_{ij};\pmb{\theta}^f)$  over the entire local data, but the loss computing in SGD cannot see all the local data during one forward pass in one batch. In practice, inspired by the widely-used moving average [48, 90] in approximating statistics over data among batches, in each batch, we obtain
$$
\hat {\tilde {z}} _ {i} ^ {g} = (1 - \mu) \cdot \hat {\tilde {z}} _ {i, o l d} ^ {g} + \mu \cdot \hat {\tilde {z}} _ {i, n e w} ^ {g}, \tag {9}
$$
where  $\hat{\bar{z}}_{i,old}^{g}$  and  $\hat{\bar{z}}_{i,new}^{g}$  are computed in the previous batch and current batch, respectively.  $\mu$  is a hyperparameter called momentum that controls the importance of the current batch. The feature extractor is updated continuously during local training but discontinuously between adjacent two iterations due to server aggregation. Thus, we only calculate  $\hat{\bar{z}}_i^g$  via Eq. (9) during local training and recalculate it in a new iteration without using its historical records. We consider the representative FedAvg+DBE as an example and show the entire learning process in Algorithm 1.
Algorithm 1 The Learning Process in FedAvg+DBE
Input:  $N$  clients with their local data; initial parameters  $\theta^{f,0}$  and  $\theta^{h,0}$ ;  $\eta$ : local learning rate;  $\kappa$  and  $\mu$ : hyperparameters;  $\rho$ : client joining ratio;  $E$ : local epochs;  $T$ : total communication iterations.  
Output: Global model parameters  $\{\theta^f,\theta^h\}$  and personalized model parameters  $\{\bar{z}_1^p,\dots,\bar{z}_N^p\}$ .
$\triangleright$  Initialization Period
1: Server sends  $\{\pmb{\theta}^{f,0},\pmb{\theta}^{h,0}\}$  to all clients to initialize their local models.
2:  $N$  clients train their local models without DBE for one epoch and collect client-specific mean  $\{\bar{z}_1^g,\dots ,\bar{z}_N^g\}$  over their data domain.
3: Server generates a consensual global mean  $\bar{z}^g$  through weighted averaging:  $\bar{z}^g = \sum_{i=1}^{N} \frac{n_i}{n} \bar{z}_i^g$ .
4: Client  $i$  initializes  $\bar{z}_i^{p,0},\forall i\in [N]$
Federated Learning Period
5: for communication iteration  $t = 1, \dots, T$  do
6: Server samples a client subset  $\mathcal{T}^t$  based on  $\rho$ .
7: Server sends  $\{\pmb{\theta}^{f,t - 1},\pmb{\theta}^{h,t - 1}\}$  to each client in  $\mathcal{I}^t$
8: for Client  $i\in \mathcal{I}^t$  in parallel do
9: Initialize  $f$  and  $\bar{h}$  with  $\pmb{\theta}^{f,t - 1}$  and  $\pmb{\theta}^{h,t - 1}$ , respectively.
10: Obtain  $\{\pmb {\theta}_i^{f,t},\bar{z}_i^{p,t},\pmb {\theta}_i^{h,t}\}$  using SGD for  $\min_{\pmb {\theta}_i}\mathcal{L}_{\hat{\mathcal{D}}_i}(\pmb {\theta}_i)$  with  $\eta ,\kappa$  and  $\mu$  for  $E$  epochs.
11: Upload  $\{\pmb {\theta}_i^{f,t},\pmb {\theta}_i^{h,t}\}$  to the server.
12: Server calculates  $n^t = \sum_{i\in \mathcal{I}^t}n_i$  and obtains
13:  $\pmb{\theta}^{f,t} = \sum_{i\in \mathcal{I}^t}\frac{n_i}{n^t}\pmb{\theta}_i^{f,t};$
14:  $\pmb{\theta}^{h,t} = \sum_{i\in \mathcal{I}^t}\frac{n_i}{n^t}\pmb{\theta}_i^{h,t}.$
15: return  $\{\pmb{\theta}^{f,T},\pmb{\theta}^{h,T}\}$  and  $\{\bar{z}_1^{p,T},\dots,\bar{z}_N^{p,T}\}$
4.4 Improved Bi-directional Knowledge Transfer
In the FL field, prior methods draw a connection from FL to domain adaptation for theoretical analysis and consider a binary classification problem [21, 55, 69, 96]. The traditional FL methods, which focus on enhancing the performance of a global model, regard local domains  $\mathcal{D}_i$ ,  $i \in [N]$  and the virtual global domain  $\mathcal{D}$  as the source domain and the target domain, respectively [96], which is called local-to-global knowledge transfer in this paper. In contrast, pFL methods that focus on improving the performance of personalized models regard  $\mathcal{D}$  and  $\mathcal{D}_i$ ,  $i \in [N]$  as the source domain and the target domain, respectively [21, 55, 69]. We call this kind of adaptation as global-to-local knowledge transfer. The local-to-global knowledge transfer happens on the server while the global-to-local one occurs on the client. Please refer to Appendix B for details and proofs.
4.4.1 Local-To-Global Knowledge Transfer
Here, we consider the transfer after the server receives a client model. We guide the feature extractor to learn representations with a global mean and gradually narrow the gap between the local domain and global domain at the first level of representation (i.e.,  $z_i^g$ ) to improve knowledge transfer:
Corollary 1. Consider a local data domain  $\mathcal{D}_i$  and a virtual global data domain  $\mathcal{D}$  for client  $i$  and the server, respectively. Let  $\mathcal{D}_i = \langle \mathcal{U}_i, c^* \rangle$  and  $\mathcal{D} = \langle \mathcal{U}, c^* \rangle$ , where  $c^* : \mathcal{X} \mapsto \mathcal{Y}$  is a ground-truth
labeling function. Let  $\mathcal{H}$  be a hypothesis space of VC dimension  $d$  and  $h: \mathcal{Z} \mapsto \mathcal{Y}, \forall h \in \mathcal{H}$ . When using DBE, given a feature extraction function  $\mathcal{F}^g: \mathcal{X} \mapsto \mathcal{Z}$  that shared between  $\mathcal{D}_i$  and  $\mathcal{D}$ , a random labeled sample of size  $m$  generated by applying  $\mathcal{F}^g$  to a random sample from  $\mathcal{U}_i$  labeled according to  $c^*$ , then for every  $h^g \in \mathcal{H}$ , with probability at least  $1 - \delta$ :