SlowGuess/NeurIPS_2023_OCR huggingface dataset:
FL methods perform machine learning through iterative communication and computation on the server and clients. To begin with, we describe the FL procedure in one iteration based on FedAvg [56], which is a famous FL method and a basic framework for later FL methods. The FL procedure includes five stages: (1) A server selects a group of clients to join FL in this iteration and sends the current global model to them; (2) these clients receive the global model and initialize their local model by overwriting their local model with the parameters in the global model; (3) these clients train their local models on their own private local data, respectively; (4) these clients send the trained local models to the server; (5) the server receives client models and aggregates them through weighted averaging on model parameters to obtain a new global model.
Then, massive traditional FL methods are proposed in the literature to improve FedAvg regarding privacy-preserving [49, 58, 95], accuracy [38, 45, 96], fairness [32, 80], overhead [28, 41, 54], etc. Here, we focus on the representative traditional FL methods that handle the heterogeneity issues in four categories: update-correction-based FL [25, 38, 60], regularization-based FL [1, 17, 40, 46], model-split-based FL [35, 45], and knowledge-distillation-based FL [27, 33, 88, 96].
Among update-correction-based FL methods, SCAFFOLD [38] witnesses the client-drift phenomenon of FedAvg under statistically heterogeneous scenarios due to local training and proposes correcting local update through control variates for each model parameter. Among regularization-based FL methods, FedProx [46] modifies the local objective on each client by adding a regularization term to keep local model parameters close to the global model during local training in an element-wise manner. Among model-split-based FL methods, MOON [45] observes that local training degenerates representation quality, so it adds a contrastive learning term to let the representations outputted by the local feature extractor be close to the ones outputted by the received global feature extractor given each input during local training. However, input-wise contrastive learning relies on biased local data domains, so MOON still suffers from representation bias. Among knowledge-distillation-based FL methods, FedGen [96] learns a generator on the server to produce additional representations, shares the generator among clients, and locally trains the classifier with the combination of the representations outputted by the local feature extractor and the additionally generated representations. In this way, FedGen can reduce the heterogeneity among clients with the augmented representations from the shared generator via knowledge distillation. However, it only considers the local-to-global knowledge transfer for the single global model learning and additionally brings communication and computation overhead for learning and transmitting the generator.
A.2 Personalized Federated Learning
Different from traditional FL, pFL additionally learns personalized models (or modules) besides the global model. In this paper, we consider pFL methods in four categories: meta-learning-based pFL [13, 22], regularization-based pFL [47, 67], personalized-aggregation-based pFL [21, 52, 89], and model-split-based pFL [3, 14, 20, 61].
Meta-learning-based pFL. Meta-learning is a technique that trains deep neural networks (DNNs) on a given dataset for quickly adapting to other datasets with only a few steps of fine-tuning, e.g., MAML [24]. By integrating MAML into FL, Per-FedAvg [22] updates the local models like MAML to capture the learning trends of each client and then aggregates the learning trends by averaging on the server. It obtains personalized models by fine-tuning the global model for each client. Similar to Per-FedAvg, FedMeta [13] also introduces MAML on each client during training and fine-tuning the global model for evaluation. However, it is hard for these meta-learning-based pFL methods to find a consensus learning trend through averaging under statistically heterogeneous scenarios.
Regularization-based pFL. Like FedProx, pFedMe [67] and Ditto [47] also utilize the regularization technique, but they modify the objective for additional personalized model training rather than the one for local model training. In pFedMe and Ditto, each client owns two models: the local model that is trained for global model aggregation and the personalized model that is trained for personalization. Specifically, pFedMe regularizes the model parameters between the personalized model and the local model during training while Ditto regularizes the model parameters between the personalized model and the received global model. Besides, Ditto simply trains the local model similar to FedAvg while pFedMe trains the local model based on the personalized model. Although the local model is initialized by the global model, but the initialized local model gradually loses global information during local training. Thus, the personalized model in Ditto can be aware of more global information than the one in pFedMe. Both pFedMe and Ditto require additional memory space to store the personalized model and double the computation resources at least to train both the local model and the personalized model.
Personalized-aggregation-based pFL. These pFL methods adaptively aggregate the global model and local model according to the local data on each client, e.g., APFL [21], or directly generate the personalized model using other client models through personalized aggregation on each client, e.g., FedFomo [89] and APPLE [52]. Specifically, APFL aggregates the parameters in the global model and the local model with weighted averaging and adaptively updates the scalar weight based on the gradients. On each client, FedFomo generates the client-specific aggregating weights for the received client models through first-order approximation while APPLE adaptively learns these weights based on the local data. Both FedFomo and APPLE require multiple communication overhead than other FL methods, but FedFomo costs less computation overhead than APPLE attributed to approximation.
Model-split-based pFL. These pFL methods split a given model into a feature extractor and a classifier. They treat the feature extractor and the classifier differently. Concretely, FedPer [3] and FedRep [20] keep the classifier locally on each client. FedPer trains the feature extractor and the classifier together while FedRep first fine-tunes the classifier and then trains the feature extractor in each iteration. For FedPer and FedRep, the feature extractor intends to extract representations to cater to these personalized classifiers, thus reducing the generic representation quality. FedRoD [14] trains the local model with the balanced softmax (BSM) loss function [64] and simultaneously learns an additional personalized classifier for each client. However, the BSM loss is useless for missing labels on each client while label missing is a common situation in statistically heterogeneous scenarios [50, 86, 89]. Moreover, the uniform label distribution modified by the BSM cannot reflect the original distribution. The above pFL methods learn personalized models (or modules) in FL, but FedBABU [61] firstly trains the global feature extractor with the frozen classifier during the FL process, then it fine-tunes the global model on each client after FL to obtain personalized models. However, this post-FL fine-tuning is beyond the scope of FL. Almost all the FL methods have multiple fine-tuning variants, e.g., fine-tuning the whole model or only a part of the model. Furthermore, training the feature extractor with the naive and randomly initialized classifier in FL has an uncontrollable risk due to randomness.
B Theoretical Derivations
B.1 Notations and Preliminaries
Following prior arts [21, 55, 69, 96], we consider a binary classification problem in FL here. Recall that  $\mathcal{X} \subset \mathbb{R}^D$  is an input space,  $\mathcal{Z} \subset \mathbb{R}^K$  is a representation space, and  $\mathcal{Y} \subset \{0,1\}$  is a label space. Let  $\mathcal{F}: \mathcal{X} \mapsto \mathcal{Z}$  be a representation function that maps from the input space to the representation space. We denote  $\mathcal{D} := \langle \mathcal{U}, c^* \rangle$  as a data domain where the distribution  $\mathcal{U} \subseteq \mathcal{X}$  and  $c^*: \mathcal{X} \mapsto \mathcal{Y}$  is a ground-truth labeling function.  $\tilde{\mathcal{U}}$  is the induced distribution of  $\mathcal{U}$  over the representation space  $\mathcal{Z}$  under  $\mathcal{F}$  [6], i.e.,  $\tilde{\mathcal{U}} \subseteq \mathcal{Z}$ , that satisfies
$$
\mathbb {E} _ {\boldsymbol {z} \sim \tilde {\mathcal {U}}} [ \mathcal {B} (\boldsymbol {z}) ] = \mathbb {E} _ {\boldsymbol {x} \sim \tilde {\mathcal {U}}} [ \mathcal {B} (\mathcal {F} (\boldsymbol {x})) ], \tag {10}
$$
where  $\mathcal{B}$  is a probability event. Given fixed but unknown  $\mathcal{U}$  and  $c^*$ , the learning task on one domain is to choose a representation function  $\mathcal{F}$  and a hypothesis class  $\mathcal{H} \subseteq \{h : \mathcal{Z} \mapsto \mathcal{Y}\}$  to approximate the function  $c^*$ .
Then, we provide the definition and theorem from Ben-David et al. [6, 7], Blitzer et al. [9], Kifer et al. [39] under their assumptions:
Definition 1. If a space  $\mathcal{Z}$  with  $\tilde{\mathcal{U}}^a$  and  $\tilde{\mathcal{U}}^b$  distributions over  $\mathcal{Z}$ , let  $\mathcal{H}$  be a hypothesis class on  $\mathcal{Z}$  and  $\mathcal{Z}_h\subseteq \mathcal{Z}$  be the subset with characteristic function  $h$ , the  $\mathcal{H}$ -divergence between  $\tilde{\mathcal{U}}^a$  and  $\tilde{\mathcal{U}}^b$  is
$$
d _ {\mathcal {H}} \left(\tilde {\mathcal {U}} ^ {a}, \tilde {\mathcal {U}} ^ {b}\right) = 2 \sup _ {h \in \mathcal {H}} \left| \operatorname * {P r} _ {\tilde {\mathcal {U}} ^ {a}} \left[ \mathcal {Z} _ {h} \right] - \operatorname * {P r} _ {\tilde {\mathcal {U}} ^ {b}} \left[ \mathcal {Z} _ {h} \right] \right|,
$$
where  $\mathcal{Z}_h = \{\pmb {z}\in \mathcal{Z}:h(\pmb {z}) = 1\} ,h\in \mathcal{H}.$
Definition 1 implies that  $d_{\mathcal{H}}\left(\tilde{\mathcal{U}}^a,\tilde{\mathcal{U}}^b\right) = d_{\mathcal{H}}\left(\tilde{\mathcal{U}}^b,\tilde{\mathcal{U}}^a\right)$ .
Theorem 1. Consider a source domain  $\mathcal{D}_S$  and a target domain  $\mathcal{D}_T$ . Let  $\mathcal{D}_S = \langle \mathcal{U}_S, c^* \rangle$  and  $\mathcal{D}_T = \langle \mathcal{U}_T, c^* \rangle$ , where  $\mathcal{U}_S \subseteq \mathcal{X}$ ,  $\mathcal{U}_T \subseteq \mathcal{X}$ , and  $c^* : \mathcal{X} \mapsto \mathcal{Y}$  is a ground-truth labeling function. Let  $\mathcal{H}$  be a hypothesis space of VC dimension  $d$  and  $h : \mathcal{Z} \mapsto \mathcal{Y}, \forall h \in \mathcal{H}$ . Given a feature extraction function  $\mathcal{F} : \mathcal{X} \mapsto \mathcal{Z}$  that shared between  $\mathcal{D}_S$  and  $\mathcal{D}_T$ , a random labeled sample of size  $m$  generated by applying  $\mathcal{F}$  to a random sample from  $\mathcal{U}_S$  labeled according to  $c^*$ , then for every  $h \in \mathcal{H}$ , with probability at least  $1 - \delta$ :
$$
\mathcal {L} _ {\mathcal {D} _ {T}} (h) \leq \mathcal {L} _ {\hat {\mathcal {D}} _ {S}} (h) + \sqrt {\frac {4}{m} \left(d \log \frac {2 e m}{d} + \log \frac {4}{\delta}\right)} + d _ {\mathcal {H}} (\tilde {\mathcal {U}} _ {S}, \tilde {\mathcal {U}} _ {T}) + \lambda ,
$$
where  $\mathcal{L}_{\tilde{\mathcal{D}}_S}$  is the empirical loss on  $\mathcal{D}_S$ ,  $e$  is the base of the natural logarithm, and  $d_{\mathcal{H}}(\cdot,\cdot)$  is the  $\mathcal{H}$ -divergence between two distributions.  $\tilde{\mathcal{U}}_S$  and  $\tilde{\mathcal{U}}_T$  are the induced distributions of  $\mathcal{U}_S$  and  $\mathcal{U}_T$  under  $\mathcal{F}$ , respectively, s.t.  $\mathbb{E}_{\boldsymbol{z}\sim \tilde{\mathcal{U}}_S}[\mathcal{B}(\boldsymbol{z})] = \mathbb{E}_{\boldsymbol{x}\sim \mathcal{U}_S}[\mathcal{B}(\mathcal{F}(\boldsymbol{x}))]$  given a probability event  $\mathcal{B}$ , and so for  $\tilde{\mathcal{U}}_T$ .  $\tilde{\mathcal{U}}_S \subseteq \mathcal{Z}$  and  $\tilde{\mathcal{U}}_T \subseteq \mathcal{Z}$ .  $\lambda := \min_h\mathcal{L}_{\mathcal{D}_S}(h) + \mathcal{L}_{\mathcal{D}_T}(h)$  denotes an oracle performance.
The traditional FL methods, which focus on enhancing the performance of a global model, regard local domains  $\mathcal{D}_i$ ,  $i \in [N]$  and the virtual global domain  $\mathcal{D}$  as the source domain and the target domain, respectively [96], which is called local-to-global knowledge transfer in this paper. In contrast, pFL methods that focus on improving the performance of personalized models regard  $\mathcal{D}$  and  $\mathcal{D}_i$ ,  $i \in [N]$  as the source domain and the target domain, respectively [21, 55, 69]. We call this kind of adaptation global-to-local knowledge transfer. The local-to-global knowledge transfer happens on the server while the global-to-local one occurs on the client.
B.2 Derivations of Corollary 1
As we focus on the local-to-global knowledge transfer on the server side, in the FL scenario, we can rewrite Theorem 1 to
Theorem 2. Consider a local data domain  $\mathcal{D}_i$  and a virtual global data domain  $\mathcal{D}$ . Let  $\mathcal{D}_i = \langle \mathcal{U}_i, c^* \rangle$  and  $\mathcal{D} = \langle \mathcal{U}, c^* \rangle$ , where  $\mathcal{U}_i \subseteq \mathcal{X}$  and  $\mathcal{U} \subseteq \mathcal{X}$ . Given a feature extraction function  $\mathcal{F}: \mathcal{X} \mapsto \mathcal{Z}$  that shared between  $\mathcal{D}_i$  and  $\mathcal{D}$ , a random labeled sample of size  $m$  generated by applying  $\mathcal{F}$  to a random sample from  $\mathcal{U}_i$  labeled according to  $c^*$ , then for every  $h \in \mathcal{H}$ , with probability at least  $1 - \delta$ :
$$
\mathcal {L} _ {\mathcal {D}} (h) \leq \mathcal {L} _ {\hat {\mathcal {D}} _ {i}} (h) + \sqrt {\frac {4}{m} \left(d \log \frac {2 e m}{d} + \log \frac {4}{\delta}\right)} + d _ {\mathcal {H}} (\tilde {\mathcal {U}} _ {i}, \tilde {\mathcal {U}}) + \lambda_ {i},
$$
where  $\tilde{\mathcal{U}}_i$  and  $\tilde{\mathcal{U}}$  are the induced distributions of  $\mathcal{U}_i$  and  $\mathcal{U}$  under  $\mathcal{F}$ , respectively.  $\tilde{\mathcal{U}}_i \subseteq \mathcal{Z}$  and  $\tilde{\mathcal{U}} \subseteq \mathcal{Z}$ .  $\lambda_i := \min_h \mathcal{L}_{\mathcal{D}_i}(h) + \mathcal{L}_{\mathcal{D}}(h)$  denotes an oracle performance.
Corollary 1. Consider a local data domain  $\mathcal{D}_i$  and a virtual global data domain  $\mathcal{D}$  for client  $i$  and the server, respectively. Let  $\mathcal{D}_i = \langle \mathcal{U}_i, c^* \rangle$  and  $\mathcal{D} = \langle \mathcal{U}, c^* \rangle$ , where  $c^* : \mathcal{X} \mapsto \mathcal{Y}$  is a ground-truth labeling function. Let  $\mathcal{H}$  be a hypothesis space of VC dimension  $d$  and  $h : \mathcal{Z} \mapsto \mathcal{Y}, \forall h \in \mathcal{H}$ . When using DBE, given a feature extraction function  $\mathcal{F}^g : \mathcal{X} \mapsto \mathcal{Z}$  that shared between  $\mathcal{D}_i$  and  $\mathcal{D}$ , a random labeled sample of size  $m$  generated by applying  $\mathcal{F}^g$  to a random sample from  $\mathcal{U}_i$  labeled according to  $c^*$ , then for every  $h^g \in \mathcal{H}$ , with probability at least  $1 - \delta$ :
$$
\mathcal {L} _ {\mathcal {D}} \left(h ^ {g}\right) \leq \mathcal {L} _ {\hat {\mathcal {D}} _ {i}} \left(h ^ {g}\right) + \sqrt {\frac {4}{m} \left(d \log \frac {2 e m}{d} + \log \frac {4}{\delta}\right)} + d _ {\mathcal {H}} \left(\tilde {\mathcal {U}} _ {i} ^ {g}, \tilde {\mathcal {U}} ^ {g}\right) + \lambda_ {i},
$$
where  $\mathcal{L}_{\tilde{\mathcal{D}}_i}$  is the empirical loss on  $\mathcal{D}_i$ ,  $e$  is the base of the natural logarithm, and  $d_{\mathcal{H}}(\cdot,\cdot)$  is the  $\mathcal{H}$ -divergence between two distributions.  $\lambda_i := \min_{h^g} \mathcal{L}_{\mathcal{D}}(h^g) + \mathcal{L}_{\mathcal{D}_i}(h^g), \tilde{\mathcal{U}}_i^g \subseteq \mathcal{Z}, \tilde{\mathcal{U}}^g \subseteq \mathcal{Z}$ , and  $d_{\mathcal{H}}(\tilde{\mathcal{U}}_i^g, \tilde{\mathcal{U}}^g) \leq d_{\mathcal{H}}(\tilde{\mathcal{U}}_i, \tilde{\mathcal{U}})$ .  $\tilde{\mathcal{U}}_i^g$  and  $\tilde{\mathcal{U}}^g$  are the induced distributions of  $\mathcal{U}_i$  and  $\mathcal{U}$  under  $\mathcal{F}^g$ , respectively.  $\tilde{\mathcal{U}}_i$  and  $\tilde{\mathcal{U}}$  are the induced distributions of  $\mathcal{U}_i$  and  $\mathcal{U}$  under  $\mathcal{F}$ , respectively.  $\mathcal{F}$  is the feature extraction function in the original FedAvg without DBE.
Proof. Computing  $d_{\mathcal{H}}(\cdot, \cdot)$  is identical to learning a classifier to achieve a minimum error of discriminating between points sampled from  $\tilde{\mathcal{U}}$  and  $\tilde{\mathcal{U}}'$ , i.e., a binary domain classification problem [6, 7]. The more difficult
the domain classification problem is, the smaller  $d_{\mathcal{H}}(\cdot ,\cdot)$  is. Unfortunately, computing the error of the optimal hyperplane classifier for arbitrary distributions is a well-known NP-hard problem [5, 6]. Thus, researchers approximate the error by learning a linear classifier for the binary domain classification [5, 9, 10]. Inspired by previous approaches [4, 43, 57], we consider using Linear Discriminant Analysis (LDA) for the binary domain classification. The discrimination ability of LDA is measured by the Fisher discriminant ratio (F1) [11, 30, 76]
$$
F 1 \left(\tilde {\mathcal {U}} ^ {a}, \tilde {\mathcal {U}} ^ {b}\right) = \max _ {k} \left[ \frac {\left(\boldsymbol {\mu} _ {\tilde {\mathcal {U}} ^ {a}} ^ {k} - \boldsymbol {\mu} _ {\tilde {\mathcal {U}} ^ {b}} ^ {k}\right) ^ {2}}{\left(\boldsymbol {\sigma} _ {\tilde {\mathcal {U}} ^ {a}} ^ {k}\right) ^ {2} + \left(\boldsymbol {\sigma} _ {\tilde {\mathcal {U}} ^ {b}} ^ {k}\right) ^ {2}} \right],
$$
where  $\pmb{\mu}_{\tilde{\mathcal{U}}^a}^k$  and  $(\sigma_{\tilde{\mathcal{U}}^a}^k)^2$  are the mean and variance of the values in the  $k$ th dimension over  $\tilde{\mathcal{U}}^a$ . The smaller the Fisher discriminant ratio is, the less discriminative the two domains are. Theorem 2 holds with every  $h \in \mathcal{H}$ , so we omit PRBM here. MR  $(\bar{z}_i^g, \bar{z}^g)$  forces the local domain to be close to the global domain in terms of the mean value at each feature dimension in the feature representation independently, therefore,  $\forall k \in [K]$ ,
$$
\boldsymbol {\mu} _ {\tilde {\mathcal {U}} _ {i} ^ {g}} ^ {k} - \boldsymbol {\mu} _ {\tilde {\mathcal {U}} ^ {g}} ^ {k} \leq \boldsymbol {\mu} _ {\tilde {\mathcal {U}} _ {i}} ^ {k} - \boldsymbol {\mu} _ {\tilde {\mathcal {U}}} ^ {k}.
$$
As the feature extractors share the same structure with identical parameter initialization and the feature representations are extracted from the same data domain  $\mathcal{D}_i$  ( $\mathcal{D}$ ) [18, 34], we assume that  $\sigma_{\tilde{\mathcal{U}}_i^g} = \sigma_{\tilde{\mathcal{U}}_i}$  and  $\sigma_{\tilde{\mathcal{U}}^g} = \sigma_{\tilde{\mathcal{U}}}$ . Thus,  $\forall k \in [K]$ ,
$$
\frac {\left(\boldsymbol {\mu} _ {\tilde {\mathcal {U}} _ {i} ^ {g}} ^ {k} - \boldsymbol {\mu} _ {\tilde {\mathcal {U}} ^ {g}} ^ {k}\right) ^ {2}}{\left(\boldsymbol {\sigma} _ {\tilde {\mathcal {U}} _ {i} ^ {g}} ^ {k}\right) ^ {2} + \left(\boldsymbol {\sigma} _ {\tilde {\mathcal {U}} ^ {g}} ^ {k}\right) ^ {2}} \leq \frac {\left(\boldsymbol {\mu} _ {\tilde {\mathcal {U}} _ {i}} ^ {k} - \boldsymbol {\mu} _ {\tilde {\mathcal {U}}} ^ {k}\right) ^ {2}}{\left(\boldsymbol {\sigma} _ {\tilde {\mathcal {U}} _ {i}} ^ {k}\right) ^ {2} + \left(\boldsymbol {\sigma} _ {\tilde {\mathcal {U}}} ^ {k}\right) ^ {2}}.
$$
As this inequality is satisfied in all dimensions including the dimension where the maximum value exists, so for the Fisher discriminant ratio, we have
$$
F 1 \left(\tilde {\mathcal {U}} _ {i} ^ {g}, \tilde {\mathcal {U}} ^ {g}\right) = \max _ {k} \left[ \frac {\left(\boldsymbol {\mu} _ {\tilde {\mathcal {U}} _ {i} ^ {g}} ^ {k} - \boldsymbol {\mu} _ {\tilde {\mathcal {U}} g} ^ {k}\right) ^ {2}}{\left(\boldsymbol {\sigma} _ {\tilde {\mathcal {U}} _ {i} ^ {g}} ^ {k}\right) ^ {2} + \left(\boldsymbol {\sigma} _ {\tilde {\mathcal {U}} g} ^ {k}\right) ^ {2}} \right] \leq \max _ {k} \left[ \frac {\left(\boldsymbol {\mu} _ {\tilde {\mathcal {U}} _ {i}} ^ {k} - \boldsymbol {\mu} _ {\tilde {\mathcal {U}}} ^ {k}\right) ^ {2}}{\left(\boldsymbol {\sigma} _ {\tilde {\mathcal {U}} _ {i}} ^ {k}\right) ^ {2} + \left(\boldsymbol {\sigma} _ {\tilde {\mathcal {U}}} ^ {k}\right) ^ {2}} \right] = F 1 \left(\tilde {\mathcal {U}} _ {i}, \tilde {\mathcal {U}}\right).
$$
The smaller the Fisher discriminant ratio is, the less discriminative the two domains are. The less discriminative the two domains are, the smaller  $d_{\mathcal{H}}(\cdot ,\cdot)$  is. Thus, finally, we have
$$
d _ {\mathcal {H}} \left(\tilde {\mathcal {U}} _ {i} ^ {g}, \tilde {\mathcal {U}} ^ {g}\right) \leq d _ {\mathcal {H}} \left(\tilde {\mathcal {U}} _ {i}, \tilde {\mathcal {U}}\right).
$$
B.3 Derivations of Corollary 2
When we focus on the global-to-local knowledge transfer on the client side, in the FL scenario, we rewrite Theorem 1 as
Theorem 3. Consider a virtual global data domain  $\mathcal{D}$  and a local data domain  $\mathcal{D}_i$ . Let  $\mathcal{D} = \langle \mathcal{U}, c^* \rangle$  and  $\mathcal{D}_i = \langle \mathcal{U}_i, c^* \rangle$ , where  $\mathcal{U} \subseteq \mathcal{X}$  and  $\mathcal{U}_i \subseteq \mathcal{X}$ . Given a feature extraction function  $\mathcal{F}: \mathcal{X} \mapsto \mathcal{Z}$  that shares between  $\mathcal{D}$  and  $\mathcal{D}_i$ , a random labeled sample of size  $m$  generated by applying  $\mathcal{F}$  to a random sample from  $\mathcal{U}$  labeled according to  $c^*$ , then for every  $h \in \mathcal{H}$ , with probability at least  $1 - \delta$ :
$$
\mathcal {L} _ {\mathcal {D} _ {i}} (h) \leq \mathcal {L} _ {\hat {\mathcal {D}}} (h) + \sqrt {\frac {4}{m} \left(d \log \frac {2 e m}{d} + \log \frac {4}{\delta}\right)} + d _ {\mathcal {H}} (\tilde {\mathcal {U}}, \tilde {\mathcal {U}} _ {i}) + \lambda_ {i},
$$
where  $\tilde{\mathcal{U}}_i$  and  $\tilde{\mathcal{U}}$  are the induced distributions of  $\mathcal{U}_i$  and  $\mathcal{U}$  under  $\mathcal{F}$ , respectively.  $\tilde{\mathcal{U}}_i \subseteq \mathcal{Z}$  and  $\tilde{\mathcal{U}} \subseteq \mathcal{Z}$ .  $\lambda_i := \min_h \mathcal{L}_{\mathcal{D}}(h) + \mathcal{L}_{\mathcal{D}_i}(h)$  denotes an oracle performance.
Corollary 2. Let  $\mathcal{D}_i$ ,  $\mathcal{D}$ ,  $\mathcal{F}^g$ , and  $\lambda_i$  defined as in Corollary 1. Given a translation transformation function  $PRBM: \mathcal{Z} \mapsto \mathcal{Z}$  that shared between  $\mathcal{D}_i$  and virtual  $\mathcal{D}$ , a random labeled sample of size  $m$  generated by applying  $\mathcal{F}'$  to a random sample from  $\mathcal{U}_i$  labeled according to  $c^*$ ,  $\mathcal{F}' = PRBM \circ \mathcal{F}^g: \mathcal{X} \mapsto \mathcal{Z}$ , then for every  $h' \in \mathcal{H}$  with probability at least  $1 - \delta$ :
$$
\mathcal {L} _ {\mathcal {D} _ {i}} \left(h ^ {\prime}\right) \leq \mathcal {L} _ {\tilde {\mathcal {D}}} \left(h ^ {\prime}\right) + \sqrt {\frac {4}{m} \left(d \log \frac {2 e m}{d} + \log \frac {4}{\delta}\right)} + d _ {\mathcal {H}} \left(\tilde {\mathcal {U}} ^ {\prime}, \tilde {\mathcal {U}} _ {i} ^ {\prime}\right) + \lambda_ {i},
$$