Year: 2021

Abstract:
Unsupervised domain adaptation (UDA) aims to transfer knowledge from one or more well-labeled source domains to improve model performance on the differentyet-related target domain without any annotations. However, existing UDA algorithms fail to bring any benefits to source domains and neglect privacy protection during data sharing. With these considerations, we define Privacy Protected MultiDomain Collaborative Learning (PMDCL) and propose a novel Mask-Driven Federated Network (MDFNet) to reach a “win-win” deal for multiple domains with data protected. First, each domain is armed with individual local model via a mask disentangled mechanism to learn domain-invariant semantics. Second, the centralized server refines the global invariant model by integrating and exchanging local knowledge across all domains. Moreover, adaptive self-supervised optimization is deployed to learn discriminative features for unlabeled domains. Finally, theoretical studies and experimental results illustrate rationality and effectiveness of our method on solving PMDCL.

heading: 1 INTRODUCTION
text: Unsupervised domain adaptation (UDA) (Tang et al., 2020; Jiang et al., 2020; Zhang et al., 2020) attempts to transfer knowledge from well-labeled source domains to annotate unlabeled target samples, which have significant domain discrepancy with source domains due to the various data collection manners and devices. Recent explorations (Na et al., 2021; Dong et al., 2020) suppose the model to be trained has access to both source and target data during the training stage. With such basic assumption, it becomes possible to measure the domain discrepancy and adopt metric-based solutions (Kang et al., 2020) or domain confusion (Cui et al., 2020; Tang & Jia, 2020) to generate domain-invariant features. However, the hypothesis violates the concerns of practical application on privacy protection, and fails to be deployed to small devices with limited storage.
This requirement motivates source-free domain adaptation (SFDA), where the source-supervised model is available to assist the target domain without any source data (Liang et al., 2020; Li et al., 2020; Kundu et al., 2020). Generally, SFDA either adapts target samples to source-like ones (Liang et al., 2020) or generates fake source samples from source-model by subsequently taking UDA strategies (Kurmi et al., 2021). To improve the training efficiency, FADA (Peng et al., 2020) employs a federated learning paradigm (Karimireddy et al., 2020; Chen et al., 2020) by allocating the target domain on a centralized server while keeping multiple source ones as clients. However, this approach is vulnerable to attacks as the source features transition to target domain. Further, these domain adaptation works ignore the improvement of model generalization on source domain, which is inconsistent with requirement of reality. For example, the long-standing hospitals already have well-annotated patients’ data, while other newly-built hospitals just collected data without annotation, which need help from long-standing hospitals with well-annotated data due to the huge labeling cost. Besides, with geographical restriction, different hospitals only record their local patients’ data resulting in various population statistics, causing model bias for long-standing hospitals.
Inspired by the above observation, we introduce a more practical scenario called Privacy Protected Multi-Domain Collaborative Learning (P2MDCL) (shown in Figure 1). Specifically, P2MDCL assumes that the well-annotated source and unlabeled target domains are distributed across different clients and there exists a global server merely communicating with each client and integrating the received model parameters from clients. Finally, the server broadcasts the consensus model to all
clients for their use to reach the win-win deal. The key challenge for P2MDCL is to learn a more generic model by solving two core issues: 1) how to achieve domain alignment during iterative communication; and 2) how to enhance discriminative feature learning.
In this paper, we propose a novel Mask-Driven Federated Network (MDFNet) to address P2MDCL. First, our MDFNet introduces two orthogonal masks following high-level features in each client to activate domain-invariant and domain-specific semantics respectively. In practice, we minimize the confusion of these two masks to achieve high-quality feature separation and semantic complementary. Second, the unlabeled target client adopts adaptive self-supervised optimization to learn more discrimina-
tive representations via pseudo labels generation. Finally, MDFNet adopts a progressive weighting scheme to balance the effect of each client in model integration on the server, which discoveries more knowledge of the labeled client to adjust the model of unlabeled client during the initial communication rounds, then the mature unlabeled client model also yields positive effect on the feature learning of labeled client. The main contributions of our work are summarized as:
• First, we are the pioneers to take into account the “win-win” and privacy requirements under unsupervised domain adaptation scenarios by introducing Privacy Protected MultiDomain Collaborative Learning (P2MDCL).
• Second, we propose an effective algorithm MDFNet to fight off the domain shift in a federated training mechanism, which reaches the win-win deal for all involved domains.
• Finally, we derive the generalized error bound for our method, which theoretically testifies the rationality of MDFNet. Moreover, extensive experimental results and analysis empirically illustrate the effectiveness of our method on solving P2MDCL.

heading: 2 RELATED WORK
text: Domain Adaptation. Unsupervised domain adaptation (Cui et al., 2020) attempts to build a model with well-labeled source and unlabeled target data at hand, by mitigating the domain mismatch. Along this line, the recent explorations mainly adopt discrepancy metric-based method (Yan et al., 2017; Tzeng et al., 2014) and adversarial training scheme (Zhang et al., 2019; Tzeng et al., 2017) to learn domain-invariant features. Although these solutions effectively reduce the influence of domain discrepancy, the practical applications difficultly permit the co-existence of source and target data due to the limited storage of small device and data privacy. The demand stimulates the development of source-free domain adaptation (Liang et al., 2020; Kurmi et al., 2021), which merely provides the well-trained source model for knowledge adaption on target domain. In addition, Peng et al. (2020) respectively considers target domain and multiple source domains as the centralized server and clients and adopts federated learning fashion to achieve domain adaptation with multiple discriminators, which is vulnerable to the attack due to the source and target features transmission into the discriminators in the centralized target domain. Even though these strategies actually achieve the comparable transferring ability with the UDA solutions, empirical studies illustrate the current domain adaptation techniques fail to learn a generalized model for source and target domains. Alternatively, they only focus on the improvement of target performance, yet neglecting any benefit to source domain. To this end, this paper posts a novel and practical scenario privacy protected multidomain collaborative learning (P2MDCL), where source and target domains are both regarded as clients independently communicating with the server which produces and broadcasts the consensus model to clients for their use.
Federated Learning (FL). FL allows multi-clients collaboratively to complete the same task without data currency across clients (Yang et al., 2019). Along with this concept, recent works mainly focus on semi-supervised scenario (FSSL) where FedMatch (Jeong et al., 2021) allocates unlabeled
data on client side and labeled data in the server while FedIRM (Liu et al., 2021) only deploys them on various clients. But they both assume the instances across all client are sampled from the identical distribution. Moreover, Smith et al. (2017); Liu et al. (2020) explore FSSL with non-i.i.d by supposing each client contains several well-annotated instances for training. Differently, our considered P2MDCL closely approximates the reality, which involves several clients without any annotations and exists significant domain discrepancy across all clients.

heading: 3 THE PROPOSED METHOD

heading: 3.1 PROBLEM DEFINITION AND MOTIVATION
text: The P2MDCL scenario assumes there are L well-annotated source clients Dli = {(xl(i)j , y l (i)j)} nli j=1 (i ∈ {1, · · · , L}) and U unlabeled target clients Duk = {xu(k)j} nuk j=1 (k ∈ {L+ 1, · · · , L+ U}), where x and y denote an input sample and its ground-truth label, respectively. The instances of these clients come from different distributions but share the identical category space and clients are not allowed to exchange private data with each other. Akin to federated learning, the additional global server in P2MDCL collects and assembles all clients’ network parameters to form the consensus model. The main motivation of P2MDCL is addressing the negative effect of insufficient training samples in Dli and label shortage in Duk to reach a “win-win” deal across all clients. We face two challenges to solve P2MDCL: 1) how to reduce the significant distribution discrepancy while protecting data privacy and 2) how to learn more generic and discriminative representations from unlabeled target clients. To this end, this work proposes an effective Mask-Driven Federated Network (MDFNet), which deploys mask-driven disentanglement to locally seek domain-specific/invariant features, and explores the adaptive self-supervised optimization to promote the discriminative ability of unlabeled target clients.

heading: 3.2 MASK-DRIVEN DISENTANGLEMENT
text: Feature separation is a commonly-used strategy in domain adaptation to disentangle latent representation into domain-specific features and domain-invariant ones (Bousmalis et al., 2016; Peng et al., 2019). However, they typically develop two separated networks to extract the corresponding features, which increase storage burden for each local device with insufficient computational resources. Peng et al. (2019) points out the high-level neurons from feature extractor actually involve domain-
specific and invariant knowledge. Inspired by (Chattopadhyay et al., 2020), we explore the binary mask to achieve feature disentanglement by activating the interested neurons.
For the brevity, we omit the symbols l/u and (k) in the following illustration. As Figure 2 shows, each client of our MDFNet contains a basic feature encoder parameterized θe mapping the raw input into the hidden space via gi = θe(xi) ∈ Rd. Subsequently, two additional parameters m̂s, m̂I ∈ Rd are introduced into the local network and activated to form the mask probabilities by using the sigmoid function σ(·) to get ms = σ(m̂s) and mI = σ(m̂I). For each feature gi, based on the mask probabilities, we sample the binary domain-specific and invariant masks (msi ,m I i ∈ {0, 1}d) from the Bernoulli distributions. To this end, we obtain the domain specific and invariant features with the element-wise multiplication⊗ over binary masks and features, i.e., gsi = msi ⊗gi and gIi = mIi ⊗gi. Moreover, we adopt three strategies to achieve high-quality feature separation. Concretely, each client firstly minimizes the semantic overlap between gIi and g s i to store complementary information in them. Motivated by Rahman & Wang (2016), we design the following soft-interactive loss as:
Ls = ∑
i 〈gsi , gIi 〉 sum(gsi + g I i − gsi ⊗ gIi ) , (1)
where 〈, 〉 means the inner product of two feature vectors, and sum(·) represents the sum of all elements for a vector. This approximately reflects the information overlap of two mask distributions. The minimization of soft-interactive loss gradually increases the difference between msi and m I i which activate different neurons. Similar to DSN (Bousmalis et al. (2016)), each client also develops the individual classifier θc(·) taking domain-invariant features as input to the category probability distribution θc(gIj ). The cross-entropy loss between ground-truth and prediction intensifies the discriminnative ability of domain-invariant features. On the other hand, we also feed the combination of gsi and g I i into decoder θd(·) to reconstruct the original input with Lr = ∑ i ‖θd(gsi , gIi ) − xi‖22. Thus, the overall loss function of mask-driven disentanglement for labeled clients is formulated as:
min θc,θe,θd,m̂s,m̂I
Llo = ∑
i −yi log
( θc(g I i ) ) + Lr + Ls, (2)
where we actually adopt straight-through estimator (Bengio et al., 2013) to progressively optimize m̂s and m̂I instead of the discrete binary masks leading to the invalid back-propagation.

heading: 3.3 ADAPTIVE SELF-SUPERVISED OPTIMIZATION
text: Due to the availability of annotation in the well-labeled clients, we can easily calibrate the predicted category distribution to generate discriminative features by using the supervision of ground-truth. However, we cannot directly adopt the supervised learning manner to optimize the model in unlabeled clients with the absence of annotation. Inspired by the successful application of pseudo-label on solving UDA issue (Xie et al., 2018; Gu et al., 2020; Liang et al., 2019; Morerio et al., 2020), we thus propose the adaptive clustering optimization module to gradually produce the pseudo-label as “ground-truth” supervision.
Specifically, after each round of communication, the unlabeled client first receives the model broadcast from the server and uses it to initialize the parameters of θe, θd, θc, m̂s, m̂I . Before further optimizing, the client annotates its local data with the received global model, i.e., ŷj = arg maxk θc(g I j )k. With the predictions, the initial centroid of each category is computed
as Ok = ∑ j 1(ŷj=k)g I j∑
j 1(ŷj=k) , where 1(·) is the indicator function. Since the server model integrates
knowledge from multiple clients, the domain shift negatively affects the accuracy of inference ŷj . To decrease the influence, we adopt an iterative approach to further update the class centers and pseudo-label with the local data points. The proposed adaptive clustering optimization mainly includes two operations. The first step is to reassign the label for each instance with the spherical K-means (Buchta et al., 2012):
ŷj = arg min k
d̃ ( gIj ,Ok ) = 1
2
( 1−
〈gIj ,Ok〉 ‖gIj ‖ · ‖Ok‖
) . (3)
With the reattached annotations, the second step is to update the class prototype with Ok =∑ j 1(ŷj=k)g I j
‖gIj ‖ . The above two steps are repeated till convergence.
After the adaptive self-supervised optimization, we attain the final pseudo-label for each sample and use them as supervision to optimize the local models. However, not all samples of unlabeled clients would contribute to the parameter sharing due to the domain mismatch, which in turn causes the low reliability and uncertainty for some samples. In this way, we consider samples to be positive and negative samples, identifying their potential benefit for the labeled clients. Therefore, it is crucial to distinguish positive and negative samples by identifying their potential benefit to the labeled clients. To this end, we add additional entropy-minimization (EM) to further improve the certainty of category prediction, reformulate the Eq. (2) for the unlabeled clients as:
min θc,θe,θd,m̂s,m̂I
Luo = ∑
j
( −I(max(θc(gIj )) ≥ σ)yj log(θc(gIj ))−θc(gIj ) log(θc(gIj )) ) +Lr+Ls,
(4) where I(·) denotes the indicator to filter out samples with θc(gIj ) less than a threshold σ, which is set as 0.1 by default throughout our experiments.

heading: 3.3.1 FEDERATED TRAINING
text: The overall training of our MDFNet involves two important procedures: a) local clients training, and b) global sever model integration. The clients and server collaboratively update these steps per communication round and repeat the process until model convergence or reaching the maximum communication rounds.
Independent Client Training. In each round, the server broadcasts the consensus model integrated from the last round to all available clients for the initialization of local models. The well-annotated clients then employ their local data to optimize all the modules for one epoch via Eq. (2), while the clients without labels rely on the adaptive clustering optimization to generate pseudo-labels for their samples and update their models with Eq. (4). The clients will locally store the parameters of domain-specific mask and decoder and use them to initialize the network in the next round.
Model Integration. After the local training, the clients will send their local models (excluding the parameters of domain-specific mask and decoder) to the server, where the models are integrated to achieve consensus. However, adopting pseudo-labels as supervision significantly reduce the reliability of the models, especially in the initial training stages. To avoid the negative effect of pseudo-labels, the server considers providing different weights to labeled and unlabeled clients as: θ̃ = 1−ηrL ∑L i=1 θ(li) + ηr U ∑L+U i=L+1 θ(ui), where θ ∈ {θe, θd, θc, m̂I , m̂s} and ηr = 1−exp(−ρr) 2(1+exp(−ρr)) , r is the round of communication and each round means one epoch, and ρ is set as 10 in experiments.

heading: 3.4 GENERALIZED ERROR BOUND ANALYSIS
text: We firstly define the basic notation and employ them to derive the generalization error bound for P2MDCL from the high-level interpretation and the specific proofs are shown in the supplementary.
Notation. Given the distributions of labeled and unlabeled clients Dli and Duj on the input space X , we have access to the ground-truth labeling function fli : X → {0, 1} for the clients with the annotation, while also have the pseudo labeling function fuj : X → {0, 1} available for the unlabeled clients. A hypothesis is corresponding to a function h : X → {0, 1} with the error, i.e., li(h, fli) := Ex∼Dli [|h(x − fli(x)|] and uj (h, fuj ) := Ex∼Duj [|h(x − fuj (x)|]. Thus, the risk and the empirical risk of hypothesis h on Dli and Duj are respectively represented as li(h), ̂li(h), and uj (h), ̂uj (h). Moreover, we define the H-divergence between two arbitrary distributions D and D′ as dH(D,D ′ ) = 2 supA∈AH |PrD(A) − PrD′ (A)|, where H means the hypothesis class for input space X and AH is the collection of subsets of X that are the support of some hypothesis inH. The symmetric difference space with the hypothesis class is formulated asH∆H = {h(x) ∗ h′(x)|h, h′ ∈ H}, where ∗ denotes the XOR operation. Our model aims to learn a consensus model through the communication between the server and all available clients. Such learning strategy actually attempts to minimize a convex combination of empirical risks over all clients with parameters αi ( ∑L+U i=1 αi = 1) as ̂α = ∑L i=1 αîli(h) +∑L+U
i=L+1 αi ui(h). Similarly, we obtain the weighted combination of the risks over all clients as α(h). In addition, since each client independently trains the model with its specific data, we denote the optimal hypothesises achieving the minimum risk on the labeled and unlabeled clients as h∗li :=
arg minh∈H li(h) and h ∗ uj := arg minh∈H uj (h). With these definitions, it still is intractable to directly deduce the generalized error bound under this scenario. Therefore, we alternatively divide the entire problem into multiple sub-problems and solve them in each client.
Concretely, we first explore the relationship between α and li or uj with Lemma 1. Second, we drive the upper bound of the difference between α and ̂α via Lemma 2. Thus, we easily deduce the generalized error bound of a hypothesis per client in Theorem 1.
Lemma 1. Suppose the h is a hypothesis of classH, for each unlabeled client, we then achieve: | α(h)− uj (h)| ≤ L∑ i=1 αi( 1 2 dH∆H(Dli ,Duj ) + λli) + L+U∑ i=L+1,i6=j αi( 1 2 dH∆H(Dui ,Duj ) + λui),
where λli := li(h ∗) + uj (h ∗) and h∗ is the hypothesis which achieves the minimum risk on Dli and Duj , and λui similarly means the risk of optimal hypothesis on the mixture of Dui and Duj . Akin to unlabeled clients, we also derive the analogous inequality in clients with ground-truth as:
| α(h)− lj (h)| ≤ L∑
i=1,i6=j
αi( 1
2 dH∆H(Dli ,Dlj ) + λli) + L+U∑ i=L+1 αi( 1 2 dH∆H(Dui ,Dlj ) + λui),
where λli is the risk of optimal hypothesis of Dli and Dlj , and λui := ui(h∗) + lj (h∗). Lemma 2. Given a hypothesis spaceH of VC-dimension d, if a random sample of size n is generated by selecting nβj data points from Dlj or Duj , and annotating them through flj and fuj , then with probability at least 1− δ, ∀h ∈ H, we have:
|̂α(h)− α(h)| ≤ √∑L+U j=1 α2j βj √ d log(2n)− log δ 2n .
Theorem 1. Suppose given nβi labeled instances from client Dli for i = 1, · · · , L, and nβj unlabeled instances from client Duj in a federated learning system, we define ĥ = arg minh∈H ̂α(h), h∗li := arg minh∈H li(h) and h ∗ uj := arg minh∈H uj (h). Then, ∀αi ∈ R +, ∑L+U i=1 αi = 1, with probability at least 1− δ over the choice of samples from each client:
uj (ĥ) ≤ uj (h∗uj ) + 2 √∑L+U j=1 α2j βj √ d log(2n)− log δ 2n
+ 2 (∑L
i=1 αi(
1 2 dH∆H(Dli ,Duj ) + λli) + ∑L+U i=L+1,i6=j αi( 1 2 dH∆H(Dui ,Duj ) + λui) ) .
For the annotated client, we can achieve the similar inequality. From Theorem 1, we explicitly observe the risk of a hypothesis with the federated training manner on the client is determined by three components: the error of the optimal hypothesis h∗uj on its own samples, the VC-dimension constraint and the distribution discrepancy across various clients. To effectively reduce the risk of a hypothesis on all clients, we should not only learn the discriminative features by the independent client training, but also attempt to solve the domain shift with the constraint of data privacy.

heading: 4 EXPERIMENTS

heading: 4.1 EXPERIMENTAL SETTINGS
text: Datasets. Image-CLEF collects visual signals from three domains: Caltech-256 (C), ImageNet ILSVRC 2012 (I) and Pascal VOC 2012 (P) with the same number of samples. Concretely, arbitrary subset includes 600 images evenly distributed in 12 categories. Office-Home (Venkateswara et al., 2017) consists of four domains: Artistic images (Ar, 2,183), Clip Art (Cl, 4,365), Product images (Pr, 4,439) and Real-World images (Rw, 4,357), which share the identical 65 object categories. To verify the “win-win” deal, we randomly split the original data of labeled client into the training and test sets evenly, and repeat this operation for ten times1.
1We further report the comparison under the original protocol of SHOT in supplemental materials, where all source samples are used for training and the evaluation is only on target domain.
Baselines. To the best of our knowledge, we are the pioneers to consider P2MDCL scenario, and we aim to assess if our algorithm can learn a server model with higher generalization ability to enhance the performance across all clients via federated training. To explicitly testify the generalization of model on each client, this section focuses on P2MDCL with a labeled client and an unlabeled one. Noted that we report the P2MDCL with more clients in supplementary material. Since this scenario is similar with UDA and source-free DA, we not only select CDAN (Long et al., 2017) and SRDC (Tang et al., 2020) achieving the state-of-the-art results on UDA problem as the benchmarks and also regard the SHOT (Liang et al., 2020) as one important competitor. In addition, we also consider the source-only method merely training the model on the labeled client. For the mentioned baselines, we use their released code with the suggested parameters to carry out each task for ten times
Evaluation Metric. In terms of the data organization, the labeled client includes the training and testing sets without any overlap, while all samples of the unlabeled client participant the model training and evaluation. For UDA and source-free solutions, the training set of labeled client is considered as the source domain and the unlabeled client servers as the target domain. During the test stage, the final model learned by each method is not only evaluated on the test set of labeled client with the corresponding source accuracy ACCs but also tested on the unlabeled client with the target accuracy ACCt. Moreover, to comprehensively reflect the generalization of model, we adopt the Harmonic Mean (HM) (Dixon & Chapman, 1980) defined as HM = 2×ACCs×ACCtACCs+ACCt .
Implementation Details. We implement our MDFNet with Pytorch as platform. The encoder of each client includes ResNet-50 pre-trianed on ImageNet dataset (Krizhevsky et al., 2012) without the last FC layer and two new additional FC layers (2,048→512→128) followed the ResNet-50. The decoder consists of two FC layers (256→512→2,048) and the classifier only includes one FC layer. The dimensions of m̂s and m̂I are both 2,048. For the training period, we fix the parameters of the pre-trained ResNet-50 and adopt the stochastic gradient descent (SGD) as the optimizer with momentum 0.9. Following (Zhang et al., 2019), the learning rate is adjusted by ζr = ζ0(1+10r)0.75 where ζ0 = 0.01 and r is the communication round. The code is available in the supplementary.

heading: 4.2 RESULT ANALYSIS
text: Table 1 and Table 2 report the average image recognition results in terms of random data split and various model training. According to the results, we achieve four meaningful and interesting conclusions as below.
First, compared with others, the model learned by our training strategy achieves the best classification accuracy, when evaluated on the test set of labeled and unlabeled clients. In terms of the average harmonic metric, our MDFNet performs better than the second best SHOT by 3.4% on Office-Home. It illustrates that even if the data privacy hinders the currency of knowledge between these two clients, our method still employs the federated
training paradigm to gradually eliminate the domain shift across different clients to improve the generalization of model. Second, although the UDA based solutions and SHOT effectively facilitates the well-trained source models to adapt the data distribution of unlabeled client, the progressive adaptation discards considerable source knowledge and results in performance degradation on the test set of labeled client. For instance, CDAN achieves better average accuracy on
unlabeled client with office-home than source-only method, i.e., 61.42 v.s. 57.22%. However, such improvement heavily affects the generalization of CDAN on the labeled client, i.e., CDAN (75.75%) vs Src-only (85.31%). Different from CDAN, our method even attains the more improvement than the source-only method on the test set, especially for the task P→I of Image-CLEF dataset, where our MDFNet surpasses the source-only method by 6.9%. Thirdly, even though SHOT and MDFNet both protect the data privacy of source domains, our MDFNet learns a better hypothesis with lower error on the unlabeled client. Concretely, for the task Ar→Rw, when assessed on the unlabeled client, our method fights off SHOT by 4.5%, which means our training manner captures more knowledge from the labeled client via the frequent communication between server and clients to improve the discriminative ability of model on recognizing unlabeled instances. Finally, we find that all methods achieve higher classification accuracy on unlabeled clients than that of labeled clients for tasks P→C and P→I. Specifically, with P as the source domain, the well-trained Src-Only model achieves better performance on the target domain than that of source test set. The main reason lies in the fact that the samples of P domain lie in a more diverse distribution within class, which makes it difficult to learn a discriminative model to recognize its images. Concretely, P domain includes many images with multiple objects but one single label. For example, one image of ”bird” class in P domain involves bird and dog, and another image includes bird and person. However, there are almost no such multi-object images in C and I domains. Moreover, the same class in P domain has more animals than that of C or I domain. For instance, besides several common birds as C and I domains, the bird class of P domain also consists chicken and ostrich, etc. Thus, the well-trained source model with P domain can easily classify the images of C and I domains, but difficultly recognize its source test set.

heading: 4.3 EMPIRICAL ANALYSIS
text: Feature Visualization & Confusion Matrix. Our MDFNet aims to eliminate the distribution discrepancy across different clients and improve the generalization of model with the data privacy protection. Thus, we extract the hidden features from Src-only, SHOT and MDFNet on task Ar→Rw
and follow (Zhang et al., 2019) to draw the the feature embedding of the test samples of labeled client and those of unlabeled client in 2-D canvas. From Figure 3, we notice that our MDFNet achieves better alignment across these two clients and significantly promotes the discriminative ability of model as class boundaries are explicit among various categories. Moreover, we utilize the confusion matrix to analyse the model performance on the test set of labeled client (P domain of Image-CLEF). As Figure 4 shows, our method accurately distinguishes several similar objects as bike and motorcycle when compared with Src-only, which illustrates our MDFNet transfers the valuable semantic from unlabeled client to assist model recognizing the samples of labeled client.
Ablation Study & Convergence. To reveal the importance of adaptive self-supervised optimization module, we attempt to remove the supervisor of pseudo-label and consider it as a variant (Ours-SSO) of MDFNet and evaluate the model on the unlabeled client. Their comparison is reported in Figure 5 (c) where the variant suffers from the obvious performance degradation, which demonstrates the self-supervised learning based module effectively facilitates the model to learn more discriminative features. In addition, we further adjust the number of training samples in the labeled client (training set/test set = 3:7) and contrast the performances of three methods on the unlabeled client. The result in Figure 5 (a) reports our MDFNet still beats others in most tasks even with insufficient well-annotated samples. Finally, we record the relationship between classification accuracy and the communication round in Figure 5 (b) where the model is assessed on the test set of labeled client and unlabeled client. The performance means MDFNet rapidly achieves the convergence and has no negative effect on recognizing samples of labeled client during adaptation.

heading: 5 CONCLUSION
text: Although UDA based methods effectively avoid performance degradation when applying source knowledge to target domain, the UDA assumption ignores the improvement of model generalization on source domain and conflicts with privacy protection. Thus, this paper formulates these practical demands with domain adaptation as a novel scenario P2MDCL and proposes mask-driven federated network (MDFNet) to address this challenge. Concretely, each individual domain explores mask disentangle mechanism to learn domain-invariant features, and the unlabeled clients exploits adaptive self-supervised optimization to generate high-quality pseudo labels facilitating discriminative feature learning. Moreover, the centralized server refines the global invariant by assembling local knowledge across all domains. Finally, theoretical and experimental analysis demonstrate the rationality and effectiveness of our MDFNet on solving P2MDCL problem.

Review summaries:
This paper studies the problem of privacy-protected multi-domain collaborative filtering, in which a "win-win" deal for source and target domains can be achieved. The proposed framework, MDFNet, contains multiple local clients and one global server. In each client, the encoder achieves feature separation, and the decoder constructs original data based on separated features. Experiments on benchmark datasets demonstrate the best performance of the proposed MDFNet compared with baselines. 

This paper proposes a Mask-Driven Federated Network (MDFNet) to reach a “win-win” deal for multiple domains with data protected to solve the Privacy Protected Multi-Domain Collaborative Learning (P2MDCL) problem. Specifically, each domain is armed with an individual local model via a mask disentangled mechanism to learn domain-invariant semantics. Second, the centralized server refines the global invariant model by integrating and exchanging local knowledge across all domains. Moreover, adaptive self-supervised optimization is deployed to learn discriminative features for unlabeled domains.