Privacy Protected Multi-Domain Collaborative Learning

1 INTRODUCTION . Unsupervised domain adaptation ( UDA ) ( Tang et al. , 2020 ; Jiang et al. , 2020 ; Zhang et al. , 2020 ) attempts to transfer knowledge from well-labeled source domains to annotate unlabeled target samples , which have significant domain discrepancy with source domains due to the various data collection manners and devices . Recent explorations ( Na et al. , 2021 ; Dong et al. , 2020 ) suppose the model to be trained has access to both source and target data during the training stage . With such basic assumption , it becomes possible to measure the domain discrepancy and adopt metric-based solutions ( Kang et al. , 2020 ) or domain confusion ( Cui et al. , 2020 ; Tang & Jia , 2020 ) to generate domain-invariant features . However , the hypothesis violates the concerns of practical application on privacy protection , and fails to be deployed to small devices with limited storage . This requirement motivates source-free domain adaptation ( SFDA ) , where the source-supervised model is available to assist the target domain without any source data ( Liang et al. , 2020 ; Li et al. , 2020 ; Kundu et al. , 2020 ) . Generally , SFDA either adapts target samples to source-like ones ( Liang et al. , 2020 ) or generates fake source samples from source-model by subsequently taking UDA strategies ( Kurmi et al. , 2021 ) . To improve the training efficiency , FADA ( Peng et al. , 2020 ) employs a federated learning paradigm ( Karimireddy et al. , 2020 ; Chen et al. , 2020 ) by allocating the target domain on a centralized server while keeping multiple source ones as clients . However , this approach is vulnerable to attacks as the source features transition to target domain . Further , these domain adaptation works ignore the improvement of model generalization on source domain , which is inconsistent with requirement of reality . For example , the long-standing hospitals already have well-annotated patients ’ data , while other newly-built hospitals just collected data without annotation , which need help from long-standing hospitals with well-annotated data due to the huge labeling cost . Besides , with geographical restriction , different hospitals only record their local patients ’ data resulting in various population statistics , causing model bias for long-standing hospitals . Inspired by the above observation , we introduce a more practical scenario called Privacy Protected Multi-Domain Collaborative Learning ( P2MDCL ) ( shown in Figure 1 ) . Specifically , P2MDCL assumes that the well-annotated source and unlabeled target domains are distributed across different clients and there exists a global server merely communicating with each client and integrating the received model parameters from clients . Finally , the server broadcasts the consensus model to all clients for their use to reach the win-win deal . The key challenge for P2MDCL is to learn a more generic model by solving two core issues : 1 ) how to achieve domain alignment during iterative communication ; and 2 ) how to enhance discriminative feature learning . In this paper , we propose a novel Mask-Driven Federated Network ( MDFNet ) to address P2MDCL . First , our MDFNet introduces two orthogonal masks following high-level features in each client to activate domain-invariant and domain-specific semantics respectively . In practice , we minimize the confusion of these two masks to achieve high-quality feature separation and semantic complementary . Second , the unlabeled target client adopts adaptive self-supervised optimization to learn more discrimina- tive representations via pseudo labels generation . Finally , MDFNet adopts a progressive weighting scheme to balance the effect of each client in model integration on the server , which discoveries more knowledge of the labeled client to adjust the model of unlabeled client during the initial communication rounds , then the mature unlabeled client model also yields positive effect on the feature learning of labeled client . The main contributions of our work are summarized as : • First , we are the pioneers to take into account the “ win-win ” and privacy requirements under unsupervised domain adaptation scenarios by introducing Privacy Protected MultiDomain Collaborative Learning ( P2MDCL ) . • Second , we propose an effective algorithm MDFNet to fight off the domain shift in a federated training mechanism , which reaches the win-win deal for all involved domains . • Finally , we derive the generalized error bound for our method , which theoretically testifies the rationality of MDFNet . Moreover , extensive experimental results and analysis empirically illustrate the effectiveness of our method on solving P2MDCL . 2 RELATED WORK . Domain Adaptation . Unsupervised domain adaptation ( Cui et al. , 2020 ) attempts to build a model with well-labeled source and unlabeled target data at hand , by mitigating the domain mismatch . Along this line , the recent explorations mainly adopt discrepancy metric-based method ( Yan et al. , 2017 ; Tzeng et al. , 2014 ) and adversarial training scheme ( Zhang et al. , 2019 ; Tzeng et al. , 2017 ) to learn domain-invariant features . Although these solutions effectively reduce the influence of domain discrepancy , the practical applications difficultly permit the co-existence of source and target data due to the limited storage of small device and data privacy . The demand stimulates the development of source-free domain adaptation ( Liang et al. , 2020 ; Kurmi et al. , 2021 ) , which merely provides the well-trained source model for knowledge adaption on target domain . In addition , Peng et al . ( 2020 ) respectively considers target domain and multiple source domains as the centralized server and clients and adopts federated learning fashion to achieve domain adaptation with multiple discriminators , which is vulnerable to the attack due to the source and target features transmission into the discriminators in the centralized target domain . Even though these strategies actually achieve the comparable transferring ability with the UDA solutions , empirical studies illustrate the current domain adaptation techniques fail to learn a generalized model for source and target domains . Alternatively , they only focus on the improvement of target performance , yet neglecting any benefit to source domain . To this end , this paper posts a novel and practical scenario privacy protected multidomain collaborative learning ( P2MDCL ) , where source and target domains are both regarded as clients independently communicating with the server which produces and broadcasts the consensus model to clients for their use . Federated Learning ( FL ) . FL allows multi-clients collaboratively to complete the same task without data currency across clients ( Yang et al. , 2019 ) . Along with this concept , recent works mainly focus on semi-supervised scenario ( FSSL ) where FedMatch ( Jeong et al. , 2021 ) allocates unlabeled data on client side and labeled data in the server while FedIRM ( Liu et al. , 2021 ) only deploys them on various clients . But they both assume the instances across all client are sampled from the identical distribution . Moreover , Smith et al . ( 2017 ) ; Liu et al . ( 2020 ) explore FSSL with non-i.i.d by supposing each client contains several well-annotated instances for training . Differently , our considered P2MDCL closely approximates the reality , which involves several clients without any annotations and exists significant domain discrepancy across all clients . 3 THE PROPOSED METHOD . 3.1 PROBLEM DEFINITION AND MOTIVATION . The P2MDCL scenario assumes there are L well-annotated source clients Dli = { ( xl ( i ) j , y l ( i ) j ) } nli j=1 ( i ∈ { 1 , · · · , L } ) and U unlabeled target clients Duk = { xu ( k ) j } nuk j=1 ( k ∈ { L+ 1 , · · · , L+ U } ) , where x and y denote an input sample and its ground-truth label , respectively . The instances of these clients come from different distributions but share the identical category space and clients are not allowed to exchange private data with each other . Akin to federated learning , the additional global server in P2MDCL collects and assembles all clients ’ network parameters to form the consensus model . The main motivation of P2MDCL is addressing the negative effect of insufficient training samples in Dli and label shortage in Duk to reach a “ win-win ” deal across all clients . We face two challenges to solve P2MDCL : 1 ) how to reduce the significant distribution discrepancy while protecting data privacy and 2 ) how to learn more generic and discriminative representations from unlabeled target clients . To this end , this work proposes an effective Mask-Driven Federated Network ( MDFNet ) , which deploys mask-driven disentanglement to locally seek domain-specific/invariant features , and explores the adaptive self-supervised optimization to promote the discriminative ability of unlabeled target clients . 3.2 MASK-DRIVEN DISENTANGLEMENT . Feature separation is a commonly-used strategy in domain adaptation to disentangle latent representation into domain-specific features and domain-invariant ones ( Bousmalis et al. , 2016 ; Peng et al. , 2019 ) . However , they typically develop two separated networks to extract the corresponding features , which increase storage burden for each local device with insufficient computational resources . Peng et al . ( 2019 ) points out the high-level neurons from feature extractor actually involve domain- specific and invariant knowledge . Inspired by ( Chattopadhyay et al. , 2020 ) , we explore the binary mask to achieve feature disentanglement by activating the interested neurons . For the brevity , we omit the symbols l/u and ( k ) in the following illustration . As Figure 2 shows , each client of our MDFNet contains a basic feature encoder parameterized θe mapping the raw input into the hidden space via gi = θe ( xi ) ∈ Rd . Subsequently , two additional parameters m̂s , m̂I ∈ Rd are introduced into the local network and activated to form the mask probabilities by using the sigmoid function σ ( · ) to get ms = σ ( m̂s ) and mI = σ ( m̂I ) . For each feature gi , based on the mask probabilities , we sample the binary domain-specific and invariant masks ( msi , m I i ∈ { 0 , 1 } d ) from the Bernoulli distributions . To this end , we obtain the domain specific and invariant features with the element-wise multiplication⊗ over binary masks and features , i.e. , gsi = msi ⊗gi and gIi = mIi ⊗gi . Moreover , we adopt three strategies to achieve high-quality feature separation . Concretely , each client firstly minimizes the semantic overlap between gIi and g s i to store complementary information in them . Motivated by Rahman & Wang ( 2016 ) , we design the following soft-interactive loss as : Ls = ∑ i 〈gsi , gIi 〉 sum ( gsi + g I i − gsi ⊗ gIi ) , ( 1 ) where 〈 , 〉 means the inner product of two feature vectors , and sum ( · ) represents the sum of all elements for a vector . This approximately reflects the information overlap of two mask distributions . The minimization of soft-interactive loss gradually increases the difference between msi and m I i which activate different neurons . Similar to DSN ( Bousmalis et al . ( 2016 ) ) , each client also develops the individual classifier θc ( · ) taking domain-invariant features as input to the category probability distribution θc ( gIj ) . The cross-entropy loss between ground-truth and prediction intensifies the discriminnative ability of domain-invariant features . On the other hand , we also feed the combination of gsi and g I i into decoder θd ( · ) to reconstruct the original input with Lr = ∑ i ‖θd ( gsi , gIi ) − xi‖22 . Thus , the overall loss function of mask-driven disentanglement for labeled clients is formulated as : min θc , θe , θd , m̂s , m̂I Llo = ∑ i −yi log ( θc ( g I i ) ) + Lr + Ls , ( 2 ) where we actually adopt straight-through estimator ( Bengio et al. , 2013 ) to progressively optimize m̂s and m̂I instead of the discrete binary masks leading to the invalid back-propagation .

Summary: This paper proposes a Mask-Driven Federated Network (MDFNet) to reach a “win-win” deal for multiple domains with data protected to solve the Privacy Protected Multi-Domain Collaborative Learning (P2MDCL) problem. Specifically, each domain is armed with an individual local model via a mask disentangled mechanism to learn domain-invariant semantics. Second, the centralized server refines the global invariant model by integrating and exchanging local knowledge across all domains. Moreover, adaptive self-supervised optimization is deployed to learn discriminative features for unlabeled domains.