Multi-Step Decentralized Domain Adaptation

MULTI-STEP DECENTRALIZED DOMAIN ADAPTATION . Anonymous authors Paper under double-blind review ABSTRACT Despite the recent breakthroughs in unsupervised domain adaptation ( uDA ) , no prior work has studied the challenges of applying these methods in practical machine learning scenarios . In this paper , we highlight two significant bottlenecks for uDA , namely excessive centralization and poor support for distributed domain datasets . Our proposed framework , MDDA , is powered by a novel collaborator selection algorithm and an effective distributed adversarial training method , and allows for uDA methods to work in a decentralized and privacy-preserving way . 1 Introduction . In practical machine learning systems , test samples are often drawn from a different data distribution than the training samples , due to variations in data acquisition processes between the training and test sets – caused by for example , different illumination conditions and cameras in the context of visual tasks . This shift in data distributions , known as domain shift , is a core reason which hinders the generalizability of predictive models to new domains . As manual labeling of data in each test domain is prohibitively expensive , unsupervised domain adaptation ( uDA ) has emerged as a promising solution to transfer the knowledge from a labeled source domain to unlabeled target domains ( Long et al . ( 2015 ; 2017 ) ; Ganin et al . ( 2016 ) ; Tzeng et al . ( 2017 ) ; Hoffman et al . ( 2018 ) ; Shen et al . ( 2018 ) ; Sankaranarayanan et al . ( 2018 ) ; Hoffman et al . ( 2018 ) ) . While uDA methods are indeed effective , little attention has been paid on how they would be incorporated in real-world machine learning systems . In this paper , we study and propose solutions for practical problems that arise while applying uDA techniques in ML systems , namely the challenges of distributed domain datasets and the overly centralized nature of existing uDA approaches . As a motivating example , consider a scenario wherein a model is trained for the task of fetal head detection from labeled ultrasound images collected in a hospital in Finland ( source domain Sfin ) . Subsequently , this pre-trained model has to be deployed in three target hospitals - one in the US ( Tus1 ) and two in China ( Tcn1 and Tcn2 ) . Due to variations in sonogram machines and medical training of sonographers , a domain shift is likely to occur in the test samples ; hence we need to apply uDA to adapt the source classifier in the target domains . Existing uDA methods are centralized by design , in that they assume that each target domain would always adapt from the labeled source domain ( Sfin ) . This raises two issues : firstly , if the machine hosting the labeled source dataset is unavailable ( e.g. , it is undergoing maintenance or has connectivity issues ) , then clearly adaptation is not possible . More importantly , we argue that this choice of always adapting from a labeled source is not optimal from an adaptation perspective , because the domain discrepancy between the labeled source and a potential target domain could be high in some cases . Our work seeks to explore an interesting proposition : in addition to adapting from the labeled source , can we also perform uDA with other target domains , which themselves may have undergone domain adaptation in the past . Further , existing uDA methods do not support distributed domain datasets and assume that source and target data are available on the same machine . Clearly , this raises privacy and legal concerns since either the source domain or the target domains need to send their sensitive data ( i.e. , sonograms ) to each other to perform uDA . Moreover , such transfer of potentially large datasets also incurs severe communication costs . In summary , this paper makes the following contributions : • We formulate and study a brand-new problem focusing on the challenges of uDA in practical machine learning systems . • We propose a multi-step uDA framework , wherein target domains can adapt not only from the labeled source , but also from other target domains . Powering this framework is a novel collaborator selector algorithm that chooses the optimal adaptation collaborator for each target domain , before initiating the adaptation . • We propose an effective technique for allowing uDA algorithms and adversarial training to work across distributed datasets . Through extensive experiments on five image and speech datasets , we demonstrate the efficacy of our proposed solution . 2 Problem Formulation and Related Work . Consider a practical scenario of deploying a uDA method in a real-world ML system . Assume that a data collection exercise yields a labeled dataset upon which a model is trained using supervised learning . Thereafter , this model needs to be deployed to a population of users ( or targets ) whose data is unlabeled and divergent from the training dataset . Mathematically , we are presented with a single source domain S = ( Rn , pS ( x , y ) ) , with input data XS and labels YS . There are multiple target domains { T j ∣∣j = 1 , . . . , K } , with data XjT drawn from target distribution pjT ( x , y ) , without labeled observations . Using supervised learning , we train a representation mapping , MS , and a classifier , CS for the source domain . However , for the target domains , due to the absence of labeled observations , supervised learning is not possible and hence we would like to do adaptation with the source domain . We assume that target domains are introduced sequentially , one at a time . Under this problem formulation , we highlight two unexplored research challenges : Collaborator Selection . For each target domain that joins the system , how do we select an optimal collaborator for domain adaptation that will lead to highest post-adaptation accuracy for the target domain ? Existing uDA methods ( e.g. , Hoffman et al . ( 2018 ) ; Tzeng et al . ( 2017 ) ; Ganin et al . ( 2016 ) always use the labeled source domain S as the adaptation collaborator for each target domain , however we argue that it is not optimal to always adapt for the labeled source domain for two reasons : i ) if the domain shift between the target domain and the labeled source is high , we may not achieve a good adaptation performance ( Wulfmeier et al . ( 2017 ) ) ; ( ii ) from a practical perspective , it makes the entire system centralized and prone to failure if the device hosting the source dataset is unavailable for adaptation ( e.g. , due to connectivity issues ) . Instead , we propose a multi-step decentralized domain adaptation approach built upon the idea that a new target domain can adapt not only from the labeled source domain , but also from other target domains in the system which have already undergone domain adaptation . More concretely , we define a collaborator set C as the set of domains that are available to collaborate with a target domain on an adaptation task . At step τ = τ0 , only the source domain is present in the system , hence C0 = { S } . At step τ = τ1 , the first target domain T 1 joins the system – at this moment , only the source domain S has a learned representationMS . Thereafter , T 1 performs uDA with the source S and learns a representation MT 1 . Subsequently , we have C1 = { S , T 1 } , i.e. , future target domains now have two candidates with whom they can collaborate to perform domain adaptation . In general , at step τ = τK , CK = { S } ∪ { T j ∣∣k = 1 , . . . , K } . In §3.3 , we propose an algorithm to select an optimal collaborator c ∈ CK for each target domain , that is likely to yield the best accuracy post-adaptation . Once the optimal collaborator is chosen , any existing uDA method could be employed to perform the pairwise adaptation between the chosen collaborator and the target domain . In §4 , we show the effectiveness of this decentralized and multistep adaptation approach against conventional baselines and across a number of uDA techniques . Distributed and Private Data . Existing uDA methods assume that the datasets from the source domain S and a given target domain T j are available on the same compute unit ( e.g. , on a server ) . However , this assumption is violated in practical scenarios , as the datasets are often private , and users or companies may not be allowed to share them with other parties due to legal reasons such as the data privacy law ( GDPR ) in Europe . In addition to the privacy issue , exchanging potentially large datasets incur high communication costs , making it undesirable in practical settings . This raises the question : can we make domain adaptation methods to work in a distributed and privacy-preserving manner such that the collaborators in the domain adaptation process can keep their data private and still receive the benefits of adaptation ? These challenges about making uDA algorithms work in a decentralized and distributed way are closely related , and addressing them is critical for practical usage of uDA methods . We first describe ( in §3.2 ) our idea of supporting uDA with distributed domain datasets , by enabling knowledge exchange between adaptation collaborators using the gradients of Discriminator , which allows for the raw data and features of each domain to remain private . Next , in §3.3 , we propose an algorithm for selecting an optimal collaborator for each target domain . 2.1 Related Work . Bobu et al . ( 2018 ) presented the problem of continuous unsupervised adaptation wherein the target domain is smoothly varying temporally . They proposed an iterative uDA method with a replay loss to prevent the model from forgetting knowledge from past domains . Although our solution of multi-step domain adaptation is also iterative , we do not assume any smooth ordering between target domains , hence warranting the need for collaboration selection . Zhao et al . ( 2018 ) proposed MDAN where a target domain can adapt from multiple labeled source domains . As there is only one labeled source domain in our setup , this method can not be directly applied . However , we can still combine the adversarial losses from multiple domains as proposed in MDAN - as such , we implement a variant of MDAN as a baseline for our method . In our paper , the training data from different domains is distributed in separate nodes , which relates our problem to distributed training . The motivation of most existing works in this field is to accelerate the training process by partitioning the training data into multiple computational nodes and calculating the gradients in parallel . If the results from all nodes are aggregated globally by a central node ( e.g. , parameter server ( Li et al . ( 2014 ) ; Dai et al . ( 2015 ) ) ) , or by message passing process ( e.g. , All-Reduce ( Sergeev & Del Balso ( 2018 ) ) ) , then it is called centralized distributed training . By contrast , if the system does not require to synchronize globally ( Lian et al . ( 2017a ) ; Tang et al . ( 2018b ) ) , it is called decentralized training . Besides , decentralized training can be further accelerated by its conjunction with low-precision training ( Tang et al . ( 2018a ) ) , or asynchronous training ( Lian et al . ( 2017b ) ) . In addition to acceleration , federated learning ( Bonawitz et al . ( 2019 ) ; Konečnỳ et al . ( 2016 ) ; Yang et al . ( 2019 ) ; Liu et al . ( 2018 ) ) considers a training setting where the training data is stored over a large number of separate devices and can not be shared due to privacy reasons . Our work builds upon the distributed training literature and we show , for the first time , how distributed training can be used in the context of uDA . Further , it is important to clarify that we use ‘ decentralization ’ in this paper in the context of domain adaptation , i.e. , target domains can choose to adapt from any one of the available domains , instead of a single source domain ( akin to a central node ) . This makes our techniques and their subsequent evaluation different from traditional decentralized learning papers . Finally , an important goal of our paper is to support uDA while keeping the domain datasets private . There has been active research recently at the intersection of privacy and ML . Melis et al . ( 2019 ) showed that collaborative learning is vulnerable to membership inference attacks and leakage of dataset properties , and suggested employing dimensionality reduction and Dropout as ways to mitigate these attacks . Aono et al . ( 2017 ) demonstrated that the gradients of a classifier exchanged in distributed training can be used to reconstruct the raw training data and proposed using additively homomorphic encryption to fix the problem . Geyer et al . ( 2017 ) proposed a differential privacy approach to hide a single client ’ s data contribution during federated learning . Finally , Nasr et al . ( 2018 ) did an in-depth analysis of passive and active white-box attacks on deep learning models . To the best of our knowledge , no prior work has shown that sharing gradients of a discriminator in adversarial learning – which is the key component of our approach – can leak raw training data . As such , our approach provides clear privacy benefits over conventional uDA techniques . That said , we do not discount the possibility that privacy attacks could be developed on our technique in the future , and leave it open as a topic of further research .

Summary: This paper tackles unsupervised domain adaptation in a decentralized setting.  The high-level observation is that the conventional unsupervised domain adaptation has two bottlenecks, namely excessive centralization and poor support for distributed domain datasets. The paper proposes Multi-Step Decentralized Domain Adaptation (MDDA) to transfer the knowledge learned from the source domain to the target domain without sharing the data.