A Simple Unified Information Regularization Framework for Multi-Source Domain Adaptation

1 INTRODUCTION . Although a large number of studies have demonstrated the ability of deep neural networks to solve challenging tasks , the tasks solved by networks are mostly confined to a similar type or a single domain . One remaining challenge is the problem known as domain shift ( Gretton et al . ( 2009 ) ) , where a direct transfer of information gleaned from a single source domain to unseen target domains may lead to significant performance impairment . Domain adaptation ( DA ) approaches aim to mitigate this problem by learning to map data of both domains onto a common feature space . Whereas several theoretical results ( Ben-David et al . ( 2007 ) ; Blitzer et al . ( 2008 ) ; Zhao et al . ( 2019a ) ) and algorithms for DA ( Long et al . ( 2015 ; 2017 ) ; Ganin et al . ( 2016 ) ) have focused on the case in which only a single-source domain dataset is given , we consider a more challenging and generalized problem of knowledge transfer , referred to as Multi-source unsupervised DA ( MDA ) . Following a seminal theoretical result on MDA ( Blitzer et al . ( 2008 ) ; Ben-David et al . ( 2010 ) ) , technical advances have been made , mainly on the adversarial methods . ( Xu et al . ( 2018 ) ; Zhao et al . ( 2019c ) ) . While most of adversarial MDA methods use multiple independent domain discriminators ( Xu et al . ( 2018 ) ; Zhao et al . ( 2018 ) ; Li et al . ( 2018 ) ; Zhao et al . ( 2019c ; b ) ) , the potential pitfalls of this setting have not been fully explored . The existing works do not provide a theoretical guarantee that the unnecessary domain-specific information is fully filtered out , because the domain-discriminative information is inevitably distributed across multiple discriminators . For example , the multiple domain discriminators focus only on estimating the domain shift between source domains and the target , while the discrepancies between the source domains are neglected , making it hard to align all the given domains . This necessitates garnering the domain-discriminative information with a unified discriminator . Moreover , the multiple domain discriminator setting is not scalable in terms of computational resources especially when large number of source domains are given , e.g. , medical reports from multiple patients . Finally , it may undermine the stability of training , as earlier works solve multiple independent adversarial minimax problems . To overcome such limitations , we propose a novel MDA method , called Multi-source Informationregularized Adaptation Networks ( MIAN ) , that constrains the mutual information between latent representations and domain labels . First , we show that such mutual information regularization is closely related to the explicit optimization of theH-divergence between the source and target domains . This affords the theoretical insight that the conventional adversarial DA can be translated into an information-theoretic-regularization problem . Second , based on our findings , we propose a new optimization problem for MDA : minimizing adversarial loss over multiple domains with a single domain discriminator . We show that the domain shift between each source domain can be indirectly penalized , which is known to be beneficial in MDA ( Li et al . ( 2018 ) ; Peng et al . ( 2019 ) ) , with a single domain discriminator . Moreover , by analyzing existing studies in terms of information regularization , we found that the variance of the stochastic gradients increases when using multiple discriminators . Despite its structural simplicity , we found that MIAN works efficiently across a wide variety of MDA scenarios , including the DIGITS-Five ( Peng et al . ( 2019 ) ) , Office-31 ( Saenko et al . ( 2010 ) ) , and Office-Home datasets ( Venkateswara et al . ( 2017 ) ) . Intriguingly , MIAN reliably and significantly outperformed several state-of-the-art methods that either employ a domain discriminator separately for each source domain ( Xu et al . ( 2018 ) ) or align the moments of deep feature distribution for every pairwise domain ( Peng et al . ( 2019 ) ) . 2 RELATED WORKS . Several DA methods have been used in attempt to learn domain-invariant representations . Along with the increasing use of deep neural networks , contemporary work focuses on matching deep latent representations from the source domain with those from the target domain . Several measures have been introduced to handle domain shift , such as maximum mean discrepancy ( MMD ) ( Long et al . ( 2014 ; 2015 ) ) , correlation distance ( Sun et al . ( 2016 ) ; Sun & Saenko ( 2016 ) ) , and Wasserstein distance ( Courty et al . ( 2017 ) ) . Recently , adversarial DA methods ( Ganin et al . ( 2016 ) ; Tzeng et al . ( 2017 ) ; Hoffman et al . ( 2017 ) ; Saito et al . ( 2018 ; 2017 ) ) have become mainstream approaches owing to the development of generative adversarial networks ( Goodfellow et al . ( 2014 ) ) . However , the abovementioned single-source DA approaches inevitably sacrifice performance for the sake of multi-source DA . Some MDA studies ( Blitzer et al . ( 2008 ) ; Ben-David et al . ( 2010 ) ; Mansour et al . ( 2009 ) ; Hoffman et al . ( 2018 ) ) have provided the theoretical background for algorithm-level solutions . ( Blitzer et al . ( 2008 ) ; Ben-David et al . ( 2010 ) ) explore the extended upper bound of true risk on unlabeled samples from the target domain with respect to a weighted combination of multiple source domains . Following these theoretical studies , MDA studies with shallow models ( Duan et al . ( 2012b ; a ) ; Chattopadhyay et al . ( 2012 ) ) as well as with deep neural networks ( Mancini et al . ( 2018 ) ; Peng et al . ( 2019 ) ; Li et al . ( 2018 ) ) have been proposed . Recently , some adversarial MDA methods have also been proposed . Xu et al . ( 2018 ) implemented a k-way domain discriminator and classifier to battle both domain and category shifts . Zhao et al . ( 2018 ) also used multiple discriminators to optimize the average case generalization bounds . Zhao et al . ( 2019c ) chose relevant source training samples for the DA by minimizing the empirical Wasserstein distance between the source and target domains . Instead of using separate encoders , domain discriminators or classifiers for each source domain as in earlier works , our approach uses unified networks , thereby improving resource-efficiency and scalability . Several existing MDA works have proposed methods to estimate the source domain weights following ( Blitzer et al . ( 2008 ) ; Ben-David et al . ( 2010 ) ) . Mansour et al . ( 2009 ) assumed that the target hypothesis can be approximated by a convex combination of the source hypotheses . ( Peng et al . ( 2019 ) ; Zhao et al . ( 2018 ) ) suggested ad-hoc schemes for domain weights based on the empirical risk of each source domain . Li et al . ( 2018 ) computed a softmax-transformed weight vector using the empirical Wasserstein-like measure instead of the empirical risks . Compared to the proposed methods without robust theoretical justifications , our analysis does not require any assumption or estimation for the domain coefficients . In our framework , the representations are distilled to be independent of the domain , thereby rendering the performance relatively insensitive to explicit weighting strategies . 3 THEORETICAL INSIGHTS . We first introduce the notations for the MDA problem in classification . A set of source domains and the target domain are denoted by { DSi } N i=1 and DT , respectively . Let XSi = { xjSi } m j=1 and YSi = { yjSi } m j=1 be a set of m i.i.d . samples from DSi . Let XT = { xT j } m j=1 ∼ ( DXT ) m be the set of m i.i.d . samples generated from the marginal distribution DXT . The domain label and its probability distribution are denoted by V and PV ( v ) , where v ∈ V and V is the set of domain labels . In line with prior works ( Hoffman et al . ( 2012 ) ; Gong et al . ( 2013 ) ; Mancini et al . ( 2018 ) ; Gong et al . ( 2019 ) ) , domain label can be generally treated as a stochastic latent random variable in our framework . However , for simplicity , we take the empirical version of the true distributions with given samples assuming that the domain labels for all samples are known . The latent representation of the sample is given by Z , and the encoder is defined as F : X → Z , with X and Z representing data space and latent space , respectively . Accordingly , ZSi and ZT refer to the outputs of the encoder F ( XSi ) and F ( XT ) , respectively . For notational simplicity , we will omit the index i from DSi , XSi and ZSi when N = 1 . A classifier is defined as C : Z → Y where Y is the class label space . 3.1 PROBLEM FORMULATION . For comparison with our formulation , we recast single-source DA as a constrained optimization problem . The true risk T ( h ) on unlabeled samples from the target domain is bounded above the sum of three terms ( Ben-David et al . ( 2010 ) ) : ( 1 ) true risk S ( h ) of hypothesis h on the source domain ; ( 2 ) H-divergence dH ( DS , DT ) between a source and a target domain distribution ; and ( 3 ) the optimal joint risk λ∗ . Theorem 1 ( Ben-David et al . ( 2010 ) ) . Let hypothesis classH be a set of binary classifiers h : X → { 0 , 1 } . Then for the given domain distributions DS and DT , ∀h ∈ H , T ( h ) ≤ S ( h ) + dH ( DS , DT ) + λ∗ , ( 1 ) where dH ( DS , DT ) = 2sup h∈H ∣∣∣ E x∼DXS [ I ( h ( x ) = 1 ) ] − E x∼DXT [ I ( h ( x ) = 1 ) ] ∣∣∣ and I ( a ) is an indicator function whose value is 1 if a is true , and 0 otherwise . The empiricalH-divergence d̂H ( XS , XT ) can be computed as follows ( Ben-David et al . ( 2010 ) ) : Lemma 1. d̂H ( XS , XT ) = 2 ( 1−min h∈H [ 1 m ∑ x∈XS I [ h ( x ) = 1 ] + 1 m ∑ x∈XT I [ h ( x ) = 0 ] ] ) ( 2 ) Following Lemma 1 , a domain classifier h : Z → V can be used to compute the empirical Hdivergence . Suppose the optimal joint risk λ∗ is sufficiently small as assumed in most adversarial DA studies ( Saito et al . ( 2017 ) ; Chen et al . ( 2019 ) ) . Thus , one can obtain the ideal encoder and classifier minimizing the upper bound of T ( h ) by solving the following min-max problem : F ∗ , C∗ = argmin F , C L ( F , C ) + βd̂H ( ZS , ZT ) = argmin F , C max h∈H L ( F , C ) + β 1 m ( ∑ i : zi∈ZS I [ h ( zi ) = 1 ] + ∑ j : zj∈ZT I [ h ( zj ) = 0 ] ) , ( 3 ) where L ( F , C ) is the loss function on samples from the source domain , β is a Lagrangian multiplier , V = { 0 , 1 } such that each source instance and target instance are labeled as 1 and 0 , respectively , and h is the binary domain classifier . Note that the latter min–max problem is obtained by converting −min into max and removing the constant term from Lemma 1 .

Summary: This paper studies the multi-source domain adaptation problem. The authors examine the existing MDA solutions, i.e. using a domain discriminator for each source-target pair, and argue that the existing ones are likely to distribute the domain-discriminative information across multiple discriminators. By theoretically analyzing from the information regularization point, the authors present a simple yet powerful architecture called multi-source information-regularized adaptation network, MIAN.