Domain Aggregation Networks for Multi-Source Domain Adaptation

1 INTRODUCTION . Many machine learning algorithms assume the learned predictor will be tested on the data from the same distribution as the training data . This assumption , although reasonable , is not necessarily true for many real-world applications . For example , patients from one hospital may have a different distribution of gender , height and weight from another hospital . Consequently , a diagnostic model constructed at one location may not be directly applicable to another location without proper adjustment . The situation becomes even more challenging when we want to use data from multiple source domains to build a model for a target domain , as this requires deciding , e.g. , how to rank the source domains and how to effectively aggregate these domains when training complex models like deep neural networks . Including irrelevant or worse , adversarial , data from certain source domains can severely reduce the performance on the target domain , leading to undesired consequences . To address this problem , researchers have been exploring methods of transfer learning ( Pan & Yang , 2009 ) or domain adaptation ( Mansour et al. , 2009a ; Cortes et al. , 2019 ) , where a model is trained based on labelled source data and unlabelled target data . Most existing works have focused on single-source to single-target ( “ one-to-one ” ) domain adaptation , using different assumptions such as covariate shift ( Shimodaira , 2000 ; Gretton et al. , 2009 ; Sugiyama & Kawanabe , 2012 ) or concept drift ( Jiang & Zhai , 2007 ; Gama et al. , 2014 ) . When dealing with multiple source domains , one may attempt to directly use these approaches by combining all source data into a large joint dataset and then apply one-to-one adaptation . This naı̈ve aggregation method will often fail as not all source domains are equally important when transferring to a specific target domain . There are some works on multi-source to single-target adaptation . Although many of them are theoretically motivated with cross-domain generalization bounds , they either use ad-hoc aggregation rules when developing actual algorithms ( Zhao et al. , 2018 ; Li et al. , 2018 ) or lack finite-sample analysis ( Mansour et al. , 2009b ; c ; Hoffman et al. , 2018a ) . This leaves a gap between the theory for multi-source adaptation and theoretically sound algorithm for domain aggregation . This research has three contributions : First , we extend prior work on one-to-one adaptation using discrepancy ( Cortes et al. , 2019 ) to develop a finite-sample cross-domain generalization bound for multi-source adaptation . We show that in order to improve performance on the specific target domain , there is a trade-off between utilizing all source domains to increase effective sample size , versus removing source domains that are underperforming or not similar to the target domain . Second , motivated by our theory and domain adversarial method ( Ganin & Lempitsky , 2015 ; Ganin et al. , 2016 ) , we propose Domain AggRegation Network ( DARN ) , that can effectively aggregate multiple source domains dynamically during the course of training . Unlike previous works , our aggregation scheme ( Eq . ( 6 ) ) , which itself is of independent interest in some other contexts , is a direct optimization of our generalization upper bound ( Eq . ( 3 ) ) without resorting to heuristics . Third , our experiments on sentiment analysis , digit recognition and object recognition show that DARN can significantly outperform state-of-the-art methods . Section 2 introduces necessary background on one-to-one adaptation based on discrepancy . Then Section 3 elaborates our theoretical analysis and the corresponding algorithm deployment . Section 4 discusses about related approaches in more detail , highlighting the key differences to the approach developed here . Section 5 empirically compares the performance of the proposed method to other alternatives . Finally , Section 6 concludes this work . 2 BACKGROUND ON CROSS-DOMAIN GENERALIZATION . This section provides necessary background from previous work on one-to-one domain adaptation ( Cortes et al. , 2019 ) . Let X be the input space , Y ⊆ R be output space andH ⊆ { h : X 7→ Y } be a hypothesis set . A loss function L : Y × Y 7→ R+ is µ-admissible1 if ∀y , y′ , y′′ ∈ Y |L ( y′ , y ) − L ( y′′ , y ) | ≤ µ |y′ − y′′| . ( 1 ) The discrepancy ( Mansour et al. , 2009a ) between two distributions P , Q over X is defined as disc ( P , Q ) = max h , h′∈H |LP ( h , h′ ) − LQ ( h , h′ ) | where LP ( h , h′ ) = Ex∼P [ L ( h ( x ) , h′ ( x ) ) ] . This quantity can be computed or approximated empirically given samples from both distributions . For classification problems with 0-1 loss , it reduces to the well-known dA-distance ( Kifer et al. , 2004 ; Blitzer et al. , 2008 ; Ben-David et al. , 2010 ) and can be approximated using a domain-classifier loss w.r.t . H ( Zhao et al. , 2018 ; Ben-David et al. , 2007 , Sec.4 ) , while for regression problems with L2 loss , it reduces to a maximum eigenvalue ( Cortes & Mohri , 2011 , Sec.5 ) ; see Section 3.2 for more details . For two domains ( P , fP ) , ( Q , fQ ) where fP , fQ : X 7→ Y are the corresponding labeling functions , we have the following cross-domain generalization bound : Theorem 1 ( Proposition 5 & 8 , Cortes et al . ( 2019 ) ) Let Rm ( H ) be the Rademacher complexity ofH given sample size m , HQ = { x 7→ L ( h ( x ) , fQ ( x ) ) : h ∈ H } be the set of functions mapping x to its loss w.r.t . fQ andH , ηH = µ×min h∈H ( max x∈supp ( P̂ ) |fP ( x ) − h ( x ) |+ max x∈supp ( Q̂ ) |fQ ( x ) − h ( x ) | ) , ( 2 ) be a constant measuring how well H can fit the true models where supp ( P̂ ) means the support of the empirical distribution P̂ ( using the µ from Eq . ( 1 ) ) , and MQ = supx∈X , h∈H L ( h ( x ) , fQ ( x ) ) be the upper bound on loss for Q . Given Q̂ with m points sampled iid from Q labelled according to fQ , for δ ∈ ( 0 , 1 ) , ∀h ∈ H , w.p . at least 1− δ , LP ( h , fP ) ≤ LQ̂ ( h , fQ ) + disc ( P , Q ) + 2Rm ( HQ ) + ηH +MQ √ log ( 1/δ ) 2m . This theorem provides a way to generalize across domains when we have sample Q̂ labelled according to fQ and an unlabelled sample P̂ . The first term is the usual loss function for the sample Q̂ , while the second term disc ( P , Q ) can be estimated based on the unlabelled data Q̂ , P̂ . ηH measures how well the model family H can fit the example from the datasets , and it is not controllable once H is given . The final term , as a function of sample size m , determines the convergence speed . 1For example , the common Lp losses are µ-admissible ( Cortes et al. , 2019 , Lemma 23 ) . 3 DOMAIN AGGREGATION NETWORKS . 3.1 THEORY . Suppose we are given k source domains { ( Si , fSi ) : i ∈ [ k ] def = { 1 , 2 , . . . , k } } and a target domain ( T , fT ) where Si , T are distributions over X and fSi , fT : X 7→ Y are their respective labelling functions . For simplicity , assume that each sample Ŝi has m points , drawn iid from Si and labelled according to fSi . We are also given m unlabelled points T̂ drawn iid from T . We want to leverage all source domains ’ information to learn a model h ∈ H minimizing LT ( h , fT ) . One naı̈ve approach could be to combine all the source domains into a large joint dataset and conduct one-to-one adaptation to the target domain using Theorem 1 . However , including data from irrelevant or even adversarial domains is likely to jeopardize the performance on the target domain , which is sometimes referred to as negative transfer ( Pan & Yang , 2009 ) . Moreover , as certain source domains may be more similar or relevant to the target domain than the others , it makes more sense to adjust their importance according to their utilities . We propose to find domain weight αi ≥ 0 such that ∑k i=1 αi = 1 to achieve this . Our main theorem below sheds some light on how we should combine the source domains ( the proof is provided in Appendix A ) : Theorem 2 Given k source domains datasets { ( x ( i ) j , y ( i ) j ) : i ∈ [ k ] , j ∈ [ m ] } with m iid examples each where Ŝi = { x ( i ) j } and y ( i ) j = fSi ( x ( i ) j ) , for any α ∈ ∆ = { α : αi ≥ 0 , ∑ i αi = 1 } , δ ∈ ( 0 , 1 ) , and ∀h ∈ H , w.p . at least 1− δ LT ( h , fT ) ≤ ∑ i αi ( LŜi ( h , fSi ) + disc ( T , Si ) + 2Rm ( HSi ) + ηH , i ) + ‖α‖2MS √ log ( 1/δ ) 2m , ( 3 ) where HSi = { x 7→ L ( h ( x ) , fSi ( x ) ) : h ∈ H } is the set of functions mapping x to the corresponding loss , ηH , i is a constant similar to Eq . ( 2 ) with Q̂ = Ŝi , P̂ = T̂ and MS = supi∈ [ k ] , x∈X , h∈H L ( h ( x ) , fSi ( x ) ) is the upper bound on loss on the source domains . There are several observations . ( 1 ) In the last term of the bound , m/‖α‖22 serves as the effective sample size . If α is uniform ( i.e. , [ 1/k , . . . , 1/k ] > ) , then the effective sample size is km ; if α is onehot , then we effectively only havem points from exactly one domain . ( 2 ) Rm ( HSi ) determines how expressive the model familyH is w.r.t . the source data Si . It can be estimated from samples ( Bartlett & Mendelson , 2002 , Theorem 11 ) , but the computation is non-trivial for a model family like deep neural networks . The ηH , i is uncontrollable once the hypothesis classH ( e.g. , neural network architecture ) is given . Using a richer H is not always beneficial . Richer H can reduce the ηH , i and also help us find a better function h with smaller source losses LŜi , but it will increase the Rm ( HSi ) . As removing these two terms does not seem to be empirically significant , we ignore them below for simplicity . ( 3 ) Let gh , i def = LŜi ( h , fSi ) + disc ( T , Si ) . Small gh , i indicates that we can achieve small loss on domain Si , and it is similar to the target domain ( i.e. , small disc ( T , Si ) , estimated from T̂ , Ŝi ) . We may want to emphasize on Si by setting αi close to 1 , but this will reduce the effective sample size . Therefore , we have to trade-off between the terms in this bound by choosing a proper α . ( 4 ) When Si and T are only partially overlapped , it might be difficult to find a suitable αi . In such cases , we can artificially split the given source domains into smaller datasets ( e.g. , by using clustering ) then apply our method on the finer scale . This strategy requires that the learning algorithm has low computational complexities w.r.t . k , the number of source domains . As we will see later in Section 3.3 , this is indeed the case for our algorithm . Before we proceed to develop an algorithm based on the theorem , let us compare Eq . ( 3 ) to existing finite-sample bounds . The bound of Blitzer et al . ( 2008 , Theorem 3 ) is informative when we have access to a small set of labelled target examples . In such cases , we can improve our bound by using this small labelled target set as an additional source domain Sk+1 . We can also perform better model selection using such labelled set . The bound of Zhao et al . ( 2018 , Theorem 2 ) is based on the dA-distance , which is a special case of disc in our bound . Moreover , our bound ( Eq . ( 3 ) ) use samplebased Rademacher complexity , which is generally tighter than other complexity measures such as VC-dimension ( Bartlett & Mendelson , 2002 ; Koltchinskii et al. , 2002 ; Bousquet et al. , 2003 ) .

Summary: This paper studies multi-source domain adaptation problem. First this paper proposes a new theory for this domain that extends generalized discrepancy theory to multi-source setting. After derive a new generalization bound, this paper also proposes a new method based on the theory. Evaluation on real world datasets are proposed to show the efficiency of the proposed method.