Domain-Invariant Representations: A Look on Compression and Weights

1 INTRODUCTION . In a standard Supervised Learning setup , we consider that training data is representative of testing data . This ideal framework is often violated in practical applications since the data generative process can be altered during the application phase e.g . different pre-processing , sample rejection , imbalanced classes , condition of data collection , time evolving data ... This so-called Distributional Shift situation ( Quionero-Candela et al . ( 2009 ) ; Kull & Flach ( 2014 ) ) is recognized as a major challenge in Machine Learning ( Amodei et al . ( 2016 ) ) . Domain Adaptation ( DA ) ( Pan & Yang ( 2009 ) ) is a widely adopted strategy to mitigate its effects . It consists in leveraging unlabeled testing data ( the target domain ) in order to transfer knowledge learned on labeled training data ( the source domain ) . A first line of study , named Importance Sampling ( IS ) , assumes some stationarity in distributions in order to approximate the risk in the target domain by weighting sample contribution ( Shimodaira ( 2000 ) ; Huang et al . ( 2007 ) ) . To be applied , IS needs statistical support sharing across domains limiting its range of application when addressing high dimensional data . In that challenging context , Ganin & Lempitsky ( 2015 ) have shown that deep classifiers can be adapted to a new target domain by learning domain Invariant Representations ( IR ) and find theoretical support in the work of Ben-David et al . ( 2007 ; 2010 ) . It assumes that a classifier trained on representations of labeled data sampled from both domains will achieve a negligible combined error ( adaptability ) . However , enforcing distribution invariance in the representation space can badly impact its adaptability , for instance in the context of Target Shift ( Zhao et al . ( 2019 ) ) or when the support of the distributions do not overlap enough ( Johansson et al . ( 2019 ) ) . In the most general case of DA , it is not clear how IR helps aligning domains and removing spurious correlations as pointed by Arjovsky et al . ( 2019 ) . In the present work , we show that the search of invariance favors learning representations with a significant loss of information from the original features . We refer to this phenomenon as the risk of Compression . In our understanding , this risk is not addressed in the original work from Ben-David et al . ( 2007 ; 2010 ) . We demonstrate that taking into account this risk has two major advantages . First , it provides a better control on ( non-trainable ) terms involving the labels in the target domain . Second , it introduces a new trainable term which embodies a risk of Compression and then may open-up the path to new learning methods . We provide a theoretical analysis illustrated with experiments on a standard Domain Adaptation benchmark . We underline that a better control on non-trainable terms may lead to an over-pessimistic risk of compression . This brings to light an inherent trade-off when learning representations for domain adaptation : How much compression risk is acceptable in order to obtain invariance ? As a first try to answer to this question , we point that Learning Weighted Representations plays a key role in aligning representation distributions in order to obtain invariance . More precisely , we show that under a constraint of strong source risk conservation , weighting representations does not impact its adaptability . 2 BACKGROUND . 2.1 A DECADE OF DOMAIN ADAPTATION . The classical setup for modelling distributional shift is to introduce two domains : the source domain i.e . dataset where the model is trained with supervision and the target domain i.e . where the model is tested or applied . Formally , for two random variables ( X , Y ) of a given space X × Y , we introduce two distributions on X × Y : the source distribution Ps ( X , Y ) and the target distribution Pt ( X , Y ) . We use the exponent notation s and t to differentiate source and target terms . We define the hypothesis space H as a subset of functions from X to Y . The distributional shift situation is then characterized by the shift : Ps ( X , Y ) 6= Pt ( X , Y ) . Learning under Distributional Shift ( Quionero-Candela et al . ( 2009 ) ) consists in minimizing the risk on the target domain εt ( h ) = Et [ ` ( h ( X ) , Y ) ] for a given loss ` : Y × Y → R+ and an hypothesis function h ∈ H. Assuming that w ( X , Y ) = Pt ( X , Y ) /Ps ( X , Y ) is tractable , it is straightforward to show that : εt ( h ) = Es [ w ( X , Y ) ` ( h ( X ) , Y ) ] ( 1 ) Roughly speaking , this equation shows that it is possible to minimize the target risk using data ( xsi , y s i ) 1≤i≤ns sampled from the source domain ( i.e . ( xi , yi ) ∼ Ps ( X , Y ) ) while re-weighting their contribution in the source expectation with a factor w ( xi , yi ) ( Importance Sampling ) . The difficulty of Learning under distributional shift arises when w ( xi , yi ) is not tractable . The first reason is that data sampled from Pt ( X , Y ) is often not easily available . In order to get a reasonable estimation of w , practitioners are used to reformulate the problem as Unsupervised Domain Adaptation . Definition 1 ( Unsupervised Domain Adaptation ) . Given a loss ` , an hypothesis class H , a source domain ( X ×Y , Ps ( X , Y ) ) and a target domain ( X ×Y , Pt ( X , Y ) ) , Unsupervised Domain Adaptation consists in minimizing the target risk ( i.e . finding h ? = arg minh∈H Et [ ` ( h ( X ) , Y ) ] ) with finite labeled sampling of ( xsi , ysi ) ∼ Ps ( X , Y ) of the source domain and a finite unlabeled sampling ( xtj ) ∼ Pt ( X ) of the target domain . If unlabeled samples in the target domain are available , a first idea is then to approximate w ( X , Y ) by w ( X ) = Pt ( X ) /Ps ( X ) . This is equivalent to assuming that conditional distributions are conserved across domains ( i.e . Ps ( Y |X ) = Pt ( Y |X ) ) , a situation known as Covariate Shift , that has already been the subject of an extensive literature ( Sugiyama et al . ( 2008 ) ; Huang et al . ( 2007 ) ) . This formulation becomes ill-posed when feature domains ( X , Ps ( X ) ) and ( X , Pt ( X ) ) suffer from non-overlapping support : w ( X ) is then undefined . To address this issue , Invariant Representations learning ( Long et al . ( 2015 ) ) performs adaptation by looking for a representation Z = ϕ ( X ) whose distribution is conserved across domains Ps ( Z ) = Pt ( Z ) . A common strategy consists in extracting features Z to learn labels Y in the source domain while controlling representation invariance . That is formulated as follows : Learning Objective 1 ( Learning Invariant Representations for Domain Adaptation ) . g ? , ϕ ? = arg min ( g , ϕ ) ∈G×Φ ( 1− α ) · εs ( g ◦ ϕ ) + α · D ( s , t ) ( ϕ ) ( 2 ) where G is the set of classifiers and Φ is the set of representations . The hypothesis space is then defined as H = { g ◦ ϕ : g ∈ G , ϕ ∈ Φ } . D ( s , t ) ( ϕ ) is a distance between distributions Ps ( Z ) and Pt ( Z ) and α is a trade-off parameter . The choice of D ( s , t ) ( ϕ ) has been the object of an extensive literature including f− divergence measures such as domain adversarial learning ( Ganin & Lempitsky ( 2015 ) ; Tzeng et al . ( 2014 ) ; Long et al . ( 2018 ) ) , Integral Probability Measures such as Maximum Mean Discrepancy ( Long et al . ( 2015 ; 2016 ) ) or Optimal Transport ( Shen et al . ( 2018 ) ) . See Kouw & Loog ( 2019 ) for an extensive review . Interestingly , learning invariant representations to perform adaptation finds theoretical support from the work of Ben-David et al . ( 2007 ; 2010 ) that we recall in the next section . 2.2 THEORETICAL GUARANTEES OF DOMAIN ADAPTATION . We consider the set of representations Φ and classifiers G. The hypothesis space is denoted H = { g ◦ ϕ : g ∈ G , ϕ ∈ Φ } . We introduce the loss ` : Y × Y → R+ which is assumed symmetric and verifies the triangular inequality . We note the risk ε ( h ) = E [ ` ( Y , h ( X ) ) ] for h ∈ H and ε ( h , h′ ) = E [ ` ( h ( X ) , h′ ( X ) ) ] for ( h , h′ ) ∈ H2 . To emphasize the role of a representation Z = ϕ ( X ) , we introduce f̃ ( Z ) = E [ Y |Z ] and H̃ = { g ◦ϕ : g ∈ G } ⊂ H. We underline that ·̃ quantities depend on ϕ where , for the ease of reading , this dependence is omitted in notations when it is not ambiguous . Ben-David et al . ( 2007 ; 2010 ) have derived a theoretical bound of the additional risk when using a representation Z = ϕ ( X ) rather than raw features X : Inequality 1 ( Ben-David et al . ( 2007 ; 2010 ) ) . For a given ϕ ∈ Φ and g ∈ G , there is : εt ( g ◦ ϕ ) ︸ ︷︷ ︸ Target risk ≤ εs ( g ◦ ϕ ) ︸ ︷︷ ︸ Source risk + sup ( h , h′ ) ∈H̃ ∣∣εs ( h , h′ ) − εt ( h , h′ ) ∣∣︸ ︷︷ ︸ = 12 ·d ( H̃∆H̃ ) : Disagreement + inf h∈H̃ εt ( h ) + εs ( h ) ︸ ︷︷ ︸ =λ ( H̃ ) : Adaptability ( 3 ) The inequality 1 ensures that the target risk is bounded by the sum of the source risk , the disagreement risk between two classifiers from representations ( d ( H̃∆H̃ ) named H̃∆H̃ distance ) , and a third term ( λ ( H̃ ) ) which quantifies the ability to perform well in both domains from representations . In the rest of the paper , we refer to the latter as the adaptability . Since Et [ Y |X ] is unknown , λ ( H̃ ) is intractable in practical applications . By conveniently assuming that λ ( H̃ ) is negligible , it is possible to derive a Learning Objective 1 which minimizes ε̃s ( g◦ϕ ) + 12d ( H̃∆H̃ ) . Indeed , minimizing D ( s , t ) on ϕ leads to learn representations Z = ϕ ( X ) such that Ps ( Z ) = Pt ( Z ) , where such situation is sufficient to ensure that d ( H̃∆H̃ ) = 0 .

Summary: This paper presents a revisit of existing theoretical frameworks in unsupervised domain adaptation in the context of learning invariant representation. They propose a novel bound that involves trainable terms taking into account some compression information and a novel interpretation of adaptability. The authors mention also contribution showing that weighting representations can be a way to improve the analysis.