Domain-Invariant Representations: A Look on Compression and Weights

1 INTRODUCTION . In a standard Supervised Learning setup , we consider that training data is representative of testing data . This ideal framework is often violated in practical applications since the data generative process can be altered during the application phase e.g . different pre-processing , sample rejection , imbalanced classes , condition of data collection , time evolving data ... This so-called Distributional Shift situation ( Quionero-Candela et al . ( 2009 ) ; Kull & Flach ( 2014 ) ) is recognized as a major challenge in Machine Learning ( Amodei et al . ( 2016 ) ) . Domain Adaptation ( DA ) ( Pan & Yang ( 2009 ) ) is a widely adopted strategy to mitigate its effects . It consists in leveraging unlabeled testing data ( the target domain ) in order to transfer knowledge learned on labeled training data ( the source domain ) . A first line of study , named Importance Sampling ( IS ) , assumes some stationarity in distributions in order to approximate the risk in the target domain by weighting sample contribution ( Shimodaira ( 2000 ) ; Huang et al . ( 2007 ) ) . To be applied , IS needs statistical support sharing across domains limiting its range of application when addressing high dimensional data . In that challenging context , Ganin & Lempitsky ( 2015 ) have shown that deep classifiers can be adapted to a new target domain by learning domain Invariant Representations ( IR ) and find theoretical support in the work of Ben-David et al . ( 2007 ; 2010 ) . It assumes that a classifier trained on representations of labeled data sampled from both domains will achieve a negligible combined error ( adaptability ) . However , enforcing distribution invariance in the representation space can badly impact its adaptability , for instance in the context of Target Shift ( Zhao et al . ( 2019 ) ) or when the support of the distributions do not overlap enough ( Johansson et al . ( 2019 ) ) . In the most general case of DA , it is not clear how IR helps aligning domains and removing spurious correlations as pointed by Arjovsky et al . ( 2019 ) . In the present work , we show that the search of invariance favors learning representations with a significant loss of information from the original features . We refer to this phenomenon as the risk of Compression . In our understanding , this risk is not addressed in the original work from Ben-David et al . ( 2007 ; 2010 ) . We demonstrate that taking into account this risk has two major advantages . First , it provides a better control on ( non-trainable ) terms involving the labels in the target domain . Second , it introduces a new trainable term which embodies a risk of Compression and then may open-up the path to new learning methods . We provide a theoretical analysis illustrated with experiments on a standard Domain Adaptation benchmark . We underline that a better control on non-trainable terms may lead to an over-pessimistic risk of compression . This brings to light an inherent trade-off when learning representations for domain adaptation : How much compression risk is acceptable in order to obtain invariance ? As a first try to answer to this question , we point that Learning Weighted Representations plays a key role in aligning representation distributions in order to obtain invariance . More precisely , we show that under a constraint of strong source risk conservation , weighting representations does not impact its adaptability . 2 BACKGROUND . 2.1 A DECADE OF DOMAIN ADAPTATION . The classical setup for modelling distributional shift is to introduce two domains : the source domain i.e . dataset where the model is trained with supervision and the target domain i.e . where the model is tested or applied . Formally , for two random variables ( X , Y ) of a given space X × Y , we introduce two distributions on X × Y : the source distribution Ps ( X , Y ) and the target distribution Pt ( X , Y ) . We use the exponent notation s and t to differentiate source and target terms . We define the hypothesis space H as a subset of functions from X to Y . The distributional shift situation is then characterized by the shift : Ps ( X , Y ) 6= Pt ( X , Y ) . Learning under Distributional Shift ( Quionero-Candela et al . ( 2009 ) ) consists in minimizing the risk on the target domain εt ( h ) = Et [ ` ( h ( X ) , Y ) ] for a given loss ` : Y × Y → R+ and an hypothesis function h ∈ H. Assuming that w ( X , Y ) = Pt ( X , Y ) /Ps ( X , Y ) is tractable , it is straightforward to show that : εt ( h ) = Es [ w ( X , Y ) ` ( h ( X ) , Y ) ] ( 1 ) Roughly speaking , this equation shows that it is possible to minimize the target risk using data ( xsi , y s i ) 1≤i≤ns sampled from the source domain ( i.e . ( xi , yi ) ∼ Ps ( X , Y ) ) while re-weighting their contribution in the source expectation with a factor w ( xi , yi ) ( Importance Sampling ) . The difficulty of Learning under distributional shift arises when w ( xi , yi ) is not tractable . The first reason is that data sampled from Pt ( X , Y ) is often not easily available . In order to get a reasonable estimation of w , practitioners are used to reformulate the problem as Unsupervised Domain Adaptation . Definition 1 ( Unsupervised Domain Adaptation ) . Given a loss ` , an hypothesis class H , a source domain ( X ×Y , Ps ( X , Y ) ) and a target domain ( X ×Y , Pt ( X , Y ) ) , Unsupervised Domain Adaptation consists in minimizing the target risk ( i.e . finding h ? = arg minh∈H Et [ ` ( h ( X ) , Y ) ] ) with finite labeled sampling of ( xsi , ysi ) ∼ Ps ( X , Y ) of the source domain and a finite unlabeled sampling ( xtj ) ∼ Pt ( X ) of the target domain . If unlabeled samples in the target domain are available , a first idea is then to approximate w ( X , Y ) by w ( X ) = Pt ( X ) /Ps ( X ) . This is equivalent to assuming that conditional distributions are conserved across domains ( i.e . Ps ( Y |X ) = Pt ( Y |X ) ) , a situation known as Covariate Shift , that has already been the subject of an extensive literature ( Sugiyama et al . ( 2008 ) ; Huang et al . ( 2007 ) ) . This formulation becomes ill-posed when feature domains ( X , Ps ( X ) ) and ( X , Pt ( X ) ) suffer from non-overlapping support : w ( X ) is then undefined . To address this issue , Invariant Representations learning ( Long et al . ( 2015 ) ) performs adaptation by looking for a representation Z = ϕ ( X ) whose distribution is conserved across domains Ps ( Z ) = Pt ( Z ) . A common strategy consists in extracting features Z to learn labels Y in the source domain while controlling representation invariance . That is formulated as follows : Learning Objective 1 ( Learning Invariant Representations for Domain Adaptation ) . g ? , ϕ ? = arg min ( g , ϕ ) ∈G×Φ ( 1− α ) · εs ( g ◦ ϕ ) + α · D ( s , t ) ( ϕ ) ( 2 ) where G is the set of classifiers and Φ is the set of representations . The hypothesis space is then defined as H = { g ◦ ϕ : g ∈ G , ϕ ∈ Φ } . D ( s , t ) ( ϕ ) is a distance between distributions Ps ( Z ) and Pt ( Z ) and α is a trade-off parameter . The choice of D ( s , t ) ( ϕ ) has been the object of an extensive literature including f− divergence measures such as domain adversarial learning ( Ganin & Lempitsky ( 2015 ) ; Tzeng et al . ( 2014 ) ; Long et al . ( 2018 ) ) , Integral Probability Measures such as Maximum Mean Discrepancy ( Long et al . ( 2015 ; 2016 ) ) or Optimal Transport ( Shen et al . ( 2018 ) ) . See Kouw & Loog ( 2019 ) for an extensive review . Interestingly , learning invariant representations to perform adaptation finds theoretical support from the work of Ben-David et al . ( 2007 ; 2010 ) that we recall in the next section . 2.2 THEORETICAL GUARANTEES OF DOMAIN ADAPTATION . We consider the set of representations Φ and classifiers G. The hypothesis space is denoted H = { g ◦ ϕ : g ∈ G , ϕ ∈ Φ } . We introduce the loss ` : Y × Y → R+ which is assumed symmetric and verifies the triangular inequality . We note the risk ε ( h ) = E [ ` ( Y , h ( X ) ) ] for h ∈ H and ε ( h , h′ ) = E [ ` ( h ( X ) , h′ ( X ) ) ] for ( h , h′ ) ∈ H2 . To emphasize the role of a representation Z = ϕ ( X ) , we introduce f̃ ( Z ) = E [ Y |Z ] and H̃ = { g ◦ϕ : g ∈ G } ⊂ H. We underline that ·̃ quantities depend on ϕ where , for the ease of reading , this dependence is omitted in notations when it is not ambiguous . Ben-David et al . ( 2007 ; 2010 ) have derived a theoretical bound of the additional risk when using a representation Z = ϕ ( X ) rather than raw features X : Inequality 1 ( Ben-David et al . ( 2007 ; 2010 ) ) . For a given ϕ ∈ Φ and g ∈ G , there is : εt ( g ◦ ϕ ) ︸ ︷︷ ︸ Target risk ≤ εs ( g ◦ ϕ ) ︸ ︷︷ ︸ Source risk + sup ( h , h′ ) ∈H̃ ∣∣εs ( h , h′ ) − εt ( h , h′ ) ∣∣︸ ︷︷ ︸ = 12 ·d ( H̃∆H̃ ) : Disagreement + inf h∈H̃ εt ( h ) + εs ( h ) ︸ ︷︷ ︸ =λ ( H̃ ) : Adaptability ( 3 ) The inequality 1 ensures that the target risk is bounded by the sum of the source risk , the disagreement risk between two classifiers from representations ( d ( H̃∆H̃ ) named H̃∆H̃ distance ) , and a third term ( λ ( H̃ ) ) which quantifies the ability to perform well in both domains from representations . In the rest of the paper , we refer to the latter as the adaptability . Since Et [ Y |X ] is unknown , λ ( H̃ ) is intractable in practical applications . By conveniently assuming that λ ( H̃ ) is negligible , it is possible to derive a Learning Objective 1 which minimizes ε̃s ( g◦ϕ ) + 12d ( H̃∆H̃ ) . Indeed , minimizing D ( s , t ) on ϕ leads to learn representations Z = ϕ ( X ) such that Ps ( Z ) = Pt ( Z ) , where such situation is sufficient to ensure that d ( H̃∆H̃ ) = 0 .

Summary: This paper introduces the compression risk in domain-invariant representations. Learning domain-invariant representations leads to larger compression risks and potentially worse adaptability. To this end, the authors presents gamma(H) to measure the compression risk. Learning weighted representations to control source error, domain discrepancy, and compression simultaneously leads to a better tradeoff between invariance and compression, which is verified by experimental results.