The balancing principle for parameter choice in distance-regularized domain adaptation

1 Introduction . Domain adaptation uses the knowledge in a source domain to improve the performance of an algorithm on a related target domain [ 1 ] . In particular , domain adaptation tackles domain shifts in machine learning applications : Medical diagnostic systems should be adapted to new physical human variations ; Industrial quality inspection systems should be accurate for new products ; Self-driving cars should be able to adapt to new geographical environments and weather conditions . In this work , we focus on unsupervised domain adaptation where labels are only available in the source domain . There are mainly two types of approaches for unsupervised domain adaptation : importance weighting [ 2 , 3 , 4 , 5 , 6 , 7 ] and feature representation learning [ 8 , 9 , 10 , 11 , 12 , 13 ] . In this work , we focus on feature representation learning which goes beyond classical importance weighting by allowing a target distribution with support outside of the source distribution . The core idea behind feature representation learning approaches is to map the data into a new feature space where the source and target 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . data representations appear similar , and where enough information is preserved for prediction [ 14 ] . The similarity is often realized by regularization using distance measures between source and target representations [ 15 , 16 , 17 , 18 , 19 , 20 , 21 ] . However , the performance of such methods crucially depends on the choice of the regularization parameter which penalizes the distance . The problem we investigate in this work is to choose this parameter , which is sophisticated without any target labels . While remarkable theoretical results have been achieved which quantify the generalization ability of domain adaptation models [ 8 , 22 , 19 , 20 , 21 ] , the choice of the regularization parameter which is crucial for finding such models has not systematically been addressed . Even though some parameter choice strategies exist , they are either purely heuristically driven or very limited by their assumptions [ 23 ] . Typical approaches are fixing the regularization parameters [ 12 ] , minimizing the source error [ 16 ] , balancing the source error and a distance [ 17 ] , multiplying a fixed weighting parameter ( e.g . 1 in [ 16 ] ) by a heuristic schedule value that increases during training , or , ( importance ) weighting the input samples by the ratio between target and source density [ 3 , 24 , 23 ] . One common problem shared among all these approaches is that they all can fail if the density ratio is unbounded . Such unbounded density ratio is typical for many of the high dimensional problems considered in machine learning [ 19 ] , e.g . see Figure 1 . Besides the aforementioned issues , the lack of principled strategies for parameter choice causes misinterpretations in the ranking of domain adaptation methods which are traditionally compared by performance , while often relying on different parameter choice strategies . In this work , we propose a principled method for choosing distance-penalizing parameters of feature representation learning approaches for unsupervised domain adaptation . Our approach starts with the observation that the distance-regularization setting of domain adaptation shares characteristics with regularized ill-posed inverse problems ( see Table 1 ) . In inverse problems , the regularization parameter can be optimally chosen by the fundamental balancing principle which optimizes an approximation-sample ( bias-variance ) trade-off [ 26 , 27 , 28 ] . We apply this principle for balancing domain distance and learning errors of target error bounds . In particular , we approach the problem of non-computable terms in the target error bound by a new algorithmic criterion for approximating the value of balance . We call our method the Balancing Principle for Domain Adaptation ( BPDA ) . The BPDA is general in the sense that it can be applied based on different target error bounds , e.g . on [ 8 , 22 , 19 , 20 , 21 ] . To the best of our knowledge , the BPDA is the first principled method for parameter choice in unsupervised domain adaptation that allows an unbounded ratio between target and source density . We provide a bound on the generalization error of the best model corresponding to the parameter chosen by the BPDA . Finally , we empirically investigate the behavior of the BPDA based on two target error bounds , different domain adaptation methods and benchmark datasets . Our results show that the BPDA outperforms or is on par with the state of the art on the problem of choosing the regularization parameter , on several domain adaptation methods ; applied on different datasets . 2 Summary of results . Notation Let X ⊂ Rn be an input space and Y be a discrete label space . Following the classical setting of unsupervised domain adaptation [ 25 ] , we consider two datasets : A source dataset ( x , y ) = ( ( x1 , lS ( x1 ) ) , . . . , ( xs , lS ( xs ) ) ) ∈ ( X × Y ) s with inputs x1 , . . . , xs independently drawn according to some source distribution ( Borel probability measure ) pS on X and labeled according to some labeling function1 lS : X → Y , and , an unlabeled target dataset x′ = ( x′1 , . . . , x′t ) ∈ X t with elements independently drawn according to some target distribution pT on X . Throughout this work , we focus on loss functions L : Y ×Y → [ 0 , ∞ ) which satisfy L ( y , y ) = 0 . For example consider the 0-1 loss L ( y1 , y2 ) : = 1 [ y1 ̸= y2 ] , where 1 [ P ] is 1 iff the predicate P is true and 0 otherwise , and the quadratic loss function L ( y1 , y2 ) : = |y1 − y2|2 . We denote the source error by εS ( f ) = εS ( f , lS ) with cross-error defined as εS ( f , g ) : = Ex∼pS [ L ( f ( x ) , g ( x ) ) ] and its empirical sample estimate by ε̂S ( f ) = ε̂S ( f , lS ) with ε̂S ( f , g ) : = ∑s i=1 L ( f ( xi ) , g ( xi ) ) . We denote the analogously defined target error by εT ( f ) , target cross-error by εT ( f , g ) and its empirical sample estimate by ε̂T ( f ) with empirical cross-error ε̂T ( f , g ) . Throughout this work , we focus on target cross-errors εT ( f , g ) which satisfy the triangle inequality . Learning setup In this work , we focus on feature representation learning algorithms for domain adaptation . These approaches aim at finding two learning models : A representation mapping ϕ ∈ Φ ⊂ { ϕ : X → R } into some representation space R ⊂ Rm and a classifier g ∈ G ⊂ { g : R → Y } . Loosely speaking , the aim is to find a mapping ϕ under which the source representations ϕ ( x ) : = ( ϕ ( x1 ) , . . . , ϕ ( xs ) ) and the target representations ϕ ( x′ ) : = ( ϕ ( x′1 ) , . . . , ϕ ( x ′ t ) ) appear similar , and , at the same time , enough information is preserved for prediction [ 14 ] by g ( x ) . A common approach to realize this aim is to solve the following objective function [ 19 ] min g∈G , ϕ∈Φ ε̂S ( g ◦ ϕ ) + α · d ( ϕ ( x ) , ϕ ( x′ ) ) ( 1 ) where d is a distance measure between source and target representations and α ∈ [ 0 , ∞ ) is a parameter2 . Good choices for d in Eq . ( 1 ) have been identified to be the Wasserstein distance [ 29 , 30 ] , the Maximum Mean Discrepancy [ 31 , 32 ] , moment distances [ 17 , 33 , 18 , 34 , 35 , 36 ] , adversarially learned distances [ 16 , 37 ] and other measures of divergence [ 38 , 39 , 19 , 20 ] . Problem For some α ∈ [ 0 , ∞ ) , let gα ◦ ϕα denote the minimizer of Eq . ( 1 ) . Given an increasing sequence of parameters α1 , . . . , αw ∈ [ 0 , ∞ ) with α1 = 0 , the problem studied in this work is to choose the parameter α in the sequence α1 , . . . , αw with the lowest target error εT ( gα ◦ ϕα ) . Approach Our approach consists in minimizing a target error bound which satisfies the form εT ( gα ◦ ϕα ) ≤ D ( α ) + E ( α ) ( 2 ) where D ( α ) gives a notion of domain distance ( cf . [ 25 ] ) by quantifying a distance between source and target data representations and E ( α ) comprises different learning errors . We assume that E ( α ) is bounded by some constant B > 0 . The general form in Eq . ( 2 ) is satisfied by many error bounds [ 8 , 22 , 20 , 21 , 34 ] which all can be taken as a basis for our approach ( more detailed examples are provided in Section 3 and Section 4 ) . One problem that complicates the minimization of these error bounds is that all of them contain terms that are not computable due to the lack of target data . That is , E ( α ) can not be directly estimated . The BPDA overcomes this problem by a new criterion for estimating the value of balance between the normalized terms E ( α ) and D ( α ) . The BPDA is detailed in Algorithm 1 . Properties of Algorithm 1 The BPDA has the following striking properties . • The BPDA is a general procedure which can be instantiated by any error bound of the form in Eq . ( 2 ) . See Section 4 and Section 5 for its application based on two different target error bounds [ 25 , 20 ] . 1For simplicity , we use labeling functions instead of the more general concept of conditional distributions . 2For simplicity we omit further regularization of ϕ and g. • In contrast to state-of-the-art methods , the BPDA does not assume a target labeling function lT that is equal to the source labeling function lS ( covariate-shift assumption ) and it does not assume that the ratio between target and source density is bounded ( sample selection bias assumption ) . See the supplementary material for a discussion of covariate-shift violations . • The learning model gαBP ◦ ϕαBP identified by the BPDA satisfies a generalization bound , see Section 4 . If the learning errors term E ( α ) is non-decreasing , then the target error of gαBP ◦ ϕαBP is only a constant factor away from the minimum minα∈ [ 0 , ∞ ) D ( α ) + E ( α ) of the instantiation bound in Eq . ( 2 ) . Algorithm 1 : Balancing principle for domain adaptation ( BPDA ) Input : Increasing sequence of parameters α1 , . . . , αw ∈ [ 0 , ∞ ) with α1 = 0 and minimizers f1 : = gα1 ◦ ϕα1 , . . . , fw : = gαw ◦ ϕαw of Eq . ( 1 ) . Output : Parameter αBP solving the problem above . Initialization : S = { } for i = 1 , . . . , w do Compute empirical cross-error ε̂T ( fi , fj ) and domain distance D ( αj ) for all j = 1 , . . . , i− 1. if ε̂T ( fi , fj ) ≤ D ( αj ) ( 2 + 2BD ( 0 ) ) for all j = 1 , . . . , i− 1 then S : = S ∪ { αi } end end return : αBP : = maxS Besides the properties above , the BPDA outperforms or is on par with the state of the art , on the problem of choosing the regularization parameter , on several domain adaptation methods ; applied on different datasets , see Section 5 . Related work Approaches which follow Eq . ( 1 ) are sometimes interpreted as learning domaininvariant representation . Note that the minimization of d ( ϕ ( x ) , ϕ ( x′ ) ) in Eq . ( 1 ) to achieve ϕ ( x ) = ϕ ( x′ ) differs from the conception as regularization [ 40 , 41 ] . In fact , minimization of the distance means unjustified over-penalization which might lead to deteriorated performance [ 19 ] . Interestingly , our interpretation as regularization problem opens up a powerful toolbox of mathematical techniques . Our approach takes up the technique of balancing stability and approximation in regularized illposed inverse problems . The balancing principle has its origins in [ 26 ] devoted to nonparametric regression estimation and has been introduced in the context of ill-posed problems [ 27 ] and in supervised learning with kernels [ 42 ] . Following this line of research , we propose to apply the mathematical techniques underlying the balancing principle in the context of domain adaptation . The most related principled parameter choice methods in the context of unsupervised domain adaptation are importance-weighted cross-validation [ 3 ] and its extensions [ 24 , 23 ] . In contrast to these methods , our method is not restricted by the assumption of a bounded ratio between target and source density . One empirically driven method which is related to ours is [ 17 , 43 ] which aims at balancing the source error and a distance between source and target data representations . However , this method is not theoretically justified as it ignores the minimal combined error of a classifier on representations as defined in [ 8 ] . Nevertheless , if the minimal combined error is negligibly small , our results provide a theoretical explanation of the principles underlying the success of [ 17 , 43 ] . Our method relies on bounds on the target error such as e.g . [ 8 , 22 , 19 , 20 , 21 , 34 ] .

Summary: In this paper, the authors study the hyperparameter selection problem in the context of unsupervised domain adaptation. While hyperparameter selection is not trivial at all due to the absence of the labeled data in the target domain, the authors employ the balancing principle to draw a connection between unsupervised domain adaptation and ill-posed inverse problems. The proposed method is based on balancing the discrepancy term and the error term, and the joint minimizer is chosen as a good hyperparameter. Notably, this method does not require any assumptions such as covariate shift and support matching as supposed in the existing researches.