The balancing principle for parameter choice in distance-regularized domain adaptation

1 Introduction . Domain adaptation uses the knowledge in a source domain to improve the performance of an algorithm on a related target domain [ 1 ] . In particular , domain adaptation tackles domain shifts in machine learning applications : Medical diagnostic systems should be adapted to new physical human variations ; Industrial quality inspection systems should be accurate for new products ; Self-driving cars should be able to adapt to new geographical environments and weather conditions . In this work , we focus on unsupervised domain adaptation where labels are only available in the source domain . There are mainly two types of approaches for unsupervised domain adaptation : importance weighting [ 2 , 3 , 4 , 5 , 6 , 7 ] and feature representation learning [ 8 , 9 , 10 , 11 , 12 , 13 ] . In this work , we focus on feature representation learning which goes beyond classical importance weighting by allowing a target distribution with support outside of the source distribution . The core idea behind feature representation learning approaches is to map the data into a new feature space where the source and target 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . data representations appear similar , and where enough information is preserved for prediction [ 14 ] . The similarity is often realized by regularization using distance measures between source and target representations [ 15 , 16 , 17 , 18 , 19 , 20 , 21 ] . However , the performance of such methods crucially depends on the choice of the regularization parameter which penalizes the distance . The problem we investigate in this work is to choose this parameter , which is sophisticated without any target labels . While remarkable theoretical results have been achieved which quantify the generalization ability of domain adaptation models [ 8 , 22 , 19 , 20 , 21 ] , the choice of the regularization parameter which is crucial for finding such models has not systematically been addressed . Even though some parameter choice strategies exist , they are either purely heuristically driven or very limited by their assumptions [ 23 ] . Typical approaches are fixing the regularization parameters [ 12 ] , minimizing the source error [ 16 ] , balancing the source error and a distance [ 17 ] , multiplying a fixed weighting parameter ( e.g . 1 in [ 16 ] ) by a heuristic schedule value that increases during training , or , ( importance ) weighting the input samples by the ratio between target and source density [ 3 , 24 , 23 ] . One common problem shared among all these approaches is that they all can fail if the density ratio is unbounded . Such unbounded density ratio is typical for many of the high dimensional problems considered in machine learning [ 19 ] , e.g . see Figure 1 . Besides the aforementioned issues , the lack of principled strategies for parameter choice causes misinterpretations in the ranking of domain adaptation methods which are traditionally compared by performance , while often relying on different parameter choice strategies . In this work , we propose a principled method for choosing distance-penalizing parameters of feature representation learning approaches for unsupervised domain adaptation . Our approach starts with the observation that the distance-regularization setting of domain adaptation shares characteristics with regularized ill-posed inverse problems ( see Table 1 ) . In inverse problems , the regularization parameter can be optimally chosen by the fundamental balancing principle which optimizes an approximation-sample ( bias-variance ) trade-off [ 26 , 27 , 28 ] . We apply this principle for balancing domain distance and learning errors of target error bounds . In particular , we approach the problem of non-computable terms in the target error bound by a new algorithmic criterion for approximating the value of balance . We call our method the Balancing Principle for Domain Adaptation ( BPDA ) . The BPDA is general in the sense that it can be applied based on different target error bounds , e.g . on [ 8 , 22 , 19 , 20 , 21 ] . To the best of our knowledge , the BPDA is the first principled method for parameter choice in unsupervised domain adaptation that allows an unbounded ratio between target and source density . We provide a bound on the generalization error of the best model corresponding to the parameter chosen by the BPDA . Finally , we empirically investigate the behavior of the BPDA based on two target error bounds , different domain adaptation methods and benchmark datasets . Our results show that the BPDA outperforms or is on par with the state of the art on the problem of choosing the regularization parameter , on several domain adaptation methods ; applied on different datasets . 2 Summary of results . Notation Let X ⊂ Rn be an input space and Y be a discrete label space . Following the classical setting of unsupervised domain adaptation [ 25 ] , we consider two datasets : A source dataset ( x , y ) = ( ( x1 , lS ( x1 ) ) , . . . , ( xs , lS ( xs ) ) ) ∈ ( X × Y ) s with inputs x1 , . . . , xs independently drawn according to some source distribution ( Borel probability measure ) pS on X and labeled according to some labeling function1 lS : X → Y , and , an unlabeled target dataset x′ = ( x′1 , . . . , x′t ) ∈ X t with elements independently drawn according to some target distribution pT on X . Throughout this work , we focus on loss functions L : Y ×Y → [ 0 , ∞ ) which satisfy L ( y , y ) = 0 . For example consider the 0-1 loss L ( y1 , y2 ) : = 1 [ y1 ̸= y2 ] , where 1 [ P ] is 1 iff the predicate P is true and 0 otherwise , and the quadratic loss function L ( y1 , y2 ) : = |y1 − y2|2 . We denote the source error by εS ( f ) = εS ( f , lS ) with cross-error defined as εS ( f , g ) : = Ex∼pS [ L ( f ( x ) , g ( x ) ) ] and its empirical sample estimate by ε̂S ( f ) = ε̂S ( f , lS ) with ε̂S ( f , g ) : = ∑s i=1 L ( f ( xi ) , g ( xi ) ) . We denote the analogously defined target error by εT ( f ) , target cross-error by εT ( f , g ) and its empirical sample estimate by ε̂T ( f ) with empirical cross-error ε̂T ( f , g ) . Throughout this work , we focus on target cross-errors εT ( f , g ) which satisfy the triangle inequality . Learning setup In this work , we focus on feature representation learning algorithms for domain adaptation . These approaches aim at finding two learning models : A representation mapping ϕ ∈ Φ ⊂ { ϕ : X → R } into some representation space R ⊂ Rm and a classifier g ∈ G ⊂ { g : R → Y } . Loosely speaking , the aim is to find a mapping ϕ under which the source representations ϕ ( x ) : = ( ϕ ( x1 ) , . . . , ϕ ( xs ) ) and the target representations ϕ ( x′ ) : = ( ϕ ( x′1 ) , . . . , ϕ ( x ′ t ) ) appear similar , and , at the same time , enough information is preserved for prediction [ 14 ] by g ( x ) . A common approach to realize this aim is to solve the following objective function [ 19 ] min g∈G , ϕ∈Φ ε̂S ( g ◦ ϕ ) + α · d ( ϕ ( x ) , ϕ ( x′ ) ) ( 1 ) where d is a distance measure between source and target representations and α ∈ [ 0 , ∞ ) is a parameter2 . Good choices for d in Eq . ( 1 ) have been identified to be the Wasserstein distance [ 29 , 30 ] , the Maximum Mean Discrepancy [ 31 , 32 ] , moment distances [ 17 , 33 , 18 , 34 , 35 , 36 ] , adversarially learned distances [ 16 , 37 ] and other measures of divergence [ 38 , 39 , 19 , 20 ] . Problem For some α ∈ [ 0 , ∞ ) , let gα ◦ ϕα denote the minimizer of Eq . ( 1 ) . Given an increasing sequence of parameters α1 , . . . , αw ∈ [ 0 , ∞ ) with α1 = 0 , the problem studied in this work is to choose the parameter α in the sequence α1 , . . . , αw with the lowest target error εT ( gα ◦ ϕα ) . Approach Our approach consists in minimizing a target error bound which satisfies the form εT ( gα ◦ ϕα ) ≤ D ( α ) + E ( α ) ( 2 ) where D ( α ) gives a notion of domain distance ( cf . [ 25 ] ) by quantifying a distance between source and target data representations and E ( α ) comprises different learning errors . We assume that E ( α ) is bounded by some constant B > 0 . The general form in Eq . ( 2 ) is satisfied by many error bounds [ 8 , 22 , 20 , 21 , 34 ] which all can be taken as a basis for our approach ( more detailed examples are provided in Section 3 and Section 4 ) . One problem that complicates the minimization of these error bounds is that all of them contain terms that are not computable due to the lack of target data . That is , E ( α ) can not be directly estimated . The BPDA overcomes this problem by a new criterion for estimating the value of balance between the normalized terms E ( α ) and D ( α ) . The BPDA is detailed in Algorithm 1 . Properties of Algorithm 1 The BPDA has the following striking properties . • The BPDA is a general procedure which can be instantiated by any error bound of the form in Eq . ( 2 ) . See Section 4 and Section 5 for its application based on two different target error bounds [ 25 , 20 ] . 1For simplicity , we use labeling functions instead of the more general concept of conditional distributions . 2For simplicity we omit further regularization of ϕ and g. • In contrast to state-of-the-art methods , the BPDA does not assume a target labeling function lT that is equal to the source labeling function lS ( covariate-shift assumption ) and it does not assume that the ratio between target and source density is bounded ( sample selection bias assumption ) . See the supplementary material for a discussion of covariate-shift violations . • The learning model gαBP ◦ ϕαBP identified by the BPDA satisfies a generalization bound , see Section 4 . If the learning errors term E ( α ) is non-decreasing , then the target error of gαBP ◦ ϕαBP is only a constant factor away from the minimum minα∈ [ 0 , ∞ ) D ( α ) + E ( α ) of the instantiation bound in Eq . ( 2 ) . Algorithm 1 : Balancing principle for domain adaptation ( BPDA ) Input : Increasing sequence of parameters α1 , . . . , αw ∈ [ 0 , ∞ ) with α1 = 0 and minimizers f1 : = gα1 ◦ ϕα1 , . . . , fw : = gαw ◦ ϕαw of Eq . ( 1 ) . Output : Parameter αBP solving the problem above . Initialization : S = { } for i = 1 , . . . , w do Compute empirical cross-error ε̂T ( fi , fj ) and domain distance D ( αj ) for all j = 1 , . . . , i− 1. if ε̂T ( fi , fj ) ≤ D ( αj ) ( 2 + 2BD ( 0 ) ) for all j = 1 , . . . , i− 1 then S : = S ∪ { αi } end end return : αBP : = maxS Besides the properties above , the BPDA outperforms or is on par with the state of the art , on the problem of choosing the regularization parameter , on several domain adaptation methods ; applied on different datasets , see Section 5 . Related work Approaches which follow Eq . ( 1 ) are sometimes interpreted as learning domaininvariant representation . Note that the minimization of d ( ϕ ( x ) , ϕ ( x′ ) ) in Eq . ( 1 ) to achieve ϕ ( x ) = ϕ ( x′ ) differs from the conception as regularization [ 40 , 41 ] . In fact , minimization of the distance means unjustified over-penalization which might lead to deteriorated performance [ 19 ] . Interestingly , our interpretation as regularization problem opens up a powerful toolbox of mathematical techniques . Our approach takes up the technique of balancing stability and approximation in regularized illposed inverse problems . The balancing principle has its origins in [ 26 ] devoted to nonparametric regression estimation and has been introduced in the context of ill-posed problems [ 27 ] and in supervised learning with kernels [ 42 ] . Following this line of research , we propose to apply the mathematical techniques underlying the balancing principle in the context of domain adaptation . The most related principled parameter choice methods in the context of unsupervised domain adaptation are importance-weighted cross-validation [ 3 ] and its extensions [ 24 , 23 ] . In contrast to these methods , our method is not restricted by the assumption of a bounded ratio between target and source density . One empirically driven method which is related to ours is [ 17 , 43 ] which aims at balancing the source error and a distance between source and target data representations . However , this method is not theoretically justified as it ignores the minimal combined error of a classifier on representations as defined in [ 8 ] . Nevertheless , if the minimal combined error is negligibly small , our results provide a theoretical explanation of the principles underlying the success of [ 17 , 43 ] . Our method relies on bounds on the target error such as e.g . [ 8 , 22 , 19 , 20 , 21 , 34 ] .

Summary: This paper studies the problem of choosing a justified regularization parameter for unsupervised domain adaptation. The authors make a link between ill-posed inverse problems and approaches based on a weighted combination of the source error and a distance between source and target feature representations. They propose a framework based on balancing approximations and sampling errors to obtain a target error bound that takes into account a balance between learning errors domain distance. This leads to develop a founded rule for the choice of the regularization parameter which can be applied with source and target distributions with disjoint supports.  A bound on the target risk is provided.  An experimental evaluation of the approach is provided using the Moons toy dataset and the Amazon reviews and DomainNet–2019 benchmark.