A General Upper Bound for Unsupervised Domain Adaptation

1 Introduction . The advent of deep convolutional neural networks ( Krizhevsky et al. , 2012 ) brings visual learning into a new era . However , the performance heavily relies on the abundance of data annotated with ground-truth labels . Since traditional machine learning assumes a model is trained and verified in a fixed distribution ( single domain ) , where generalization performance is guaranteed by VC theory ( N. Vapnik , 2000 ) , thus it can not always be applied to real-world problem directly . Take image classification task as an example , a number of factors , such as the change of light , noise , angle in which the image is pictured , and different types of sensors , can lead to a domain-shift thus harm the performance when predicting on test data . Therefore , in many practical cases , we wish that a model trained in one or more source domains is also applicable to another domain . As a solution , domain adaptation ( DA ) aims to transfer the knowledge learned from a source distribution , which is typically fully labeled into a different ( but related ) target distribution . This work focus on the most challenging case , i.e , unsupervised domain adaptation ( UDA ) , where no target label is available . Ben-David et al . ( 2010 ) suggests that target error can be minimized by bounding the error of a model on the source data , the discrepancy between distributions of the two domains , and a small optimal joint error . Owing to the strong representation power of deep neural nets , many researchers focus on learning domain-invariant features such that the discrepancy of two feature spaces can be minimized . For aligning feature distributions across domains , mainly two strategies have been substantially explored . The first one is bridging the distributions by matching all their statistics ( Long et al. , 2015 ; 2017 ; Pan et al. , 2009 ) . The second strategy is using adversarial learning ( Goodfellow et al. , 2014 ) to build a minimax game between domain discriminator and feature extractor , where a domain discriminator is trained to distinguish the source from the target while the feature extractor is learned to confuse it simultaneously ( Ganin & Lempitsky , 2015 ; Ganin et al. , 2016 ; Tzeng et al. , 2017 ) . In spite of the remarkable empirical results accomplished by feature distribution matching schemes , they still suffer from a major limitation : the joint distributions of feature spaces and categories are not well aligned across data domains . As is reported in Ganin et al . ( 2016 ) , such methods fail to generalize in certain closely related source/target pairs , e.g. , digit classification adaptation from MNIST to SVHN . One potential reason is when matching marginal distributions of source and target domains , samples from different classes can be mixed together , where the joint error becomes nonnegligible since no hypothesis can classify source and target at the same time . This work aims to address the above problem by incorporating joint error to formalize an optimizable upper bound such that the undesired overlap due to a wrong match can be properly penalized . We evaluate our proposal on several different classification tasks . In some experimental settings , our method outperforms other methods by a large margin . The contributions of this work can be summarized as follows : · We propose a novel upper bound taking joint error into account and theoretically prove that our proposal can reduce to several other methods under certain simplifications . · We construct a constrained hypothesis space such that a much tighter bound can be obtained during optimization . · We adopt a novel measurement , namely cross margin discrepancy , for the dissimilarity of two hypotheses on certain domain to alleviate the instability during adversarial learning and provide reliable performance . 2 Related Work . The upper bound proposed by Ben-David et al . ( 2010 ) invokes numerous approaches focusing on reducing the gap between source and target domains by learning domain-invariant features , which can be achieved through statistical moment matching . Long et al . ( 2015 ; 2017 ) use maximum mean discrepancy ( MMD ) to match the hidden representations of certain layers in a deep neural network . Transfer Component Analysis ( TCA ) ( Pan et al. , 2011 ) tries to learn a subspace across domains in a Reproducing Kernel Hilbert Space ( RKHS ) using MMD that dramatically minimize the distance between domain distributions . Adaptive batch normalization ( AdaBN ) ( Li et al. , 2018 ) modulates the statistics from source to target on batch normalization layers across the network in a parameterfree way . Another way to learn domain-invariant features is by leveraging generative adversarial network to produce target features that exactly match the source . Ganin & Lempitsky ( 2015 ) relax divergence measurement in the upper bound by a worst case which is equivalent to the maximum accuracy that a discriminator can possibly achieve when distinguishing source from target . Tzeng et al . ( 2017 ) follow this idea but separate the training procedure into classification stage and adversarial learning stage where an independent feature extractor is used for target . Saito et al . ( 2017b ) explore a tighter bound by explicitly utilizing task-specific classifiers as discriminators such that features nearby the support of source samples will be favored by extractor . Zhang et al . ( 2019 ) introduce margin disparity discrepancy , a novel measurement with rigorous generalization bounds , tailored to the distribution comparison with the asymmetric margin loss to bridge the gap between theory and algorithm . Methods perform distribution alignment on pixel-level in raw input , which is known as image-to-image translation , are also proposed ( Liu & Tuzel , 2016 ; Bousmalis et al. , 2017 ; Sankaranarayanan et al. , 2017 ; Shrivastava et al. , 2016 ; Hoffman et al. , 2018 ; Murez et al. , 2017 ) . Distribution matching may not only bring the source and target domains closer , but also mix samples with different class labels together . Therefore , Saito et al . ( 2017a ) ; Sener et al . ( 2016 ) ; Zhang et al . ( 2018 ) aim to use pseudo-labels to learn target discriminative representations encouraging a lowdensity separation between classes in the target domain ( Lee , 2013 ) . However , this usually requires auxiliary data-dependent hyper-parameter to set a threshold for a reliable prediction . Long et al . ( 2018 ) present conditional adversarial domain adaptation , a principled framework that conditions the adversarial adaptation models on discriminative information conveyed in the classifier predictions , where the back-propagation of training objective is highly dependent on pseudo-labels . 3 Proposed Method . 3.1 A General Upper Bound . We consider the unsupervised domain adaptation as a binary classification task ( our proposal holds for multi-class case ) where the learning algorithm has access to a set of n labeled points { ( xis , yis ) ∈ ( X × Y ) } ni=1 sampled i.i.d . from the source domain S and a set of m unlabeled points { ( xit ) ∈ X } mi=1 sampled i.i.d . from the target domain T . Let fS : X → { 0 , 1 } and fT : X → { 0 , 1 } be the optimal labeling functions on the source and target domains , respectively . Let ( usually 0-1 loss ) denotes a distance metric between two functions over a distribution that satisfies symmetry and triangle inequality . As a commonly used notation , the source risk of hypothesis h : X → { 0 , 1 } is the error w.r.t . the true labeling function fS under domain S , i.e. , S ( h ) : = S ( h , fS ) . Similarly , we use T ( h ) to represent the risk of the target domain . With these notations , the following bound holds : T ( h ) = T ( h , fT ) = T ( h , fT ) − T ( h , fS ) + T ( h , fS ) + S ( h , fS ) − S ( h , fS ) + S ( h , fT ) − S ( h , fT ) = S ( h , fS ) + ( T ( h , fT ) − T ( h , fS ) ) + ( S ( h , fT ) − S ( h , fS ) ) + T ( h , fS ) − S ( h , fT ) ≤ S ( h ) + T ( fS , fT ) + S ( fS , fT ) + T ( h , fS ) − S ( h , fT ) = S ( h ) + CS , T ( fS , fT , h ) ( 1 ) For simplicity , we use CS , T ( fS , fT , h ) to denote T ( fS , fT ) + S ( fS , fT ) + T ( h , fS ) − S ( h , fT ) . The above upper bound is minimized when h = fS , and it is equivalent to T ( fS , fT ) owing to the triangle inequality : S ( h ) + S ( fS , fT ) = S ( h , fS ) + S ( fS , fT ) ≥ S ( h , fT ) ( 2 ) Furthermore , we demonstrate in such case , our proposal is equivalent to an upper bound of optimal joint error λ because : T ( fS , fT ) = T ( fS , fT ) + S ( fS , fS ) = T ( fS ) + S ( fS ) ≥ min h ( T ( h ) + S ( h ) ) = λ ( 3 ) Fig . 1b illustrates a case where common methods fail to penalize the undesirable situation when samples from different classes are mixed together during distribution matching , while our proposal is capable to do so ( for simplicity we assume fS takes a specific form , then T ( fS , fT ) measures the overlapping area 2 and 5 , which is equivalent to the optimal joint error λ ) . 3.2 Hypothesis Space Constraint . Since optimal labeling functions fS , fT are not available during training , we shall further relax the upper bound by taking supreme w.r.t fS , fT within a hypothesis space H : T ( h ) ≤ S ( h ) + CS , T ( fS , fT , h ) ≤ S ( h ) + sup f1 , f2∈H CS , T ( f1 , f2 , h ) ( 4 ) Then minimizing target risk T ( h ) becomes optimizing a minimax game and since the max-player taking two parameters f1 , f2 is too strong , we introduce a feature extractor g to make the min-player stronger . Applying g to the source and target distributions , the overall optimization problem can be written as : min g , h ( g ( S ) ( h ) + max f1 , f2∈H Cg ( S ) , g ( T ) ( f1 , f2 , h ) ) ( 5 ) However , if we leave H unconstrained , the supreme term can be arbitrary large . In order to obtain a tight bound , we need to restrict the size of hypothesis space as well as maintain the upper bound . For fS ∈ H1 ≤ H and fT ∈ H2 ≤ H , the following holds : Cg ( S ) , g ( T ) ( fS , fT , h ) ≤ sup f1∈H1 , f2∈H2 Cg ( S ) , g ( T ) ( f1 , f2 , h ) ≤ sup f1 , f2∈H Cg ( S ) , g ( T ) ( f1 , f2 , h ) ( 6 ) The constrained subspace for H1 is trivial as according to its definition , fS must belong to the space consisting of all classifiers for source domain , namely Hsc . However , the constrained subspace for H2 is a little problematic since we have no access to the true labels of the target domain , thus it is hard to locate fT . Therefore , the only thing we can do is to construct a hypothesis space for H2 that most likely contains fT . As is illustrated in Fig . 1c , when matching distributions of source and target domain , if the ideal case is achieved where the conditional distributions of source and target are perfectly aligned , then it is fare to assume fT ∈ Hsc . However , if the worst case is reached where samples from different class are mixed together , then we tend to believe fT /∈ Hsc . Considering this , we present two proposals in the following sections based on different constraints .

Summary: This paper presents a novel theoretical analysis for unsupervised domain adaptation by revisiting the \lambda joint error term charactering adaptability in the seminal analysis of Ben-David et al. They propose to replace it by considering discrepancy between information on constrained class hypothesis and a possible discrepancy term that related the learned model with the class A cross-margin discrepancy is proposed in the multiclass context. They extend their approach by proposing to reweight differently some errors with an expected performance of target models on source and an additional one where they distinguish the performance with respect to accuracy on pseudo-labeled target data. Their approach leads to an adversarial-based loss function to optimize which is evaluated on two visual domain adaptation tasks.