Unsupervised Multi-Target Domain Adaptation: An Information Theoretic Approach

1 INTRODUCTION . In real-world data , the training and test instances often do not come from the same underlying distribution ( Sun et al . ( 2016 ) ) . For example , in the task of object recognition/classification from image data , this may be due to the image noise , changes in the object view , etc. , which induce different biases in the observed data sampled during the training and test stage . Consequently , assumptions made by traditional learning algorithms are often violated , resulting in degradation of the algorithms ’ performance during inference of test data . Domain Adaptation ( DA ) approaches ( Fernando et al . ( 2013 ) ; Gong et al . ( 2012 ) ; Kodirov et al . ( 2015 ) ; Yoo et al . ( 2016 ) ) aim to tackle this by transferring knowledge from a source domain ( training data ) to an unlabeled target domain ( test data ) to reduce the discrepancy between the source and target data distributions , typically by exploring domain-invariant data structures . Existing DA methods tackle the adaptation problems in one of the two settings : ( semi ) supervised DA and unsupervised DA ( Csurka ( 2017 ) ) . The former assume that in addition to the labeled data of the source domain , some labeled data from the target domain are also available for training/adapting the classifiers . In contrast , the latter does not require any labels from the target domain but rather explores the similarity in the data distributions of the two domains . In this work , we focus on the unsupervised DA ( uDA ) scenario , which is more challenging due to the lack of correspondences in source and target labels . Most works on uDA today focus on a single-source-single-target-domain scenario . However , in many real-world applications , unlabeled data may come from different domains , thus , with different statistical properties but with common task-related content . For instance , we may have access to images of the same class of objects ( e.g. , cars ) recorded by various types of cameras , and/or under different camera views and at different times , rendering multiple different domains ( e.g. , datasets ) . Likewise , facial expressions of emotions , such as joy and surprise , shown by different people and recorded under different views , result in multiple domains with varying data distributions . In most cases , these domains have similar underlying data distributions , which can be leveraged to build more effective and robust classifiers for tasks such as the object or emotion recognition across multiple datasets/domains . To this end , traditional uDA methods focus on the single-source-single-target DA scenario . However , in the presence of multiple domains , as typically encountered in real-world settings , this pair-wise adaptation approach may be suboptimal as it fails to leverage simultaneously the knowledge shared across multiple task-related domains . Recently , Zhao et al . ( 2017 ) showed that by having access to multiple source domains can facilitate better adaptation to a single target domain , when compared to the pair-wise DA approach . While this is intuitive due to the access to multiple labelled source domains , offering more adaptation flexibility for the target domain ( i.e. , by efficiently exploring the data labels across multiple source domains that are most related to the target domain ) , it comes at the expense of the data labelling in multiple source domains , which can be costly and time-consuming . In either case , a single source domain or readily available multiple source domains , to the best of our knowledge , a simultaneous adaptation to multiple and unlabelled target domains remains an unexplored DA scenario . However , this DA scenario is important as we usually have access to multiple unlabeled domains ; yet , the adaptation process is also more challenging due to the lack of supervision in the target domains . Still , multi-target DA can have advantages over a single-target DA when : ( i ) there is direct knowledge sharing between the source and multiple target domains ( fig . 1a ) , and ( ii ) the source and a target domain are related through another target domain ( fig . 1b ) . While this seems intuitive , it is critical how the data from multiple unlabelled target domains are leveraged within the multi-target DA approach , in order to improve its performance over the single target DA approaches and naive fusion of multiple target domains . To this end , we propose a Multi-Target DA-Information-Theoretic-Approach ( MTDA-ITA ) for single-sourcemulti-target DA . We exploit a single source domain and focus on multiple target domains to investigate the effects of multi-target DA ; however , the proposed approach can easily be extended to multiple source domains . This approach leverages the data from multiple target domains to improve performance compared to individually learning from pair-wise source-target domains . Specifically , we simultaneously factorize the information from each available target domain and learn separate subspaces for modeling the shared ( i.e. , correlated across the domains ) and private ( i.e. , independent between the domains ) subspaces of the data ( Salzmann et al . ( 2010 ) ) . To this end , we employ deep learning to derive an information theoretic approach where we jointly maximize the mutual information between the domain labels and private ( domain-specific ) features , while minimizing the mutual information between the the domain labels and the shared ( domaininvariant ) features . Consequently , the more robust feature representations are learned for each target domain by exploiting dependencies between multiple target domains . We show on benchmark datasets for DA that this approach leads to overall improved performance on each target domain , compared to independent DA for each pair of source-target domains , or the naive combination of multiple target domains , and state-of-the-art models applicable to the target task . 2 THE PROPOSED METHOD . Without loss of generality , we consider a multi-class ( K-class ) classification problem as the running example . Furthermore , let ( X , Y , D ) = { ( xi , yi , di ) } Ni=0 be a collection of M domains ( a labeled source domain , and M − 1 unlabeled target domains ) , where xi denotes the i-th sample , and yi = [ y1i , y2i , ... , yKi ] and di = [ d 1 i , d 2 i , ... , d M i ] are the K-D and M -D encoding of the class and domain labels for xi , respectively . Note that the class labels are only available for the source samples . The latent space representation of the data point x is denoted as z = [ zs , zp ] , where zs and zp are the ( latent ) shared and private features of the data point x , respectively . By factorizing the joint distribution p ( x , y , d , zs , zp ) as p ( x , y , d , zs , zp ) = p ( x ) p ( d ) p ( zs|x ) p ( zp|x ) p ( y|zs ) , ( 1 ) we propose to maximize the following objective function : L ( θs , θp , θc ; x , y , d ) = λrI ( x ; z ) + λcI ( y ; zs ) + λd ( I ( d ; zp ) − I ( d ; zs ) ) , ( 2 ) where p ( x ) and p ( d ) denote the underlying ( true ) data distribution and domain label distribution , respectively , I ( x ; y ) denotes the Mutual Information between the random variables x and y. λr , λc and λd denote the hyper-parameters controlling the weights of the objective terms . The proposed objective function ( 2 ) maximizes the three terms described below : • I ( x ; z ) : encourages the latent features ( both shared and private ) to preserve information about the data samples ( that can be used to reconstruct x from z ) . • I ( y ; zs ) : enables to correctly predict the true class label of the samples out of their common shared features . • I ( d ; zp ) − I ( d ; zs ) : encourages the latent private features to preserve the information about the domain label and penalizes the latent shared features to be domain informative . This not only reduces the redundancy in the shared and private features , but also , penalizes the redundancy of different private spaces , while preserving the shared information . An additional term could be used to minimize the mutual information between the shared ( zs ) and private ( zp ) features . However , computing the mutual information ( even approximating it ) is intractable due to the highly complex joint distribution p ( zs , zp ) . Since we want zs and zp features to encode different aspects of x , we enforce such constraint by jointly maximizing the term : I ( d ; zp ) − I ( d ; zs ) . 2.1 OPTIMIZATION . The following lower bound for mutual information is derived using the non-negativity of KL-divergence ( Barber & Agakov ( 2003 ) ) ; i.e. , Σxp ( x|z ) ln p ( x|z ) q ( x|z ) ≥ 0 gives : I ( x ; z ) ≥ H ( x ) + Ep ( x , z ) [ ln q ( x|z ; φ ) ] ( 3 ) where H ( x ) denotes the Shanon Entropy ( Lin ( 1991 ) ) of the random variable x. q ( x|z ; φ ) is any arbitrary distribution parameterized by φ . We need a variational distribution q ( x|z ; φ ) because the posterior distribution p ( x|z ) = p ( z|x ) p ( x ) /p ( z ) is intractable since the true data distribution p ( x ) is assumed to be unknown . Similarly , we can derive lower bounds for I ( d ; zp ) ≥ H ( d ) + Ep ( d , zp ) [ ln q ( d|zp ; ψ ) ] and I ( d ; zs ) ≥ H ( d ) + Ep ( d , zs ) [ ln q ( d|zs ; ψ ) ] , where q ( d|zp ; ψ ) is any arbitrary distribution parameterized by ψ.1 We further drive lower bound for I ( y ; zs ) as I ( y ; zs ) ≥ H ( y ) + Ep ( y , zs ) [ ln q ( y|zs ; θc ) ] , where q ( y|zs ; θc ) is a variational distribution parameterized by θc approximating p ( y|zs ) . Let next Es ( x ; θs ) be a function parameterized by θs that maps a sample x to its corresponding shared feature zs , and Ep ( x ; θp ) be an analogous function which maps x to zp , the feature that is private to each domain ( fig . 2 ) . We also define F ( zs , zp ; φ ) as a decoding function mapping the concatenation of the latent features zs and zp to a sample reconstruction x̂ , and D ( z ; ψ ) as a decoding function mapping zs and zp to a M -dimensional vector : the predictions of the domain label d̂ . Finally , C ( zs ; θc ) is a task-specific function mapping zs to a K-dimensional probability vector of the class label ŷ . We represent p ( d ) , p ( x ) , p ( y ) as the empirical distribution of a finite training set ( e.g . p ( d ) = 1 N ∑N i=1 δ ( d − di ) ) as in the case of variational autoencoders ( VAE ) ( Abbasnejad et al . ( 2017 ) ; Pu et al . ( 2017 ) ) , p ( zs|x ) , p ( zp|x ) as deterministic functions of x as p ( zs|x ) = δ ( zs − Es ( x ; θs ) ) and p ( zp|x ) = δ ( zp − Ep ( x ; θp ) ) , and the variational distributions q ( y|zs ) , q ( x|z ) and , q ( d|z ) as q ( y|zs ) = SoftMax ( C ( zs ; θc ) ) , q ( d|z ) = SoftMax ( D ( z ; ψ ) ) , q ( x|z ; φ ) ∝ exp ( ‖x− F ( z ; φ ) ‖1 ) ( 4 ) where Softmax ( · ) denotes the softmax or normalized exponential function ( Bridle ( 1990 ) ) , and ‖.‖1 denotes the L1 norm . Then , the optimization task can be posed as a minimax saddle point problem , where we use adversarial training to maximize ( 2 ) w.r.t . the parameters ( θs , θp , θc ) , and to minimize ( 2 ) w.r.t . the parameters ( φ , ψ ) , using Stochastic Gradient Descent ( SGD ) . Optimizing the parameters φ of the decoder F φ̂ = arg min φ LF = λr N N∑ i=1 ‖xi − F ( Es ( xi ) , Ep ( xi ) ) ‖1 . ( 5 ) The decoder F ( zs , zp ; φ ) is trained in such a way so as to minimize the difference between original input x and its decoding from corresponding shared and private features via the decoder F . 1Note that , for simplicity , we shared the parameters ψ between the approximate posterior distributions q ( d|zs , ψ ) and q ( d|zp ; ψ ) . Optimizing the parameters ψ of the domain classifier D ψ̂ = arg min ψ LD = − λd N N∑ i=1 d > i lnD ( Es ( xi ) ) − λd N N∑ i=1 d > i lnD ( Ep ( xi ) ) . ( 6 ) D ( z ; ψ ) can be considered as a classifier whose task is to distinguish between the shared/private features of the different domains . More precisely , the two terms in Eq . 6 encourage D to correctly predict the domain labels from the shared and private features , respectively . Optimizing the parameters θc of the label classifier C θ̂c = arg min θc { −H ( y ) − Ep ( y , zs ) [ ln q ( y|zs ) ] } . ( 7 ) Since we have access to the source labels , H ( y ) is a constant for source samples . we can approximate H [ y ] for the target samples using the output of the classifier C , leading to the following optimization problem : θ̂c = arg min θc LC =− 1 N Ns∑ i=1 yTi lnC ( Es ( xi ) ) − λc N −Ns N∑ i=Ns+1 C ( Es ( xi ) ) > lnC ( Es ( xi ) ) + λc N −Ns N∑ i=Ns+1 C ( Es ( xi ) ) > ln ( 1 N −Ns N∑ i=Ns+1 C ( Es ( xi ) ) ) , ( 8 ) where Ns denotes the number of source samples . Intuitively , we enforce the classifier C to correctly predict the class labels of the source samples by the first term in Eq . 8 . We use the second term to minimize the entropy of q ( y|zs ) for the target samples ; effectively , reducing the effects of `` confusing '' labels of target samples , as given by p ( y|zs ) that leads to decision boundaries occur far away from target data-dense regions in the feature space . The intuition behind the last term is that by minimizing only the entropy ( second term ) , we may arrive at a degenerate solution where every target point xt is assigned to the same class . Hence , the last term encourages the classifier C to have balanced labeling for the target samples where it reaches its minimum , lnK , when each class is selected with uniform probability . Optimizing the parameter θp of the private encoder Ep θ̂p = arg min θp LP = λr N N∑ i=1 ‖xi − F ( Es ( xi ) , Ep ( xi ) ) ‖1 − λd N N∑ i=1 d > i D ( Ep ( xi ) ) . ( 9 ) The first term in Eq . 9 encourages the private encoder Ep to preserve the recovery ability of the private features . The second term enforces distinct private features be produced for each domain by penalizing the representation redundancy in different private spaces . This , in turn , encourages moving this common information from multiple domains to their shared space . Optimizing the parameter θs of the shared encoder Es θ̂s = arg min θs LS = λr N N∑ i=1 ‖x− F ( Es ( xi ) , Ep ( xi ) ) ‖1 − λc N Ns∑ i=1 yTi lnC ( Es ( xi ) ) − λd N N∑ i=1 d > i lnD ( Es ( xi ) ) − λc N −Ns N∑ i=Ns+1 C ( Es ( xi ) ) > lnC ( Es ( xi ) ) + λc N −Ns N∑ i=Ns+1 C ( Es ( xi ) ) > ln ( 1 N −Ns N∑ i=Ns+1 C ( Es ( xi ) ) ) . ( 10 ) The first term in Eq . 10 encourages the shared encoder Es to preserve the recovery ability of the shared features . The second term is the source domain classification loss penalty that encourages Es to produce discriminative features for the labeled source samples . The third term simulates the adversarial training by trying to fool the domain classifier D when predicting the domain labels d , given the shared features zs . The effect of this is two-fold : ( i ) the rendered shared features are more distinct from the corresponding private features , ( ii ) the shared features of different domains are encouraged to be similar to each other . The last two terms encourage Es to produce the shared features for target samples so that the classifier is confident on the unlabeled target data , driving the shared features away from the decision boundaries . To train our model , we alternate between updating the shared encoder Es , the private encoder Ep , the decoder F , the classifier C , and the domain classifier D using the SGD algorithm ( see Algorithm 1 in Appendix E for more details ) .

Summary: This paper investigates multi-target domain adaptation which is an unexplored domain adaptation scenario compared with adapting single/multiple source to single target. A mutual information-based loss is proposed to encourage part of the features to be domain-specific while the other part to be domain-invariant. Instead of optimizing the proposed loss which is intractable, this work proposes to use neural network to model the relative functions and optimize proposed loss’ lower bound by SGD.