Transfer Alignment Network for Double Blind Unsupervised Domain Adaptation

1 INTRODUCTION How can we minimize data distribution gap between source and target domains under constraint of data visibility , such that each side can not observe the data in the other side ? Recent state-of-the-art deep learning algorithms perform well on test data when given abundant labeled training data . However , in real applications , collecting a large amount of labeled data needs tedious labor work while unlabeled data are abundant . Given a certain number of unlabeled data from a target domain , unsupervised transfer learning exploits the knowledge from a related source domain , which has enough unlabeled data accompanying small number of labeled data , to improve the model performance on target domain . Since the source domain is related to but different from the target domain , data distribution shift exists between the two domains . Domain adaptation attempts to reduce the data distribution shift by effectively transferring knowledge from the source to the target domains . Recent state-of-the-art methods focus on digging out domain-invariant features , and learn more transferable representations to improve transferability . Cao et al . ( 2018 ) proposed transfer learner by extracting invariant feature representations while Long et al . ( 2018 ) applied generative adversarial model and generated feature extractor to confuse domain classifier . The premise of applying such methods is that source and target data are visible simultaneously . However , in real-world settings including hospital cases , it is not allowed to access source and target data simultaneously for privacy reason . In this paper , we define the problem of unsupervised domain adaptation under double blind constraint which restricts mutual accesses between source and target domains . As shown in Figure 1 , general domain adaptation can leverage source and target data simultaneously , but domain adaptation under double blind constraint can not utilize the two data simultaneously because of the restriction on data accessibility . Due to the constraint , existing methods on domain adaptation are not suitable for the new problem . We are exposed to two challenges : ( 1 ) since source and target data are not visible simultaneously , we have no way of knowing how much the data distribution shifts between the two domains , and ( 2 ) the target data can not be used to train a meaningful classifier due to the missing label . To address the challenges , we propose TAN ( Transfer Alignment Network for Double Blind Domain Adaptation ) , an effective method for double blind domain adaptation which resolves the data distribution shift problem by aligning source and target feature spaces . TAN maps target features into source feature space in order to utilize the classifier trained in source domain to target data . The experimental results show the superior performance of TAN . Our contributions in this paper are the followings : • Problem Definition . We propose a new problem of unsupervised domain adaptation under double blind constraint , which involves data inaccessibility during training process . • Method . We introduce a simple but effective model TAN which aligns data distribution of source and target domains . • Experiments . Extensive experiments on real-world multivariate datasets show that TAN provides the state-of-the-art performance . In the rest of this paper , we describe preliminaries , proposed method , experimental results , and related works , and conclude in the end . 2 PRELIMINARIES In this section , we provide brief description of autoencoder pre-training , and transfer learning with fine-tuning . 2.1 AUTOENCODER PRE-TRAINING In order to leverage unlabeled data in neural network training , the most common approach is to initialize parameters by weights pre-trained using autoencoder ( Erhan et al . ( 2010 ) ) . Autoencoder ( Bengio et al . ( 2009 ) ) is trained to reconstruct feature vector from code constrained either in dimensionality or sparsity . After training , the encoded vector from each input feature has enough information to reconstruct the original feature , and represents the hidden structure in the input feature space . Autoencoder is composed of two parts : encoder and decoder . Encoder maps the input feature into a constrained code , and decoder maps the code back to feature reconstruction . Given input feature vector x , encoder f maps x to code h = f ( x ) . The decoder g maps h to reconstruction x̂ = g ( h ) = g ( f ( x ) ) . Both encoder and decoder are defined as neural networks , with iterative applications of affine transformation and activation function . Then the model is trained to minimize the reconstruction error for each data point di in dataset . The objective function is defined as minθ i ||x̂i − xi||2 where x̂i is a reconstruction g ( f ( xi ) ) from input xi and θ denotes neural network parameters for encoder f and decoder g. Given large amount of unlabeled data and few labeled data , we use code vector f ( x ) from encoder as input in a classifier to effectively train it using the few labeled data . If the code vector has enough information to reconstruct the input feature vector , then it would contain enough information for the classifier as well . 2.2 TRANSFER LEARNING WITH FINE-TUNING Transfer learning with fine-tuning approach is simple and easy to apply in neural network training . In the training process , a neural network classifier learns useful intermediate hidden units ( Bengio ( 2012 ) ) . These hidden units provide enough information for simple classifier to achieve the stateof-the-art performance not only in the same task ( Sharif Razavian et al . ( 2014 ) ) but also in related tasks ( Sermanet et al . ( 2013 ) ) . This property has been exploited to use neural network trained on a source domain as initialization for further fine-tuning on a target domain . With careful analysis ( Yosinski et al . ( 2014 ) ) , it has been shown that transferring pre-trained neural network features improves performance . Fine-tuning is important to transform source-specific features into targetspecific features and to alleviate layer co-adaptation problem . It has been a common practice in neural network transfer learning to adapt parameters that are pre-trained from a source domain and fine-tune the model on a target domain . 3 PROPOSED METHOD In this section , we precisely define the problem of double blind unsupervised domain adaptation and its challenges . Then we describe our method TAN in detail . Figure 2 illustrates the architecture of our method in detail and Table 1 provides the definitions of symbols used . 3.1 PROBLEM DEFINITION AND CHALLENGES In unsupervised domain adaptation , a source domain has abundant unlabeled data Xs , u = { xs , ui } ns , u i=1 and small number of labeled data X s , l = { ( xs , li , y s , l i ) } ns , l i=1 . A target domain has only limited number of unlabeled data Xt , u = { ( xt , ui ) } nt i=1 . Note that the number ns , u of unlabeled samples from the source domain is larger than the number nt from the target domain . Both of source and target data contain the same number K of classes . The source and the target data are sampled from distributions p and q , respectively , where p ∕= q . Our goal is to learn a classifier which performs well in target classification task under double blind constraint , where each domain can not observe the data in the other domain , by leveraging both data in the source and target domains . Note that this problem of double blind domain adaptation is slightly different from blind domain adaptation ( Uzair & Mian ( 2017 ) ) since the former uses the data from both source and target domains for training , while the latter uses the data only from the source domain for training . Due to the double blind constraint , 1 ) existing domain adaptation models can not be implemented directly , and 2 ) the distribution discrepancy between source and target data is not measured easily . Another complication is that the target data have no labels , which makes it impossible to train a classifier if only using the target data without accessing the labeled data at the source domain . We need a novel method to match feature distributions of source and target domains . 3.2 TRANSFER ALIGNMENT NETWORK FOR DOUBLE BLIND DOMAIN ADAPTATION The target data can not be used to generate a classifier due to missing labels , which forces us to utilize the source domain data to train a classifier and make use of it in the target domain . Even though there are a large amount of data in the source domain , most of them are unlabeled ; it is hard to train a good classifier using only the small amount of labeled data in the source domain . Thus , we use the source unlabeled data to train an autoencoder to extract meaningful features . Then , the source labeled data go through the autoencoder to become valuable features , and we utilize the features with labels to train a classifier . Due to the double blind constraint , we can only transfer the model trained in the source domain , not data . After the model is transferred to the target domain , we finetune the autoencoder with the target unlabeled data . Because of data distribution discrepancy between the source and the target data , directly feeding the extracted target domain features into the pretrained classifier will seriously damage the accuracy . To tackle this problem , we add and train transfer adaptation layers , which we call “ transfer aligner ” , between the autoencoder and the classifier using only the target unlabeled data . After being passed through the transfer aligner , the target domain features have a distribution as close to the source distribution , so that we can use the classifier trained in the source domain to classify the target unlabeled data . Algorithm 1 TAN ( Transfer Alignment Network for Double Blind Domain Adaptation ) Input : source labeled data Xs , l , source unlabeled data Xs , u , and target unlabeled data Xt , u Output : encoder for autoencoder fencode , transfer aligner falign , and classifier fclassify 1 : ( Step 1 ) Train fencode and fdecode on Xs , u with L2 reconstruction loss 2 : ( Step 2 ) Train fencode and fclassify on Xs , l with cross-entropy classification loss 3 : Take snapshot fencode as fs , encode 4 : ( Step 3 ) Train fencode and fdecode on Xt , u with L2 reconstruction loss 5 : Take snapshot fencode as ft , encode 6 : ( Step 4 ) falign = TrainAligner ( fs , encode , ft , encode , Xt , u ) in Algorithm 2 Algorithm 2 Train Transfer Aligner Input : source encoder fs , encode , target encoder ft , encode , and target unlabeled data Xt , u Output : transfer aligner falign 1 : Initialize falign with random weights 2 : Calculate source code zsi = fs , encode ( xi ) and target code z t i = ft , encode ( xi ) for each data instance xi ∈ Xt , u 3 : Train falign to minimize nt i=1 ||zsi − zai ||2 using gradient based optimization , where zai = falign ( z t i ) The whole architecture is simple but effective , consisting of encoder , transfer aligner , and multilayer perceptron . Figure 2 shows the training and the test steps of TAN . We train an autoencoder with unlabeled source instances in Step 1 . The blue rectangle is a feature vector extracted from the encoder . In Step 2 , we utilize the encoder part trained from Step 1 by adding multilayer perceptron on top of the code vector and the model is trained with labeled source instances . In Step 3 , we transfer the model to the target domain , but only fine-tune the autoencoder using unlabeled target instances . In Step 4 , the transfer aligner ( in red ) is trained to directly map the target code vector into source code space using the unlabeled target instances . We train the transfer aligner falign to map the code of the target encoder ft , encode to the code of the source encoder fs , encode for the target unlabeled data Xt , u . As shown in Figure 2 , the encoder , the transfer aligner , and the classifier are stacked together for the complete model . We test the final model with unlabeled target test data . The whole algorithms are described in Algorithms 1 and 2 . We further elaborate each step as follows : • Step 1 . The encoder fencode and decoder fdecode are trained on source unlabeled data Xs , u . The model is trained to minimize the following autoencoder reconstruction objective function . xi∈Xs , u ||xi − fdecode ( fencode ( xi ) ) ||2 ( 1 ) • Step 2 . We train a classifier fclassify on top of the previous encoder fencode on source labeled data Xs , l with fine-tuning . The training process minimizes the following crossentropy objective function . − xi∈Xs , l k∈K yk log fclassify ( fencode ( xi ) ) k ( 2 ) We call the encoder fencode and the classifier fclassify at this step as source encoder fs , encode and source classifier fs , classify , respectively . Snapshot of current parameters of the source encoder fs , encode is stored separately . • Step 3 . The encoder fencode and the decoder fdecode is fine-tuned on target unlabeled data Xt , u again with the following standard autoencoder reconstruction objective function . xi∈Xt , u ||xi − fdecode ( fencode ( xi ) ) ||2 ( 3 ) We call the encoder fencode with fine-tuned parameters at this step as target encoder ft , encode . Snapshot of current parameters of target encoder ft , encode is again stored separately . We present experimental results to answer the following questions . • Q1 . ( Predictive Performance ) What are the accuracies of TAN and competitors for double blind domain adaptation ? • Q2 . ( Effect of Transfer Aligner Training ) Does the benefit of transfer aligner come from alignment of features rather than deployment of more layers ? • Q3 . ( Effect of Amount of Target Data ) Is the model affected by the proportion of source and target data ? 4.1 EXPERIMENTAL SETTING Datasets . We conduct experiments on five multivariate datasets summarized in Table 2 : HIGGS1 , HEPMASS2 , SUSY3 , Sensorless4 , and Gas5 . HEPMASS , HIGGS and SUSY are sensor observations of particle collision experiments where labels are binary and denote the actual generation of the particles of interest . Sensorless and Gas are also sensor observations where labels denote the diagnosis of motor status and the type of gas in chemical reaction , respectively . Data preprocessing . In order to perform the domain adaptation experiments , we require a dataset with source/target labeled/unlabeled instances . We divide the total training data into 10 clusters using K-means so that instances in each cluster are closer to each other compared to those in other clusters . We first select a random point xr . Then , we set the farthest point c1 from xr as the first centroid . The 2nd centroid c2 is determined to be the point with the farthest distance to c1 . The kth centroid ck for k = 3 , 4 , ... , 10 is determined as the point with the farthest shortest distance to the existing centroids : ck = argmaxxi min ( ||xi − c1||2 , ... , ||xi − ck−1||2 ) . The found 10 clusters can be divided into source and target data with 9:1 ratio , which leads to 10 possible combinations of source and target data . We select the cluster with the largest H-divergence to other clusters as the target data . For efficient computation of H-divergence , we approximate it using Proxy A-distance d̂A = 2 ( 1 − 2 ) where is the error of correctly discriminating source and target domains ( BenDavid et al . ( 2010 ) ) using SVM . Because target data instances are sampled from only one cluster , source and target datasets have different feature distribution as intended . The final data ratio is Source Unlabeled : Source Labeled : Source Test : Target Unlabeled : Target test = 80.1:8.9:1:9:1 . Model settings . We examine the influence of transfer aligner for domain adaptation with the following architecture : 80-40-20-10 autoencoder , 30-80-150 transfer aligner , and 300-300-300-300- 1http : //archive.ics.uci.edu/ml/datasets/HIGGS 2http : //archive.ics.uci.edu/ml/datasets/HEPMASS 3http : //archive.ics.uci.edu/ml/datasets/SUSY 4http : //archive.ics.uci.edu/ml/datasets/Dataset+for+Sensorless+Drive+

Summary: This paper tackles the transfer learning problem with the double-blind unsupervised domain adaptation, where either the source or the target domain cannot observe the data in the other domain, but data from both domains are used for training. The high-level intuition of this paper is based on the observation that in some practical settings, the transferring source data to the target domain is restricted due to the privacy policy. The goal is to learn a classifier which performs well in target classification task under double-blind constraint.