Title: PRINCIPLED HYBRIDS OF GENERATIVE AND DISCRIM-INATIVE DOMAIN ADAPTATION

Abstract: We propose a probabilistic framework for domain adaptation that blends both generative and discriminative modeling in a principled way. Under this framework, generative and discriminative models correspond to specific choices of the prior over parameters. This provides us a very general way to interpolate between generative and discriminative extremes through different choices of priors. By maximizing both the marginal and the conditional log-likelihoods, models derived from this framework can use both labeled instances from the source domain as well as unlabeled instances from both source and target domains. Under this framework, we show that the popular reconstruction loss of autoencoder corresponds to an upper bound of the negative marginal log-likelihoods of unlabeled instances, where marginal distributions are given by proper kernel density estimations. This provides a way to interpret the empirical success of autoencoders in domain adaptation and semi-supervised learning. We instantiate our framework using neural networks, and build a concrete model, DAuto. Empirically, we demonstrate the effectiveness of DAuto on text, image and speech datasets, showing that it outperforms related competitors when domain adaptation is possible.

1. INTRODUCTION:

Making accurate predictions relies heavily on the existence of labeled data for the desired tasks. However, generating labeled data for new learning tasks is often time-consuming. As a result, this poses an obstacle for applying machine learning methods to broader application domains. Domain adaptation focuses on the situation where we only have access to labeled data from source domain, which is assumed to be different from the target domain we want to apply our model to. The goal of domain adaptation algorithms under this setting is to generalize better in the target domain by exploiting labeled data in the source domain and unlabeled data in the target domain.

In this paper we propose a probabilistic framework for domain adaptation that combines both generative and discriminative modeling in a principled way. We start from a very simple yet general generative model, and show that a special choice on the prior distribution of model parameters leads to the usual discriminative modeling. Due to its generative nature, the framework provides us a principled way to use unlabeled instances from both the source and the target domains. Under this framework, if we use non-parametric kernel density estimators for the marginal distribution over instances, we can show that the popular reconstruction loss of autoencoders corresponds to an upper bound of the negative marginal log-likelihoods of unlabeled instances. This provides a novel probabilistic interpretation on why unsupervised training with general autoencoders may help with discriminative tasks, though interpretations exist for specific variants of autoencoders, e.g., denoising autoencoders (Vincent et al., 2008) and contractive autoencoders (Rifai et al., 2011) . Our interpretation may also be used to explain the recent success of autoencoders in semi-supervised learning (Rasmus et al., 2015) .

We instantiate our framework with flexible neural networks, which are powerful function approximators, leading to a concrete model, DAuto. DAuto is designed to achieve the following three objectives simultaneously in a unified model: 1). It learns representations that are informative for the main learning task in the source domain. 2). It learns domain-invariant features that are indistinguishable between the source and the target domains. 3). It learns robust representations under reconstruction loss for instances in both domains. To demonstrate the effectiveness of DAuto, we first conduct a synthetic experiment using the MNIST dataset, showing its superior performance when adaptation is possible. We further compare DAuto with state-of-the-art models on the Amazon benchmark dataset. As another contribution, we extend DAuto so that it can also be applied in time-series modeling. We evaluate it on a speech recognition task, showing its effectiveness in improving recognition accuracies when trained from utterances with different accents. In the end, we also provide qualitative analysis in the case where domain adaptation is hard and all the methods we test fail.

2. RELATED WORK:

Recently, due to the availability of rich data and powerful computational resources, non-linear representations and hypothesis classes for domain adaptation have been increasingly explored (Glorot et al., 2011; Baktashmotlagh et al., 2013; Chen et al., 2012; Ajakan et al., 2014; Ganin et al., 2016) . This line of works focuses on building common and robust feature representations among multiple domains using either supervised neural networks (Glorot et al., 2011) , or unsupervised pretraining using denoising auto-encoders (Vincent et al., 2008; 2010) . Other works focus on learning feature transformations such that the feature distributions in the source and target domains are close to each other (Ben-David et al., 2007; 2010; Ajakan et al., 2014; Ganin et al., 2016) . In practice it was observed that unsupervised pretraining using stacked denoising auto-encoders (mSDA) (Vincent et al., 2008; Chen et al., 2012) often improves the generalization accuracy (Ganin et al., 2016) . One of the limitations of mSDA is that it needs to explicitly form the covariance matrix of input features and then solves a linear system, which can be computationally expensive to solve exactly in high dimensional settings, but approximate scheme exists. Domain adversarial neural networks (DANN) is a discriminative model to learn domain-invariant features (Ganin et al., 2016) . It can be formulated as a minimax problem where the feature transformation component tries to learn a representation to confuse a following domain classification component. DANN also enjoys a nice theoretical justification to learn a feature map to decrease the A-distance measure (Ben-David et al., 2007) between source and target domains. Other distance measures between distributions can also be applied. Tzeng et al. (2014) and Long et al. (2015) propose similar models where the maximum mean discrepancy (MMD) (Gretton et al., 2012) between two domains are minimized. Tzeng et al. (2017) also proposed a variant of DANN, known as ADDA, where the encoders for source and target domains are not shared. Instead of joint training objective classifier and domain discriminator, ADDA runs in two stages. At the first stage, source encoder and classifier are trained in a supervised way on the source domain. At the second stage, both source encoder and source classifier are fixed, and both domain discriminator and target encoder are trained using only unsupervised samples from both domains. Very recently, Bousmalis et al. (2016) propose a model where orthogonal representations that are shared between domains and unique to each domain are learned simultaneously. They achieve this goal by incorporating both similarity and difference penalties for features into the objective function. Finally, domain adaptation can also be viewed as a semi-supervised learning problem by ignoring the domain shift, where source instances are treated as labeled data and target instances are unlabeled data (Dai et al., 2007; Rasmus et al., 2015) .

DAuto improves over mSDA (Chen et al., 2012) when the dimension of feature vectors is high and as a result the covariance matrix cannot be explicitly formed. As we will see later, DAuto can also be naturally extended to modeling in time-series domain. On the other hand, compared with DANN, DAuto has a clear probabilistic generative model interpretation and provides us a principled way to use unlabeled data from both the source and target domains during training.

3. A PRINCIPLED HYBRID MODEL FOR DOMAIN ADAPTATION:

Generative models provide a principled way to use both labeled data from source domain and unlabeled data from both domains. In this section we start with a general probabilistic framework for domain adaptation using a principled hybrid of both generative and discriminative models. We then provide a concrete instantiation of our framework, and show that it leads to popular reconstructionbased domain adaptation models. Our derivation can also be used to explain the prevalence and success of autoencoders in both domain adaptation and semi-supervised learning (Rasmus et al., 2015; Bousmalis et al., 2016) . We end this section by proposing the DAuto model that follows our probabilistic framework and combines the minimax adversarial loss as a regularizer for domain adaptation.

3.1. A PROBABILISTIC FRAMEWORK FOR DOMAIN ADAPTATION:

Let x ∈ R d be an input instance and y be its target variable: y ∈ {0, 1} in the classification setting or y ∈ R in the regression setting. A fully generative model can be specified as follows:

p(x, y; φ, ψ) = p(x; φ)p(y | x; ψ)p(φ, ψ) (1)

where φ is the model parameter that governs the generation process of x; ψ is the model parameter for the conditional distribution y | x, and p(φ, ψ) is the prior distribution over both model parameters. At first glance, one might find the above generative model unsuitable under the domain adaptation setting since it implicitly assumes that the marginal distribution p(x; φ) is shared among all instances from both domains. The key point that makes the above framework still valid under domain adaptation lies in the possible richness of the generation process parametrized by φ: although the marginal distribution may be different in the input space R d , we can still find proper transformation f : R d → R D such that the induced marginal distributions (by f ) over both domains are similar in R D (Fig. 1 ). In fact, this is a necessary and implicit assumption that underlies the recent success of regularization based discriminative models (Tzeng et al., 2014; Long et al., 2015; Ganin et al., 2016) .

To better understand this, we illustrate the generative process for x in Fig. 1 .

Using the above joint model, if we assume that the prior distribution p(φ, ψ) factorizes as p(φ, ψ) = p(φ)p(ψ), then we will have:

EQUATION

Note that in this case only the first term in R.H.S. in (2) is concerned with the prediction, which means unsupervised learning on p(x; φ)p(φ) does not help generalization on prediction. In other words, the independence assumption between φ and ψ equivalently reduces our joint model over both x and y into discriminative models that only contain parameters ψ if we only care about prediction accuracy. We emphasize that in this reduction, the assumption p(φ, ψ) = p(ψ)p(φ) is crucial, otherwise the optimal ψ * will still depend on φ, i.e., ψ * = ψ * (φ). On the other extreme, if we have φ = ψ, then this corresponds to having a prior p(φ, ψ) = p 0 (φ, ψ)δ(φψ) that constrains φ and ψ to be shared in both generative processes:

EQUATION

where p 0 is a base distribution and δ(•) denotes the Kronecker delta function. It can be seen that when φ = ψ, the formulation exactly reduces to the usual MAP inference scheme in a generative model over both x and y, given parameter φ.

The above discussion shows that the formulation given in (1) is general enough to incorporate both discriminative and generative modelings as two extreme cases, and depending on the choice of the prior distribution over φ and ψ, we can easily recover both. In a nutshell, when φ and ψ are independent, we recover the discriminative modeling; otherwise if φ and ψ are exactly the same, we recover the generative modeling. However in practice the sweet spot often lies in a mix of both models (Ng & Jordan, 2002) : discriminative training usually wins at predictive accuracy, while generative modeling provides a principled way to use unlabeled data. To achieve the best of both worlds, now let us consider the case where φ and ψ have a common subspace, i.e., some model parameters are shared in both the generation process of x and y | x. Clearly under this case the factorization assumption of the prior distribution does not hold anymore, and we cannot hope to recover a discriminative model by simply optimizing ψ. To make our discussion concrete, think if we have p(x; φ) = g(f (x; ζ); φ\ζ) and p(y | x; ψ) = h(f (x; ζ), y; ψ\ζ), where ζ are the shared parameters of both φ and ψ. Domain adaptation is possible under this setting whenever f (•; ζ) forms a rich class of transformations so that unlabeled instances from both domains have similar induced marginal distribution. This is a necessary condition for domain adaptation to succeed under our framework. As a generative model, it also allows algorithms to use unlabeled instances from both domains to optimize the marginal likelihood function p(x; φ), which also helps the predictive task p(y | x; ψ) due to the shared component ζ.

3.2. AN INSTANTIATION USING KERNEL DENSITY ESTIMATION:

Now we use our probabilistic framework and instantiate it with proper choices of both the marginal distributions as well as the conditional distributions. On one hand, we would like to make as few assumptions as possible about the generation process of x; on the other hand, the model should be rich enough such that even though instances from source and target domains may have different distributions in R d , the model contains flexible transformation under which the induced distributions are similar enough in both domains. Taking both considerations into account, we propose to use nonparametric kernel density estimator (KDE) to model p(x; φ). Specifically, let K(•) be the chosen kernel and {x i } n i=1 be a set of unlabeled instances. Our KDE for p(x; φ) is given by:

p(x; φ) ∝ 1 nw n i=1 K x -g(f (x i ; ζ); φ\ζ) w (4)

where w > 0 is the bandwidth and f : R d → R D and g : R D → R d are two feature transformations.

Our definition of KDE differs from the original one (Wassermann, 2006) by the additional parametric transformations g • f applied to x, and when g • f = I, the identity map, our definition reduces to the original one. Note that when applied to the source and target domains separately, the original KDE does not give similar density estimations if their empirical distributions are far from each other, which is exactly the case under domain adaptation.

R d X T X S R D X S X T f g R d p S (x)

p T (x)

Figure 1 : Density estimator with additional transformations f and g. Input instances X S and X T are from source and target domains, respectively. f transforms both X S and X T to R D so that they are close in R D ; g transforms them back so that they have close density estimations in R d .

For the conditional distribution y | x, depending on whether y ∈ R or y ∈ {0, 1}, typical choices include linear regression or logistic regression. While both these two models are linear and limited, we can first augment them with rich nonlinear transformation f applied to the input instance.

Note that the transformation f , along with its parameters, are shared between both p(x; φ) and p(y | x; ψ). Finally, our model is completed by specifying the prior distribution as follows:

p(φ, ψ) = p 0 (φ, ψ)δ(φ(ζ) -ψ(ζ))

The δ(•) constrains the common parameter ζ to be shared by both p(x; φ) and p(y | x; ψ). The base distribution p 0 (φ, ψ) can be chosen as a flat (possibly improper) prior, which corresponds to the usual MLE criterion; or other forms of distributions that effectively introduce regularizations on both φ and ψ.

3.3. LEARNING BY MAXIMIZING JOINT LIKELIHOOD:

One of the advantages a generative model lends us is a principled way of using unlabeled data from both domains. Let {(x i , y i )} m i=1 be labeled data from source domain and {x j } n j=1 be unlabeled data from both domains. Instead of just maximizing the conditional likelihood using labeled data, we can jointly maximize both the conditional likelihood and the marginal likelihood:

EQUATION

It is clear that the negative conditional log-likelihood function is exactly the cross-entropy loss between the true label y i and its predicted counterpart. For the negative marginal likelihood, if we choose a Gaussian kernel and plug in our KDE estimator given in (4), we have:

-log p(x j ; φ) ∝ -log   1 nw n j =1 exp - ||x j -g(f (x j ; ζ); φ\ζ)|| 2 2 2w 2   ≤ -log 1 nw exp - ||x j -g(f (x j ; ζ); φ\ζ)|| 2 2 2w 2 = λ • ||x j -g(f (x j ; ζ); φ\ζ)|| 2 2 + c (6)

where λ > 0 only depends on the bandwidth and c is a constant that does not depend on x j . The upper bound becomes tight as w → 0. Note that in the above derivation we omit the prior distribution

p(φ, ψ) = p 0 (φ, ψ)δ(φ(ζ) -ψ(ζ)

) by assuming that p 0 is a flat constant distribution. If other design choice is made, e.g., a Gaussian over φ(ψ), then there will be a corresponding 2 regularization term for the model parameters φ(ψ). Putting all together, maximizing a combination of conditional and marginal likelihoods correspond to the following unconstrained minimization problem:

minimize ψ,φ - m i=1 log p(y i | x i ; ψ) + λ n j=1 ||x j -g(f (x j ; ζ); φ\ζ)|| 2 2 (7)

Remark. The maximum of joint likelihood estimator is also the maximum-a-posteriori estimator, leading to a fully probabilistic interpretation of ( 7). The second term in the objective function has an interesting interpretation: it essentially measures the reconstruction error after a transformation by g • f . Based on ( 6), if we interpret f as an encoder and g as the corresponding decoder, then minimizing the reconstruction loss from an autoencoder exactly corresponds to maximizing a lower bound of the marginal probability distribution p(x), where p(x) is given by our kernel density estimation. Furthermore, the squared 2 measure in the reconstruction error is not restricted: for example, if instead of a Gaussian kernel we chose the Laplacian kernel, then we would have an 1 measure of the reconstruction loss in ( 7). This interpretation may also be used to explain the practical success of autoencoders in semi-supervised learning (Rasmus et al., 2015) . The model contains three different objectives: the domain classification loss, reconstruction loss as well as the label prediction loss. In the inference phase, only f and h will be used.

3.4. A CONCRETE MODEL (DAUTO):

We use neural networks as flexible function approximators for our desired transformations f and g. Specifically, we use fully-connected neural networks to parametrize f and g and softmax function to parametrize h. If y ∈ R, we can simply change the softmax function to be an affine function as the output. For the simplicity of discussion, assume we only use a one-layer fully connected network to represent f and g: f

(x) = σ(W 1 x) and g(z) = σ(W 2 z), where W 1 ∈ R D×d , W 2 ∈ R d×D and σ(•)

is an element-wise nonlinear activation function, e.g., the rectify linear unit. For ease of notation, we also omit the translation terms from above affine transformations. Let h(z) = softmax(W h z) be the softmax layer to compute the conditional probability of class assignment.

Although our model has the capacity to learn the shared transformation f under which unlabeled data from both domains have similar marginal distributions, the objective function discussed so far does not necessarily induce such a transformation. For the purpose of domain adaptation, it is necessary to add a regularizer that enforces this constraint. One popular and effective choice is the A-distance introduced by (Kifer et al., 2004; Ben-David et al., 2007; 2010) . It can be shown that the A-distance can be approximated by the binary classification error of the domain classifier that discriminates instances from the source or the target domain (Ben-David et al., 2007) . The intuition here is: given a fixed class of binary labeling functions, if there exists a function that is easy to tell instances in the source domain from those in the target domain, then the distance between these two domains is large.

On the other hand, if the set of labeling functions are confused by such task, then we can think the distance between these two domains is small. Following DANN, here we take the same approach: let h = softmax(W d z) be the domain classifier where z = σ(W 1 x) is the shared representation constructed by encoder f . The regularizer takes the form as a convex surrogate loss for the binary 0-1 error. A common choice is the cross-entropy loss. Putting all together, the optimization problem of our joint model is given by:

min W1,W2,W h max W d m i=1 L y (x i , y i ; W 1 , W h ) + λ n j=1 L r (x j ; W 1 , W 2 ) -µ n j=1 L d (x j ; W 1 , W d ) (8)

where L y (•, •) is the prediction loss, L r (•) is the reconstruction loss and L d (•) is the domain classification loss. We illustrate the model architecture in Fig. 2 . As we show above, the cross-entropy loss for the learning task and the reconstruction loss are essentially the negative joint log-likelihoods of both labeled instances and unlabeled instances. The domain classification loss works as a regularizer to incorporate our prior knowledge that the encoder f should form an invariant transformation. As a result, DAuto is designed to achieve the following three objectives simultaneously in a unified framework: 1). It learns representations that are informative for the main learning task in the source domain. 2). It learns robust representations under reconstruction loss. 3). It learns domain-invariant features that are indistinguishable between the source and the target domains.

4. EXPERIMENTS:

We first evaluate DAuto on synthetic experiments with MNIST, and then compare it with state-of-theart models, including mSDA, the Ladder network, DANN and ADDA. We report experimental results on the Amazon benchmark dataset, three digit datasets (MNIST, SVHN and USPS) and a large-scale time-series dataset for speech recognition.

4.1. DATASETS AND EXPERIMENTAL SETUP:

Synthetic experiment with MNIST. The experiment contains 4 tasks: each task is a binary classification problem on judging whether the given image is a specific number or not. We choose 4 digits for these tasks: 3, 7, 8 and 9, because 9 and 7 are similar in many handwritten images while 3 and 9 (7 and 8) are quite different. Clearly domain adaptation is not always possible. We would like to verify that DAuto succeeds when two domains are close to each other, and also show a failure case when domains are sufficiently different. To do so, for each task on recognizing digit i ∈ {3, 7, 8, 9}, we sample 1000 images from the training set, of which 500 are digit i and the others are digits not in {3, 7, 8, 9}; we sample ∼1,500 images from the original test set, of which ∼750 images are digit i and the others are digits not in {3, 7, 8, 9}. There are 16 pairs of experiments altogether for each possible pair of i, j ∈ {3, 7, 8, 9} as source and target domains. We design a well-controlled experiment to compare DAuto with a standard multilayer perceptron (MLP) and DANN: all algorithms share the same network structure. Also, we apply the same training procedure to all algorithms so that the difference in performance can only be explained by the additional domain regularizer as well as the reconstruction loss in DAuto.

Multi-class Classification. DAuto can be easily applied in multi-class classification as well. To see this, we test our model on MNIST, USPS and SVHN, all of which contain images of 10 digits.

MNIST contains 60,000/10,000 train/test instances; USPS contains 7,291/2,007 train/test instances, and SVHN contains 73,257/26,032 train/test instances. Before training, we preprocess all the instances into gray scale single-channel images of size 16x16, so that they can be used by the same network. Again, we perform a controlled experiment to ensure a fair comparison between the evaluated models. The network structures are exactly the same for all the approaches: 2 hidden layers with 1024, 512 units, following by a softmax output layer with 10 units. The domain discriminators in all the models are chosen to be logistic regression. During training, both batch size (400) and dropout rate (0.3) are also fixed to be same among all the experiments. Hence again, the difference in performance can only be explained by the different objective functions of different models.

Sentiment Analysis. The Amazon dataset consists of reviews of products on Amazon (Blitzer et al., 2007) . The task is to predict the polarity of a text review, i.e., whether the review for a specific product is positive or negative. The dataset contains text reviews for the following four categories: books (B), DVDs (D), electronics (E), and kitchen appliances (K). Each of the product contains 2000 text reviews as training data, and 4465 (B), 3586 (D), 5681 (E), 5945 (K) reviews as test data. Each text review is described by a feature vector of 5000 dimensions, where each dimension correspond to a word in the dictionary. The dataset is a benchmark data that has been frequently used for the purpose of sentiment analysis (Blitzer et al., 2006; 2007; Chen et al., 2012; Ajakan et al., 2014; Ganin & Lempitsky, 2015; Long et al., 2016) . For each source-target pair, we train the corresponding models completely on labeled source instances with access to unlabeled target instances. We use the classification accuracy in the target domain as our main metric.

Speech recognition. DAuto can be naturally extended to time-series modeling. In this experiment we apply DAuto to speech recognition, where a recurrent neural network is trained to ingest speech spectrograms and generate text transcripts. The model we use is a variant of DeepSpeech 2 (Amodei et al., 2015) , which is composed of 1 convolution layer followed by 3 stacked bidirectional LSTM layers, and one fully connected layer before a softmax layer. At each time step, the input to the network is a log spectrogram feature, and the output is a character or blank. The network is trained end-to-end using the connectionist temporal classification (CTC) loss (Graves et al., 2006) , which is the negative log-likelihood of training utterances. To extend DAuto in sequential models, besides the global CTC loss, we regularize it at each time step with both the reconstruction loss from the autoencoder as well as the adversarial loss from the domain classifier. We evaluate DAuto and compare it with other algorithms for an adaptation task across three different accented datasets, of which one is recorded from native English speakers, and the other two are recorded from speakers with Mandarin and Indian accents, respectively. Each dataset contains ∼33 hours of labeled audio data from ∼25,000 user utterances, and we randomly sample 80 percent of them as training set and the rest are used as test set.

We compare DAuto with the following methods: 1. No-Adapt. This is the baseline model which ignores the possible shifts between domains. In order to make the comparison as fair as possible, in all our experiments DAuto shares the same prediction model with No-Adapt. 2. mSDA. mSDA pretrains all the unlabeled instances from both the source and the target domains to build a feature map for the input space. The constructed representations from mSDA are used to train a SVM classifier as suggested in the original paper (Chen et al., 2012) . In all the experiments, we set the corruption level to be 0.5 in training mSDA, and stack the same number of layers of autoencoders as in DAuto. 3. Ladder Network (Ladder). The Ladder network (Rasmus et al., 2015) is a novel structure aiming for semi-supervised learning. It is a hierarchical denoising autoencoder where reconstruction errors between each pair of hidden layers are incorporated into the objective function. 4. DANN and 5.

ADDA. Again, we use exactly the same inference structure for DANN and ADDA as in No-Adapt, Ladder, and DAuto. For all the experiments, we use early-stop to avoid overfitting. We implement all the models and ensure that all the preprocessing for data are the same for all the algorithms, so that the differences in experimental results can only be explained by the differences in models themselves. We defer detailed description about models used in each experiment to the supplementary material.

4.2. RESULTS AND ANALYSIS:

MNIST. We list the classification accuracies on 16 pairs of tasks using No-Adapt, DANN and DAuto in Table 1 . Besides the 12 pairs of tasks under the domain adaptation setting, we also show 4 additional tasks where both the training and test sets are from the same domain. The scores from these 4 tasks can be used as empirical upper bounds to compare with the performance of domain On the other hand, we would also like to highlight that not all domain adaptation tasks are successful: the prediction accuracies of all the three algorithms on task 7 → 8 and 7 → 3 are only marginally above random guesses (0.5). When the digits are similar, we observe very successful domain transfer using DAuto even the algorithm does not see any instances from the target category during training.

To qualitatively study both the successful and failure cases, we project both representations with and without DAuto adaptation onto 2 dimensional space using PCA, shown in Fig. 3 . Several interesting observations can be made from Fig. 3 : when domain adaptation is successful (Fig. 3g, 3f ), the principal directions of learned representations from both domains are well-aligned with each other; on the other hand, when adaptation fails (Fig. 3e ), representations do not share the same principal directions. As a special case, when the source and target domains share the same distribution (Fig. 3h ), DAuto still works and will not degrade.

Multi-class Classification. The results on multi-class classification of digits are shown in Table 2 , where we highlight the successful domain adaptations using green colors and failure cases using red colors. The datasets in row correspond to the source domains and those in columns correspond to the target domains. DANN, ADDA and DAuto all contain one failure case (USPS→SVHN, USPS→SVHN and MNIST→USPS), which may be explained by the intrinsic difference between the SVHN datasets and the other two. However, on those three datasets, whenever both DANN and DAuto succeed in domain adaptation, DAuto usually outperforms DANN by around 2 percent accuracy. On the other hand, ADDA achieves far better result on MNIST→USPS than both DANN and DAuto, while it also performs worse than DAuto on SVHN→MNIST and USPS→MNIST. Note that in this experiment all the methods share exactly the same experimental protocol, hence the difference can only be explained by their different objective functions and model designs. In other words, the adaptation can benefit from the reconstruction error from autoencoders, which works as an unsupervised regularizer.

Amazon. To show the effectiveness of different domain adaptation algorithms when labeled instances are scarce, we evaluate the five algorithms on the 16 tasks by gradually increasing the size of the labeled training instances, but still use the whole test dataset to measure the performance. More specifically, we use 0.2, 0.5, 0.8 and 1.0 fraction of the available labeled instances from source domain during training. A successful domain adaptation algorithm should be able to take advantage of the unlabeled instances from the target domain to help generalization even when the amount of labeled instances available is small. We plot the results in Fig. 4 : all the domain adaptation algorithms are able to use the unlabeled instances from the target domain to help generalization. However, the differences between the baseline MLP and DAuto are getting smaller with the increase of the size of training data. This phenomenon supports our probabilistic framework showing that DAuto can effectively use the unlabeled data. We report the detailed numeric scores achieved by each algorithm and their statistical significance test in the supplementary material. In summary, DAuto outperforms all the other competitors on 12 out of the 16 tasks, which is consistently better than its competitors.

Speech Recognition. To avoid possible external error during the decoding process, which is usually not stable on noisy data, we directly report the CTC loss obtained from different algorithms. We have 3 × 3 pairs of domain adaptation tasks, where each source/target domain ranges from speech with {IN, CN, US} accents. We show the results in Table 3 . Again, we observe decreases in performance when applying domain adaptation algorithms: the transfer between speech with Indian accents and the other two fails. However, when two domains are similar (US vs CN) where adaptation is possible, DAuto consistently outperforms DANN. We want to bring readers' attention that both DANN and DAuto improve over baseline method when training with unlabeled instances from the same domain (diagonal of the table ). This means both DANN and DAuto have the effect of semi-supervised learning algorithms: when the source and the target domains are indeed same, both DANN and DAuto benefit from using unlabeled instances. From the generative interpretation of our probabilistic framework, DAuto achieves this goal by maximizing the marginal probability of unlabeled instances, and due to the shared component, this further helps training the discriminative model.

5. CONCLUSION:

We propose a probabilistic framework that incorporates both generative and discriminative modeling in a principled way, which also helps us to interpolate between generative and discriminative extremes through a specific choice of prior distribution where a subset of model parameters is shared. The instantiated model, DAuto, allows us to use unlabeled instances from both domains in a principled way. This instantiation also shows that the empirical success of autoencoders in semi-supervised learning and domain adaptation can be explained as maximizing the marginal log-likelihoods of unlabeled data, where kernel density estimators are used to model the marginal distributions. This provides the first probabilistic justification for joint training with autoencoders in practice. Experimentally we show that DAuto can be successfully applied to domain adaptation problems, and has a natural extension to time series as well.

A MORE DETAILS ABOUT EXPERIMENTS A.1 MNIST SYNTHETIC EXPERIMENTS:

The networks we used in this experiment contains 3 hidden layers, each contain 500, 200 and 100 hidden units, respectively. The input layer contains 28 × 28 = 784 units, and the output layer is a single unit parametrized by a logistic regression model. Except the output layer, all the hidden layers use ReLU as the nonlinear activation function. The same network structure is used for both DANN and DAuto. During training, we use AdaDelta as the optimization algorithm for all the models. For DAuto, both hyperparameters λ and µ are chosen from the following range: λ, µ ∈ {10 -8 , 10 -7 , . . . , 10 2 }, where we further partition the target domain set as development set and the held-out test set, combined with early stopping for model selection. For all the experiments, we fix the learning rate to be 1.0.

A.2 AMAZON BENCHMARK DATASET MLP, Ladder, DANN and DAuto all share the same network structure: the input layer contains 5,000 units, followed by a fully-connected hidden layer with 500 units and the output layer, which is a logistic regression model. Correspondingly, mSDA pretrains a two-layer stacked auto-encoder, hence the feature representation pretrained by mSDA in this experiment has 10,000 dimensions. Ladder and DAuto also use unlabeled instances to pretrain model parameters in an purely unsupervised way.

Again, for all the neural network based models, we use ReLU as the nonlinear activation function and use AdaDelta as the optimization algorithm during training. For DAuto, both hyperparameters λ, µ ∈ {10 -4 , . . . , 10 2 }. Again, we fix the learning rate to be 1.0.

We report the classification accuracies on the test data of the 16 pairs of tasks to have a thorough comparisons among the 5 models in Table 4 . DAuto outperforms all the other competitors on 12 out of 16 tasks, whereas DANN achieves the best test set accuracy on D → D; Ladder scores highest on D → E and D → K; mSDA performs the best on E → B and E → D. We emphasize that DAuto performs consistently better than all the other competitors. Note that the only difference between DANN and DAuto is the autoencoder regularizer that forces the feature learning part to learn robust features, we conclude that DAuto successfully helps to build robust representations. To check whether the accuracy differences between those five methods are significant or not, we perform a paired t-test and report the two-sided p-value under the null hypothesis that two related

:



Figures and Tables:

Figure 2: Model architecture of DAuto and its computation flow during learning phase. DAuto contains three components: the autoencoder (f and g), the domain classifier ( h) and the label predictor (h). The autoencoder instantiates the marginal distribution p(x); the encoder f and the label predictor instantiates the conditional probability p(y | x). The domain classifier works as a regularizer.The model contains three different objectives: the domain classification loss, reconstruction loss as well as the label prediction loss. In the inference phase, only f and h will be used.

Figure 3: 2 dimensional PCA embeddings of learned representations with and without DAuto adaptation. All the representations are extracted from the last hidden layer of the networks. Blue and dots represent instances from the labeled training set and unlabeled target set, respectively.

Figure 4: Test set performances of MLP, Ladder, mSDA, DANN and DAuto with increasing training set sizes: from 0.2 to 1.0. DAuto achieves the best accuracy on 12 out of 16 tasks.

Classification accuracies on 16 tasks between No-Adapt, DANN and DAuto. S → T means that S is the source domain and T is the target domain. Improvements over baselines are highlighted.

Table content: <table><tr><td>Task</td><td colspan="3">No-Adapt DANN DAuto</td><td>Task</td><td colspan="3">No-Adapt DANN DAuto</td></tr><tr><td>9 → 9 9 → 7 9 → 8 9 → 3 8 → 9 8 → 7 8 → 8 8 → 3</td><td>0.967 0.883 0.595 0.532 0.535 0.591 0.965 0.697</td><td>0.967 0.911 0.599 0.539 0.547 0.650 0.963 0.656</td><td colspan="2">0.965 7 → 9 0.917 7 → 7 0.655 7 → 8 0.686 7 → 3 0.640 3 → 9 0.705 3 → 7 0.957 3 → 8 0.698 3 → 3</td><td>0.777 0.977 0.519 0.523 0.535 0.738 0.612 0.973</td><td>0.807 0.979 0.524 0.514 0.643 0.795 0.671 0.976</td><td>0.837 0.979 0.540 0.553 0.734 0.806 0.737 0.973</td></tr></table>

Classification accuracy on the digits experiment from No-Adapt, DANN, ADDA and DAuto. Improvements over baseline method are highlighted in green, and decreases in performance are shown in red.Table best viewed in color.

Table content: <table><tr><td/><td/><td>No Adapt</td><td/><td/><td>DANN</td><td/></tr><tr><td/><td colspan="2">SVHN MNIST</td><td>USPS</td><td colspan="2">SVHN MNIST</td><td>USPS</td></tr><tr><td>SVHN</td><td>0.8553</td><td>0.5459</td><td>0.5277</td><td>0.8596</td><td>0.5690</td><td>0.5426</td></tr><tr><td>MNIST</td><td>0.2054</td><td>0.9883</td><td>0.6442</td><td>0.2241</td><td>0.9880</td><td>0.6500</td></tr><tr><td>USPS</td><td>0.1628</td><td>0.3396</td><td>0.9507</td><td>0.1585</td><td>0.3562</td><td>0.9517</td></tr><tr><td/><td/><td>ADDA</td><td/><td/><td>DAuto</td><td/></tr><tr><td/><td colspan="2">SVHN MNIST</td><td>USPS</td><td colspan="2">SVHN MNIST</td><td>USPS</td></tr><tr><td>SVHN</td><td>0.8707</td><td>0.5542</td><td>0.5561</td><td>0.8626</td><td>0.5864</td><td>0.5655</td></tr><tr><td>MNIST</td><td>0.2091</td><td>0.9894</td><td>0.6856</td><td>0.2086</td><td>0.9869</td><td>0.6428</td></tr><tr><td>USPS</td><td>0.1602</td><td>0.3570</td><td>0.9512</td><td>0.1717</td><td>0.3762</td><td>0.9537</td></tr></table>

CTC loss on 9 tasks from LSTM (No-Adapt), DANN and DAuto. A lower loss indicates a better speech model. Improvements over baseline method are highlighted in green, and decreases in performance are shown in red. Table best viewed in color.

Table content: <table><tr><td colspan="3">LSTM (No Adapt)</td><td/><td>DANN</td><td/><td/><td>DAuto</td><td/></tr><tr><td>US</td><td>CN</td><td>IN</td><td>US</td><td>CN</td><td>IN</td><td>US</td><td>CN</td><td>IN</td></tr><tr><td colspan="9">US 263.7 160.4 408.9 189.3 112.4 428.9 185.9 97.9 486.1</td></tr><tr><td colspan="9">CN 226.6 110.9 375.4 186.8 66.4 453.0 160.7 45.7 494.8</td></tr><tr><td colspan="9">IN 389.7 245.5 376.5 498.4 429.1 244.7 493.0 470.3 241.3</td></tr></table>

Sentiment classification (positive/negative) accuracies on 16 tasks of 5 models: MLP, Ladder, mSDA, DANN and DAuto. S → T means that S is the source domain and T is the target domain.

Table content: <table><tr><td>Task</td><td colspan="4">MLP mSDA Ladder DANN DAuto</td></tr><tr><td colspan="2">B→B 0.823 0.812</td><td>0.817</td><td>0.828</td><td>0.834</td></tr><tr><td colspan="2">B→D 0.770 0.785</td><td>0.764</td><td>0.773</td><td>0.776</td></tr><tr><td colspan="2">B→E 0.728 0.734</td><td>0.735</td><td>0.734</td><td>0.749</td></tr><tr><td colspan="2">B→K 0.762 0.770</td><td>0.775</td><td>0.769</td><td>0.782</td></tr><tr><td colspan="2">D→B 0.768 0.769</td><td>0.757</td><td>0.765</td><td>0.776</td></tr><tr><td colspan="2">D→D 0.830 0.820</td><td>0.832</td><td>0.834</td><td>0.834</td></tr><tr><td colspan="2">D→E 0.753 0.768</td><td>0.784</td><td>0.776</td><td>0.784</td></tr><tr><td colspan="2">D→K 0.776 0.793</td><td>0.802</td><td>0.789</td><td>0.795</td></tr><tr><td colspan="2">E→B 0.693 0.726</td><td>0.683</td><td>0.693</td><td>0.707</td></tr><tr><td colspan="2">E→D 0.696 0.741</td><td>0.715</td><td>0.694</td><td>0.724</td></tr><tr><td colspan="2">E→E 0.850 0.857</td><td>0.854</td><td>0.847</td><td>0.864</td></tr><tr><td colspan="2">E→K 0.845 0.858</td><td>0.848</td><td>0.843</td><td>0.863</td></tr><tr><td colspan="2">K→B 0.687 0.715</td><td>0.694</td><td>0.710</td><td>0.723</td></tr><tr><td colspan="2">K→D 0.711 0.738</td><td>0.731</td><td>0.727</td><td>0.748</td></tr><tr><td colspan="2">K→E 0.838 0.825</td><td>0.843</td><td>0.838</td><td>0.849</td></tr><tr><td colspan="2">K→K 0.869 0.867</td><td>0.874</td><td>0.873</td><td>0.882</td></tr></table>

Peer review (iclr2018):
 This is a very well-written paper that shows how to successfully use (generative) autoencoders together with the (discriminative) domain adversarial neural network (DANN) of Ganin et al. The construction is simple but nicely backed by a probabilistic analysis of the domain adaptation problem. The only criticism that I have towards this analysis is that the concept of shared parameter between the discriminative and predictive model (denoted by zeta in the paper) disappear when it comes to designing the learning model. The authors perform numerous empirical experiments on several types of problems. They successfully show that using autoencoder can help to learn a good representation for discriminative domain adaptation tasks. On the downside, all these experiments concern predictive (discriminative) problems. Given the paper title, I would have expected some experiments in a generative context. Also, a comparison with the Generative Adversarial Networks of Goodfellow et al. (2014) would be a plus. I would also like to see the results obtained using DANN stacked on mSDA representations, as it is done in Ganin et al. (2016). Minor comments: - Paragraph below Equation 6: The meaning of is unclear - Equation (7): phi and psi seems inverted - Section 4: The acronym MLP is used but never defined. === update === I lowered my score and confidence, see my new post below.