Title: VARIATIONAL RECURRENT ADVERSARIAL DEEP DOMAIN ADAPTATION

Abstract: We study the problem of learning domain invariant representations for time series data while transferring the complex temporal latent dependencies between domains. Our model termed as Variational Recurrent Adversarial Deep Domain Adaptation (VRADA) is built atop a variational recurrent neural network (VRNN) and trains adversarially to capture complex temporal relationships that are domain-invariant. This is (as far as we know) the first to capture and transfer temporal latent dependencies of multivariate time-series data. Through experiments on real-world multivariate healthcare time-series datasets, we empirically demonstrate that learning temporal dependencies helps our model's ability to create domain-invariant representations, allowing our model to outperform current state-of-the-art deep domain adaptation approaches.
Many real-world applications require effective machine learning algorithms that can learn invariant representations across related time-series datasets. For example, precision medicine for patients of various age groups, mobile application recommendation for users based on locations, and so on.
In these examples, while the domains (i.e. age group and location) may vary, there exist common predictive patterns that can aid in inferring knowledge from one domain to another. More often than not, some domains have a significantly larger number of observations than others (e.g., respiratory failure in adults vs. children). Therefore effective domain adaption of time-series data is in great demand.
The general approach to tackling domain adaptation has been explored under many facets which include reducing the domain discrepancy between the source and target domains(Ben-David et al. (2007)), instance re-weighting (Jiang & Zhai (2007)), subspace alignment (Fernando et al. (2013)), and deep learning (Tzeng et al. (2015); Ganin & Lempitsky (2014)). Many of these approaches work very well for non-sequential data but are not suitable for multivariate time-series data as they do not usually capture the temporal dependencies present in the data. For sequential data, earlier work has successfully used dynamic Bayesian Networks(Huang & Yates (2009)) and Recurrent Neural Networks (Socher et al. ( 2011)) to learn latent feature representations which were domaininvariant. Unfortunately, these works were not flexible enough to model non-linear dynamics or did not explicitly capture and transfer the complex latent dependencies needed to perform domain adaptation of time-series data.
In this paper, we address this problem with a model that learns temporal latent dependencies (i.e. dependencies between the latent variables across timesteps) that can be transferred across domains that experience different distributions in their features. We draw inspiration from the Variational Recurrent Neural Network (Chung et al. (2016) ) and use variational methods to produce a latent representation that captures underlying temporal latent dependencies. Motivated by the theory of domain adaptation (Ben-David et al. ( 2010)), we perform adversarial training on this representation

:

t-SNE projections for the latent representations of DNN, R-DANN, and our VRADA model. We show adaption from Adult-AHRF to Child-AHRF data. Source data is represented with red circles and target data with blue circles. From left to right, one can see that domain adaptation results in mixing the source and target domain data distributions. We can also see a story of how encoding more temporal dependency into the latent representation induces more domain-invariant representations. As models capture more underlying factors of variation, post domain adaptation representations gradually smoothen and become evenly dispersed, indicating that temporal dependency acts synergestically with domain adaptation.

similarly to the Domain Adversarial Neural Network (DANN) (Ganin et al. (2016) ) to make the representations invariant across domains. We call our model the Variational Recurrent Adversarial Deep Domain Adaptation (VRADA) model. As far as we know, this is the first model capable of accomplishing unsupervised domain adaptation while transferring temporal latent dependencies for complex multivariate time-series data. Figure 1 shows an example of the domain invariant representations learned by different deep learning models including our VRADA model. From this figure, we can see that our model (VRADA) shows better mixing of the domain distributions than the competing models indicating that it learns better domain invariant representations.

In order to prove the efficacy of our model, we perform domain adaptation using real-world healthcare time-series data. We choose healthcare data for two primary reasons. (1) Currently, a standard protocol in healthcare is to build, evaluate, and deploy machine learning models for particular datasets that may perform poorly on unseen datasets with different distributions. For example, models built around patient data from particular age groups perform poorly on other age groups because the features used to train the models have different distributions across the groups (Alemayehu & Warner (2004) ; Lao et al. (2004) ; Seshamani & Gray (2004) ). Knowledge learned from one group is not transferrable to the other group. Domain adaptation seems like a natural solution to this problem as knowledge needs to be transferred across domains which share features that exhibit different distributions.

(2) Healthcare data has multiple attributes recorded per patient visit, and it is longitudinal and episodic in nature. Thus, healthcare data is a suitable platform on which to study a model which seeks to capture complex temporal representations and transfer this knowledge across domains.

The rest of the paper is structured as follows. In the following section, we briefly discuss the current state-of-the-art deep domain adaptation approaches. Afterwards, we present our model mathematically, detailing how it simultaneously learns to capture temporal latent dependencies and create domain-invariant representations. In Section 4, we compare and contrast the performance of proposed approach with other approaches on two real-world health care datasets, and provide analysis on our domain-invariant representations.

2. RELATED WORK:

Domain adaptation is a specific instance of transfer learning in which the feature spaces are shared but their marginal distributions are different. A good survey on the two has been done in several previous works (Pan & Yang (2009) ; Jiang (2008); Patel et al. (2015) ). Domain adaptation has been thoroughly studied in computer vision (Saenko et al. (2010) ; Gong et al. (2012) ; Fernando et al. (2013) ) and natural language processing (NLP) (Blitzer (2007) 2016)) at each time-step to learn a complex relationship between the latent hidden factors across time-steps. Like the VAE, its latent variable is parametric. Combined, these things make it well-suited for multimodal sequential data such as multivariate time-series. In the following section, we discuss our approach, Variational Adversarial Deep Domain Adaptation (VRADA), which uses a VRNN to model and transfer complex domain-invariant temporal latent relationships for unsupervised domain adaptation of multivariate time-series.

3. VARIATIONAL RECURRENT ADVERSARIAL DEEP DOMAIN ADAPTATION:

In this section, we present our Variational Recurrent Adversarial Deep Domain Adaptation (VRADA) model for the purpose of capturing and transferring temporal latent dependencies across domains via domain-invariant representations. First, we introduce the notations used in this paper and then discuss our VRADA model in detail.

3.1. NOTATIONS:

Let us denote a multivariate variable-length time series with N data samples as {x i = (x i t ) T i t=1 } N i=1 , where x i t ∈ R D . (Note: in our experiments, for all data samples T i = τ , but for generality we maintain T i ). We denote {x i S } n i=1 as source domain data and {x i T } N i=n+1 as target domain data. We assume that each source domain data sample x i S comes with L labels y i ∈ {0, 1} L (for example, these labels may correspond to a clinical outcome such as mortality or ICD9 diagnosis codes), while target domain has no labeled data samples. We assign a domain label d i ∈ {0, 1} to each data sample to indicate if it comes from the source or target domain. d i will be used for adversarial training.

3.2. VRADA:

The block diagram of our VRADA model is shown in Figure 2 . To explicitly model the dependencies between the latent random variable across time steps, the VRADA model utilizes Variational Recurrent Neural Networks (VRNN) (Chung et al. (2016) ). The VRNN effectively contains a Variational Auto-Encoders (Kingma & Welling (2013) ) at every time step, all of which are conditioned on previous auto-encoders via the hidden state h t-1 of an RNN, such as an LSTM (Hochreiter & Schmidhuber (1997) ). Therefore, for each time-step of x i t , we infer a latent random variable z i t via

z i t |x i t ∼ N (µ z,t , diag(σ z,t )), where [µ z,t , σ z,t ] = ϕ enc τ (ϕ x τ (x i t ), h t-1 ) with prior z i t ∼ N (µ 0,t , diag(σ 0,t )), where [µ 0,t , σ 0,t ] = ϕ prior τ (h t-1 )

where µ * ,t , σ * ,t denote parameters of a generating distribution, and ϕ * τ can be any highly flexible function such as deep neural networks. For each z i t , x i t is generated via

x i t |z i t ∼ N (µ x,t , diag(σ x,t )), where [µ x,t , σ x,t ] = ϕ dec τ (ϕ z τ (z i t ), h t-1

) and learned by optimizing the VRNN objective function:

L r (x i t ; θ e , θ g ) = E q θe (z i ≤T i |x i ≤T i ) [ T i t=1 (-D(q θe (z i t |x i ≤t , z i <t )||p(z i t |x i <t , z i <t ))+log p θg (x i t |z i ≤t , x i <t ))])

where

q θe (z i t |x i ≤t , z i <t ) is the inference model, p(z i t |x i <t , z i <t ) is the prior, p θg (x i t |z i ≤t , x i <t )

is the generative model, θ e is the parameters of the VRNN's encoder, θ g the parameters of the VRNN's decoder, and D(•||•) refers to KL-Divergence. Note: z ≤T refers to the set of all z t such that t ≤ T , likewise for z <T . For each x i , we use zi ∼ q θe (z i T i |x i ≤T i , z i <T i ) as our feature representation for source domain classification task since it captures temporal latent dependencies across the time-steps. Training the VRNN for the source domain classification involves solving the following optimization: min θe,θg,θy

EQUATION

where R(θ e ) is a regularizer for the parameters of VRNN encoder (which is also the feature extractor of VRADA) with a tuning hyperparameter λ.

As we are interested in achieving domain adaptation via the latent representation zi (i.e. to make zi domain-invariant), we can adversarially train the above objective function (equation 1) by employing the domain adaptation idea proposed in Ganin et al. (2016) . Let G y (z i ; θ y ) and G d (z i ; θ d ) represent the source label classifier (to predict source labels y i ) and domain label classifier (to predict domain labels d i ) respectively with parameters θ y and θ d for a given input zi . Here, G y (.) and G d (.) can be deep neural networks. Let us denote their loss functions respectively as

L y (x i ; θ y , θ e ) = L B (G y (V e (x i ; θ e ); θ y ), y i ); L d (x i ; θ d , θ e ) = L B (G d (V e (x i ; θ e ); θ d ), d i )

where L B is the classification loss such as a binary or categorical cross-entropy loss function and V e (x i ; θ e ) is the VRNN encoder that maps input x i to zi . Now, for adversarial training, we consider the following domain adaptation term as the regularizer of equation 1.

EQUATION

)

where n is the number of target domain samples. As shown in Ganin et al. (2016) , R is the domain regularizer and it is derived from the empirical H-divergence between the source domain and target domain samples ( Ben-David et al. (2010) ).

Combining the joint optimization problem of equations 1 and 2 leads to our VRADA model, where we minimize the source classification risk and at the same time achieve domain adaptation. Mathematically, we optimize the following complete objective function:

EQUATION

where λ is a trade-off between optimizing on making domain-invariant representations and optimizing source classification accuracy. Our optimization involves minimization with respect to some parameters, and maximization with respect to the others, i.e., we iteratively solve the following:

( θg , θy , θe ) = arg min θg,θy,θe

E(θ e , θ g , θ y , θd ) θd = arg max θ d E( θe , θg , θy , θ d )

with the gradient updates calculated as:

θ e ← θ e -η( ∂Lr ∂θe + ∂Ly ∂θy -λ ∂L d ∂θ d ) (4) θ g ← θ g -η ∂Lr ∂θg (5) θ d ← θ d -η ∂L d ∂θ d (6) θ y ← θ y -ηλ ∂Ly ∂θy (7)

where η is the learning rate. We can use stochastic gradient descent (SGD) to solve the equations (5-7). To solve equation ( 4), we can use SGD and the gradient reversal layer (GRL) (Ganin et al. (2016) ). The role of GRL is to reverse the gradient sign while performing backpropagation. This ensures that the domain classification loss is maximized which makes the feature representations domain-invariant.

Thus, VRADA results in learning feature representations which are domain-invariant (due to domain regressor R) and which capture the temporal latent dependencies (due to optimizing VRNN objective function L r ). These things combine to allow the VRADAs' discriminative power on the source domain to transfer to the target domain.

4. EXPERIMENTS:

We conduct experiments on two real-world health care datasets to answer the following questions: (a) How does our VRADA model perform when compared to the state-of-the-art domain adaptation and non-adaptation approaches? (b) How different are the domain-invariant representations learned by various domain adaptation methods? (c) How do we show that the temporal latent dependencies are transferred between domains? In the remainder of this section, we will describe the datasets, methods, empirical results, and show visualizations to answer the above questions.

4.1. DATASET DESCRIPTION:

We conduct experiments on two health care datasets, including the MIMIC-III dataset and a Pediatric ICU (PICU) dataset from Children's Hospital Los Angeles. 

MIMIC-III(:



4.1.1. PREDICTION AND DOMAIN ADAPTATION TASKS:

Mortality Prediction: For Adult-AHRF and Child-AHRF datasets, we are interested in predicting mortality, i.e. whether a patient dies from AHRF during their hospital stay. 20.10% of all the patients in Child-AHRF and 13.84% of all patients in Adult-AHRF have a positive mortality label (i.e. the patients who die in hospital).

ICD9 Code Prediction: Each admission record in MIMIC-III dataset has multiple ICD-9 diagnosis codes. We group all the occurrences of the ICD-9 codes into 20 diagnosis groups [2] . For the ICD9 dataset, we are interested in predicting these 20 ICD-9 Diagnosis Categories for each admission record. We treat this as a multi-task prediction problem.

Domain Adaptation Tasks::

We study unsupervised domain adaptation (i.e. target domain labels are unavailable during training and validation) task with-in age groups of Adult-AHRF dataset, ICD9 dataset and across Adult and Child-AHRF datasets. For Adult-AHRF and ICD9 datasets, we created 12 source-target domain pairs using the age groups, pairing up each domain D i with another domain D j =i , for example, the source-target pair 2-5 was used for adapting from group 2 (working-age adult) to group 5 (old elderly). We also created 4 source-target pairs for performing domain adaptation from 4 adult age-groups to 1 child age-group.

4.2. METHODS AND IMPLEMENTATION DETAILS:

We categorize the methods used in our main experiments into the following groups:

• • Our method: Variational Recurrent Adversarial Deep Domain Adaptation (VRADA) [3] .

In all our experiments, we conducted unsupervised domain adaptation where target domain labels are unavailable during training and validation. For R-DANN, we used LSTM (Hochreiter & Schmidhuber (1997) ) as the feature extractor network instead of the feed-forward neural networks used in DANN.

For VFAE, DANN and all the non-domain adaptive approaches we flattened the time series along time axis and treat it as the input to the model. For fairness, the classifier and feature extractors of the VRADA and R-DANN were equivalent in depth and both had the same model capacity. We also ensure that the size of latent feature representation zi are similar for VRADA and DANN models. The model capacity of VFAE was chosen to be similar to VRADA. All the deep domain adaptation models including ours had depth of size 8 (including output classifier layers). We used the Adam optimizer ( Kingma & Ba ( 2014)) and ran all models for 500 epochs with a learning rate of 3e-4.

We set an early stopping criteria that the model does not experience a decrease in the validation loss for 20 epochs. Source domain data was split into train/validation subsets with a 70/30 ratio and target domain data into train/validation/test subsets with a 70/15/15 ratio. In order to compare all the methods, we report AUC scores on the entire target domain set, and the test subset for each target domain data of a source-target pair.

4.3. QUANTITATIVE RESULTS:

In Table 1 , we compare non domain adaptation and domain adaptation models' performance on the target domain test subset for the AHRF mortality prediction task. It is immediately clear that domain adaptation methods consistently outperform non domain adaptation methods. We see that generally the VRADA outperforms both variants of the DANN with it consistently seeing scores ∼ 4% higher. While the standard deviation for the VRADA was about 1%, it was about 2% for the R-DANN, further showing our models efficacy as it converges to more stable local optima. Our model VRADA beats state-of-the-art DANN (Ganin et al. (2016) ) and VFAE (Louizos et al. (2015) ) on all the source-pair domain adaptation tasks for Adult-AHRF dataset. For the domain adaptation from Adult-AHRF to Child-AHRF dataset, we observe that VRADA mostly outperforms all the competing models. This shows that our model can perform well even for smaller target domain datasets. As the AHRF mortality prediction task made it clear that domain adaptation is necessary for intergroup adaptation, for the ICD9 multi-task prediction task that involved data with time-steps of length 12, we focused strictly on domain adaptive models (i.e. the DANN, R-DANN, and VRADA). Table 2 shows the aggregated AUC scores on the entire target domain dataset and test data of the target domain for the 20 tasks of the ICD9 Code Prediction task. Here, we clearly see that VRADA and R-DANN models outperform DANN Ganin et al. (2016) by significant margins. We also observe that VRADA outperforms R-DANN by 1.5 ∼ 2% when averaged over all the source-target domain pairs.

4.4. DISCUSSION:

Figure 3 shows the temporal latent dependencies captured by our VRADA as compared to the R-DANN for 3-4 source-target pair. While both models learn temporal latent dependencies fairly well, the VRADA outperforms the R-DANN in two ways. First, the VRADA's neurons learned stronger predictions of whether features are relevant towards modeling the data. If we look at the VRADA row, for both AHRF and ICD9 we see that the neural activation patterns are more consistent across time-steps than for R-DANN. Figure 4 shows the unrolled memory cell states (in the form Examples × (Time * Neurons)) for all the source and target domain data points. We see a consistent activation firing patterns across all these data points for VRADA but not for R-DANN. Together with the stronger performance on 3-4 for AHRF and 2-5 for ICD9, this potentially indicates that VRADA is better learning the temporal dependencies.

Second, nuanced values are consistent across time-steps for the VRADA, exhibiting a gradual transition towards stronger activation with time, whereas the temporal activation pattern of the R-DANN seems somewhat sporadic. While activation gradients across time are consistent for both the R-DANN and VRADA, more consistent inhibitory and excitatory neuron firing patterns indicate that the VRADA better transfers knowledge. Another indication of domain adaptation was shown in Figure 1c . Looking at the t-SNE projections of feature representations of DNN, R-DANN, and VRADA we can see that the addition of temporal latent dependencies might help in better mixing of the domain distributions since we observe that the data is more evenly spread out. Figure 1c and Figure 3 together indicate that the VRADA's temporal latent dependency capturing power and ability to create domain-invariant representations act synergistically. For plots of activation patterns without domain adaptation, please see appendix section 6.2.3.

5. SUMMARY:

Because of its diverse range of patients and its episodic and longitudal nature, healthcare data provides a good platform to test domain adaptation techniques for temporal data. With it as our example, we showcase the Variational Recurrent Adversarial Domain Adaptation (VRADA) model's ability to learn temporal latent representations that are domain-invariant. By comparing our model's latent representations to others', we show its ability to use variational methods to capture hidden factors of variation and produce more robust domain-invariant representations. We hope this work serves as a bedrock for future work capturing and adapting temporal latent representations across domains. L y (x i ; θ y ) -λ(

EQUATION

and (c) loading a pretrained VRNN encoder and using the objective as presented in equation 3 (denoted by III). Key to note is that in method II, we do not apply variational methods towards learning the shared latent representation. This was done to test whether they were helpful or harmful towards the learned latent representation used for classification. In method III, we train VRADA as normal but load a pretrained encoder. We pretrain the encoder by training the VRNN on all source and target domain samples for a desired source-target adaptation pair. In order to choose how many samples would be used for training, we looked at which domain had more examples and chose the larger of the two. For example, if the source domain was group 2 with 508 patients and the target domain was group 5 with 437 patients, the VRNN would see 508 samples of each domain, with group 5 being sampled with replacement after seeing all its samples. As the encoder was used for learning latent representations, we thought it worth investigating whether if pretrained it better captured the latent representations that were being used by the domain classifier for adversarial training. We thought beginning domain classification at a better initialization point might help VRADA avoid local minima. For each method, we fed one source domain sample to G y and either a source or target domain sample to G d . (For this training and all training samples, order was randomized.) We only calculated the loss L r once for the G d samples so as to not bias the optimization of the VRNN. 4 ) and found the results to be sub-optimal when compared to only performing adversarial training at the last time-step (Table 1 ). Below are results for the R-DANN and VRADA models for adversarial training at every time-step. DANN .651 .599 .598 .557 .679 .534 .563 .768 .588 .528 .696 .669 VRADA .681 .691 .643 .594 .733 .641 .733 .794 .675 .583 .755 .726 

6.2.2. EFFECT OF RECONSTRUCTION LOSS:

Table 5 shows the effect of reconstruction loss for our VRADA model. We observe that reconstructing the original data (i.e. using the decoder for reconstructing the data) helps in the overall performance improvement of our VRADA model. In figures 5 and 6 we show the cell state activations for the VRADA and R-DANN without domain adaptation (i.e. no adversarial training). From these figures, we see that the dependencies between source and target domains are not transferred correctly since we do not perform adversarial training. On the otherhand, as discussed in section 4.4, figure 3 shows that adversarial training helps in transferring the dependencies between source and target domains efficiently.

6.3. R-DANN MODEL INFORMATION:

Here we provide more details on the network architectures of the R-DANN and DANN. Please refer to Figure 7 Step along the x-axis refers to activation per time-step. The figure shows a single example in adapting 2-5 for ICD9 dataset.



Figures and Tables:

Figure 1: A Story of Temporal Dependency and Domain Invariance (a) DNN (b) R-DANN (c) VRADA

Non-adaptive baseline methods: Logistic Regression (LR), Adaboost with decision regressors (Adaboost), and feed forward deep neural networks (DNN) • Deep Domain adaptation methods: Domain Adversarial Neural Networks (DANN) (Ganin et al. (2016)); DANN with a RNN (LSTM) as feature extractor (R-DANN); Variational Fair Autocoder (VFAE)(Louizos et al. (2015))

Figure 3: Cell states of memory cell for R-DANN and VRADA showing temporal latent dependencies captured by neurons of the R-DANN and VRADA for the source domain and transferred to the target domain. Each step along the y-axis refers to the activation of a single neuron with blue for strong inhibition and yellow for strong excitation. Step along the x-axis refers to activation per time-step. The left shows a single example in adapting 3-4 and the right for adapting 2-5.

of training VRADA: (a) training VRADA regularly as discussed in Section 3 (denoted by I), (b) loading a pretrained VRNN encoder and optimizing strictly off the classification errors, i.e. E(θ e , θ y , θ d ) = 1 n n i=1

for a diagram of the R-DANN model showing the dimensions of each layer and the connections between layers. The R-DANN and DANN were essentially identical except that, for the DANN, the first layer used a fully-connected layer instead of an RNN and took input flattened over the time-dimension. Thus the input dimensions corresponded to f and t × f for the R-DANN and DANN, respectively, where f is the number of features and t is the length of the time-dimension.

Figure 5: Cell states of memory cell for R-DANN and VRADA showing temporal latent dependencies captured by neurons of the R-DANN and VRADA for the source domain and the target domain. Each step along the y-axis refers to the activation of a single neuron with blue for strong inhibition and yellow for strong excitation. Step along the x-axis refers to activation per time-step. The figure shows a single example in adapting 3-4 for AHRF dataset.

Figure 6: Cell states of memory cell for R-DANN and VRADA showing temporal latent dependencies captured by neurons of the R-DANN and VRADA for the source domain and the target domain. Each step along the y-axis refers to the activation of a single neuron with blue for strong inhibition and yellow for strong excitation. Step along the x-axis refers to activation per time-step. The figure shows a single example in adapting 2-5 for ICD9 dataset.

Block diagram of VRADA. Blue lines show the inference process, q θe (zt|x ≤t , z<t). Brown lines show the generation process, p θg (xt|z ≤t , x<t). Red lines show the recurrence process where ht is informed by ht-1, which is informed by zt-1 and xt-1. Black lines indicate classification.(Ganin et al. (2016)) is a deep domain adaptation model which uses two core components to create domain-invariant representations, a feature extractor that produces the data's latent representation,

Table content: <table><tr><td/><td/><td/><td>Gy</td><td>Gd</td></tr><tr><td>z1</td><td>z2</td><td>z3</td><td/><td>zt</td></tr><tr><td/><td/><td/><td>. . .</td><td/></tr><tr><td>h1</td><td>h2</td><td>h3</td><td>. . .</td><td>ht</td></tr><tr><td/><td/><td/><td>. . .</td><td/></tr><tr><td>x1</td><td>x2</td><td>x3</td><td/><td>xt</td></tr><tr><td colspan="5">Figure 2: data, as one can with DBNs, but</td></tr><tr><td colspan="5">not both. To overcome the challenges of DBNs and RNNs, Variational Recurrent Neural Network</td></tr><tr><td colspan="5">(VRNN)( Chung et al. (2016)) was proposed recently to capture the complex relationship between</td></tr><tr><td colspan="5">the underlying hidden factors of variation and the output variables at different time-steps. The VRNN</td></tr><tr><td colspan="5">uses Variational Autoencoders (VAEs)( Kingma &amp; Welling (2013); Goodfellow et al. (</td></tr></table>

We grouped the patients into 4 groups/cohorts based on their age[1] -Group 2: working-age adult (20 to 45 yrs, 508 patients); Group 3: old working-age adult (46 to 65 yrs, 1888 patients); Group 4: elderly (66 to 85 yrs, 2394 patients); Group 5: old elderly (85 yrs and up, 437 patients). We treated each group as a separate domain with which we could perform domain adaptation. For each patient, we used the first 4 day after admission (with each day serving as a single time-step) as time series data for training and testing our models.• ICD9 dataset: For this dataset we extracted 99 time series features from 19714 admission records from 4 modalities including input-events (fluids into patient, e.g., insulin), outputevents (fluids out of the patient, e.g., urine), lab-events (lab test results, e.g., blood pH values, platelet count, etc.) and prescription-events (drugs prescribed by doctors, e.g., aspirin, potassium chloride, etc.). These modalities are known to be extremely useful for monitoring ICU patients. All the time series are of more than 48 hours of duration, and only the first 24 hours (after admission) 2-hourly sampled time series data is used for training and testing our models. We use this dataset to predict the ICD9 Diagnosis code categories for each patient's admission record.Child-AHRF dataset: This is a PICU dataset which contains health records of 398 children patient with acute hypoxemic respiratory failure in the intensive care unit at Children's Hospital Los Angeles (CHLA)(Khemani et al. (2009)). Similar to Adult-AHRF, this dataset has 20 time series features collected for 4 days after ICU admission. This dataset is considered as one group (Group 1: children, age 0 to 19 yrs) and represents one domain.

Table content: <table/>

AUC Comparison for AHRF Mortality Prediction task with and without Domain Adaptation

Table content: <table><tr><td>Source-Target</td><td>LR</td><td colspan="6">Adaboost DNN DANN VFAE R-DANN VRADA</td></tr><tr><td>3-2</td><td>0.555</td><td>0.562</td><td>0.569</td><td>0.572</td><td>0.615</td><td>0.603</td><td>0.654</td></tr><tr><td>4-2</td><td>0.624</td><td>0.645</td><td>0.569</td><td>0.589</td><td>0.635</td><td>0.584</td><td>0.656</td></tr><tr><td>5-2</td><td>0.527</td><td>0.554</td><td>0.551</td><td>0.540</td><td>0.588</td><td>0.611</td><td>0.616</td></tr><tr><td>2-3</td><td>0.627</td><td>0.621</td><td>0.550</td><td>0.563</td><td>0.585</td><td>0.708</td><td>0.724</td></tr><tr><td>4-3</td><td>0.681</td><td>0.636</td><td>0.542</td><td>0.527</td><td>0.722</td><td>0.821</td><td>0.770</td></tr><tr><td>5-3</td><td>0.655</td><td>0.706</td><td>0.503</td><td>0.518</td><td>0.608</td><td>0.769</td><td>0.782</td></tr><tr><td>2-4</td><td>0.585</td><td>0.591</td><td>0.530</td><td>0.560</td><td>0.582</td><td>0.716</td><td>0.777</td></tr><tr><td>3-4</td><td>0.652</td><td>0.629</td><td>0.531</td><td>0.527</td><td>0.697</td><td>0.769</td><td>0.764</td></tr><tr><td>5-4</td><td>0.689</td><td>0.699</td><td>0.538</td><td>0.532</td><td>0.614</td><td>0.728</td><td>0.738</td></tr><tr><td>2-5</td><td>0.565</td><td>0.543</td><td>0.549</td><td>0.526</td><td>0.555</td><td>0.659</td><td>0.719</td></tr><tr><td>3-5</td><td>0.576</td><td>0.587</td><td>0.510</td><td>0.526</td><td>0.533</td><td>0.630</td><td>0.721</td></tr><tr><td>4-5</td><td>0.682</td><td>0.587</td><td>0.575</td><td>0.548</td><td>0.712</td><td>0.747</td><td>0.775</td></tr><tr><td>5-1</td><td>0.502</td><td>0.573</td><td>0.557</td><td>0.563</td><td>0.618</td><td>0.563</td><td>0.639</td></tr><tr><td>4-1</td><td>0.565</td><td>0.533</td><td>0.572</td><td>0.542</td><td>0.668</td><td>0.577</td><td>0.636</td></tr><tr><td>3-1</td><td>0.500</td><td>0.500</td><td>0.542</td><td>0.535</td><td>0.570</td><td>0.591</td><td>0.631</td></tr><tr><td>2-1</td><td>0.520</td><td>0.500</td><td>0.534</td><td>0.559</td><td>0.578</td><td>0.630</td><td>0.637</td></tr><tr><td colspan="8">In the above table, we test classification without adaptation using Logistic Regression (LR), Adaboost with</td></tr><tr><td colspan="8">decision tree classifiers and Feed forward Deep Neural Networks (DNN); and with adaptation using Deep</td></tr><tr><td colspan="8">Domain Adversarial Neural Networks (DANN), a DANN with an LSTM in its feature extractor (R-DANN),</td></tr><tr><td colspan="8">Variational Fair Autoencoder (VFAE) and our Variational Adversarial Domain Adaptation Model (VRADA). All</td></tr><tr><td colspan="4">results are reported on the target domain test subset dataset.</td><td/><td/><td/><td/></tr></table>

AUC Comparison for ICD9 Diagnosis Code Prediction taskHere, we compare results for the ICD9 Diagnosis Code Prediction task on the ICD9 dataset. For each model, the top row corresponds to the performance on the entire target domain dataset and the bottom row corresponds to performance on the test subset (15%) of the target domain dataset.

Table content: <table><tr><td>Model</td><td/><td>23</td><td>24</td><td>25</td><td>32</td><td>34</td><td>35</td><td>42</td><td>43</td><td>45</td><td>52</td><td>53</td><td>54</td></tr><tr><td>DANN</td><td>entire target target test</td><td>0.513 0.509</td><td>0.508 0.513</td><td>0.509 0.531</td><td>0.511 0.527</td><td>0.508 0.515</td><td>0.514 0.531</td><td>0.511 0.515</td><td>0.507 0.521</td><td>0.512 0.521</td><td>0.505 0.518</td><td>0.508 0.514</td><td>0.506 0.519</td></tr><tr><td>R-DANN</td><td>entire target target test</td><td>0.608 0.605</td><td>0.581 0.579</td><td>0.562 0.570</td><td>0.618 0.628</td><td>0.610 0.609</td><td>0.586 0.589</td><td>0.604 0.614</td><td>0.607 0.616</td><td>0.575 0.586</td><td>0.573 0.573</td><td>0.558 0.563</td><td>0.566 0.564</td></tr><tr><td>VRADA</td><td>entire target target test</td><td>0.620 0.609</td><td>0.564 0.563</td><td>0.557 0.560</td><td>0.611 0.620</td><td>0.617 0.617</td><td>0.580 0.580</td><td>0.598 0.606</td><td>0.615 0.623</td><td>0.588 0.594</td><td>0.571 0.576</td><td>0.582 0.581</td><td>0.576 0.576</td></tr></table>

AUC Comparison for AHRF Mortality Prediction task for different types of VRADA training

Table content: <table><tr><td colspan="2">Training 23</td><td>24</td><td>25</td><td>32</td><td>34</td><td>35</td><td>42</td><td>43</td><td>45</td><td>52</td><td>53</td><td>54</td></tr><tr><td>I</td><td colspan="12">0.704 0.777 0.682 0.540 0.764 0.721 0.603 0.727 0.710 0.616 0.782 0.738</td></tr><tr><td>II</td><td colspan="12">0.724 0.656 0.719 0.627 0.748 0.683 0.656 0.770 0.755 0.595 0.736 0.732</td></tr><tr><td>III</td><td colspan="12">0.721 0.688 0.656 0.654 0.757 0.691 0.609 0.766 0.775 0.602 0.709 0.714</td></tr><tr><td colspan="12">Richard Socher, Cliff C Lin, Chris Manning, and Andrew Y Ng. Parsing natural scenes and natural</td></tr><tr><td colspan="12">language with recursive neural networks. In Proceedings of the 28th international conference on</td></tr><tr><td colspan="6">machine learning (ICML-11), pp. 129-136, 2011.</td><td/><td/><td/><td/><td/><td/></tr><tr><td colspan="12">Eric Tzeng, Judy Hoffman, Trevor Darrell, and Kate Saenko. Simultaneous deep transfer across</td></tr><tr><td colspan="12">domains and tasks. In Proceedings of the IEEE International Conference on Computer Vision, pp.</td></tr><tr><td colspan="3">4068-4076, 2015.</td><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td colspan="12">Min Xiao and Yuhong Guo. Domain adaptation for sequence labeling tasks with a probabilistic</td></tr><tr><td colspan="8">language adaptation model. In ICML (1), pp. 293-301, 2013.</td><td/><td/><td/><td/></tr><tr><td colspan="12">Yi Yang and Jacob Eisenstein. Unsupervised multi-domain adaptation with feature embeddings.</td></tr></table>

If done at every time-step, the network learns to create domain-invariant representations of subsets of your input x ≤T . Do these domain-invariant representations help the network find more optimal domain-invariant representations of x? We empirically tested this scenario (Table

Table content: <table><tr><td>shows the results of AHRF Mortality Prediction task for different types of VRADA training.</td></tr><tr><td>From these experiments, we found that jointly training VRADA (i.e method I) usually performed</td></tr><tr><td>better than the other pretrained training approaches.</td></tr><tr><td>6.2 MODEL VARIATIONS</td></tr><tr><td>6.2.1 ADVERSARIAL TRAINING AT EVERY TIME-STEP</td></tr></table>

AUC Comparison for AHRF Mortality Prediction task with adversarial training done at every time-step

Table content: <table><tr><td>Model</td><td>23 24 25 32 34 35 42 43 45 52 53 54</td></tr><tr><td>R-</td><td/></tr></table>

AUC Comparison of VRADA model for AHRF Mortality Prediction task with and without reconstruction loss .703 0.623 0.570 0.647 0.622 0.564 0.577 0.608 0.552 0.599 0.640 0.676 With reconstruction 0.724 0.777 0.719 0.654 0.764 0.721 0.656 0.770 0.775 0.616 0.782 0.738 6.2.3 IMPACT OF ADVERSARIAL TRAINING

Table content: <table><tr><td>Model</td><td>23</td><td>24</td><td>25</td><td>32</td><td>34</td><td>35</td><td>42</td><td>43</td><td>45</td><td>52</td><td>53</td><td>54</td></tr><tr><td>Without reconstruction 0</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr></table>

Peer review (iclr2017):
This paper combines variational RNN (VRNN) and domain adversarial networks (DANN) for domain adaptation in the sequence modelling domain. The VRNN is used to learn representations for sequential data, which is the hidden states of the last time step. The DANN is used to make the representations domain invariant, therefore achieving cross domain adaptation. Experiments are done on a number of data sets, and the proposed method (VRADA) outperforms baselines including DANN, VFAE and R-DANN on almost all of them. I don*t have questions about the proposed model, the model is quite clear and seems to be a simple combination of VRNN and DANN. But a few questions came up during the pre-review question phase: - As the authors have mentioned, DANN in general outperforms MMD based methods, however, the VFAE method which is based on MMD regularization on the representations seems to outperform DANN across the board. That seems to indicate VRNN + MMD should also be a good combination. - One baseline the authors showed in the experiments is R-DANN, which is an RNN version of DANN. There are two differences between R-DANN and VRADA: (1) R-DANN uses deterministic RNN for representation learning, while VRADA uses variational RNN; (2) on target domain R-DANN only optimizes adversarial loss, while VRADA optimizes both adversarial loss and reconstruction loss for feature learning. It would be good to analyze further where the performance gain comes from.