Title: Variational Recurrent Adversarial Deep Domain Adaptation
Authors: Sanjay Purushotham
Wilka Carvalho
Tanachat Nilanon
Yan Liu
Venue: ICLR 2017
Keywords: Deep learning
Transfer Learning

Abstract: We study the problem of learning domain invariant representations for time series data while transferring the complex temporal latent dependencies between the domains. Our model termed as Variational Recurrent Adversarial Deep Domain Adaptation (VRADA) is built atop a variational recurrent neural network (VRNN) and trains adversarially to capture complex temporal relationships that are domain-invariant. This is (as far as we know) the first to capture and transfer temporal latent dependencies in multivariate time-series data. Through experiments on real-world multivariate healthcare time-series datasets, we empirically demonstrate that learning temporal dependencies helps our model's ability to create domain-invariant representations, allowing our model to outperform current state-of-the-art deep domain adaptation approaches.

TL;DR: We propose Variational Recurrent Adversarial Deep Domain Adaptation approach to capture and transfer temporal latent dependencies in multivariate time-series data

Review discussion:
1. p. 7, 4.2, Could you provide more details for the baseline models (e.g. for DANN or R-DANN)? Diagrams with exact number filter maps/neurons and layer wiring would be perfect.
2. Have the authors tried simpler alternatives to adversarial training, e.g. MMD (reference: Long, 2015, ICML)?
3. Do you think your model would benefit from applying adversarial training at every time step (not only at the last one)? It would be interesting to see how your approach compares to R-DANN with a per-step adversary.
(Reply Updated) Thanks for your questions!
1. Yes, we will provide these details in the appendix of our revised draft.
2.  We have compared to Variational FAIR Autoencoder (VFAE) (Louizos, 2016, ICLR) which uses MMD to achieve domain invariance. We haven’t compared to the MMD approach presented in (Long, 2015, ICML). As demonstrated in DANN paper (Ganin, JMLR 2016), adversarial training generally outperforms MMD approaches (Long, 2015, ICML). Please note that as shown in Table 1, our proposed model outperforms both DANN and VFAE on many of the source-target pairs.
3. We have tried adversarial training at every time step and its results were not as good as applying adversarial training at the last time step. We will provide the results of adversarial training at every time step in the appendix of our revised draft.
Moreover, for healthcare problems, propagating the labels at every time step can be misleading for learning the model parameters.
Thus, in this paper, we have employed adversarial training at the last time step.
On unlabeled data, the R-DANN baseline seems to be only trained to fool the adversary, but in the proposed VRADA the RNN part is also trained to do reconstruction, as in the VRNN model.  Is the reconstruction part critical for the performance gain?
Thanks for this question.
In our preliminary experiments on a few source-target pairs, we did not observe that reconstruction part was critical for performance gain in VRADA. Now, we are conducting more experiments on all the source-target pairs to confirm our preliminary findings.
In equation (1), why does the VRNN part only use unlabeled data from source domain, but not those of target ?
Thanks for the question.
Reply Updated: There is no typo in equation (1). In equation (1), we show the optimization problem on the source domain data.
However, as shown in equation (3), our VRADA uses data from both the source and target domain for training the VRNN part of our model.
This paper combines variational RNN (VRNN) and domain adversarial networks (DANN) for domain adaptation in the sequence modelling domain.  The VRNN is used to learn representations for sequential data, which is the hidden states of the last time step.  The DANN is used to make the representations domain invariant, therefore achieving cross domain adaptation.
Experiments are done on a number of data sets, and the proposed method (VRADA) outperforms baselines including DANN, VFAE and R-DANN on almost all of them.
I don't have questions about the proposed model, the model is quite clear and seems to be a simple combination of VRNN and DANN.
But a few questions came up during the pre-review question phase:
- As the authors have mentioned, DANN in general outperforms MMD based methods, however, the VFAE method which is based on MMD regularization on the representations seems to outperform DANN across the board.  That seems to indicate VRNN + MMD should also be a good combination.
- One baseline the authors showed in the experiments is R-DANN, which is an RNN version of DANN.  There are two differences between R-DANN and VRADA: (1) R-DANN uses deterministic RNN for representation learning, while VRADA uses variational RNN; (2) on target domain R-DANN only optimizes adversarial loss, while VRADA optimizes both adversarial loss and reconstruction loss for feature learning.  It would be good to analyze further where the performance gain comes from.
Thanks for your comments and suggestions. We have discussed and highlighted our paper’s key novel contributions in the above 'Response to all reviewers' comment.
Q: As the authors have mentioned, DANN in general outperforms MMD based methods, however, the VFAE method which is based on MMD regularization on the representations seems to outperform DANN across the board.  That seems to indicate VRNN + MMD should also be a good combination.
A: We agree with the reviewer’s observation that VFAE performs better than DANN but does not always outperform R-DANN (For example: See Table 1, on many src-tgt pairs R-DANN outperforms VFAE). Thanks for the suggestion for VRNN+MMD approach. This approach is in our on-going works and we plan to include it in our future work.
Q: It would be good to analyze further where the performance gain comes from.
A: As per reviewer’s suggestions, we ran experiments on our VRADA without reconstruction loss. Results are shown in the appendix section 6.2.2. To our surprise, the performance did not change for a few source-target pairs but it did reduce for the many others. This indicates that reconstruction loss is important and might contribute to the performance improvement of our VRADA model. Appendix 6.2.2 empirically demonstrates that learning how to reconstruct the data helps VRADA learn structural and temporal dependencies present in the data.
Update: I thank the authors for their comments! After reading them, I still think the paper is not novel enough so I'm leaving the rating untouched.
This paper proposes a domain adaptation technique for time series. The core of the approach is a combination of variational recurrent neural networks and adversarial domain adaptation (at the last time step).
Pros:
1. The authors consider a very important application of domain adaptation.
2. The paper is well-written and relatively easy to read.
3. Solid empirical evaluation. The authors compare their method against several recent domain adaptation techniques on a number of datasets.
Cons:
1. The novelty of the approach is relatively low: it’s just a straightforward fusion of the existing techniques.
2. The paper lacks any motivation for use of the particular combination (VRNN and RevGrad). I still believe comparable results can be obtained by polishing R-DANN (e.g. carefully penalizing domain discrepancy at every step)
Additional comments:
1. I’m not convinced by the discussion presented in Section 4.4. I don’t think the visualization of firing patterns can be used to support the efficiency of the proposed method.
2. Figure 1(c) looks very suspicious. I can hardly believe t-SNE could produce this _very_ regular structure for non-degenerate (non-synthetic, real-world) data.
Overall, it’s a solid paper but I’m not sure if it is up to the ICLR standard.
Thank you for your comments and questions. Below we provide more detailed comments to your questions.
1. The novelty of the approach is relatively low: it’s just a straightforward fusion of the existing techniques.
A: We have clarified and highlighted our paper’s key novel contributions in the above 'Response to all reviewers' comment. To briefly summarize - in this paper, we have focused on a very important problem in the healthcare application domain. As far as we know, we are the first to propose unsupervised deep domain adaptation techniques to capture and transfer complex temporal dependencies present in healthcare multivariate time series data. Our domain adaptation framework is general and is suitable for applications where dependencies are present in the multivariate time series data.
2. The paper lacks any motivation for use of the particular combination (VRNN and RevGrad). I still believe comparable results can be obtained by polishing R-DANN (e.g. carefully penalizing domain discrepancy at every step)
A: We apologize for not being clear as to why we specifically used VRNN and reversal gradients. Using a VRNN in VRADA, adds two capabilities: capture complex temporal dependencies present in the data and learns to create the data’s reconstruction.  Both of these help our model learn more significant patterns within the data. Using adversarial training via reversal gradients helps to learn domain-invariant representations. We used this particular combination of VRNN with adversarial training since in healthcare applications matching the data distributions is more interesting than minimizing the distance between the data distribution means across domains (such as Mean Maximum Discrepancy (MMD)).
Moreover, from a theoretical perspective, adversarial training idea [Ganin et. al, JMLR 2016] is derived from the seminal works of [Ben-David et . al, MLJ 2010].
We would also like to inform that we have extensively fined tuned R-DANN to show the best results in our paper. We have compared our approach fairly to Variational FAIR Autoencoder (denoted by VFAE) [Louizos C. et. al, ICLR 2016], which uses MMD criterion. We have empirically shown that VRADA outperforms both R-DANN and  VFAE on the healthcare datasets used in our paper.
Additional comments:
Q 1. I’m not convinced by the discussion presented in Section 4.4. I don’t think the visualization of firing patterns can be used to support the efficiency of the proposed method.
A: The discussion in section 4.4 is used to visualize and qualitatively compare the temporal dependencies captured by different domain adaptation approaches. We used memory cell state activations for visualization. In order to show that domain adaptation results in regular firing patterns, we have added plots to the appendix that show how firing patterns differ when domain adaptation isn’t applied. You can then see a clear direction in the regularity of the firing patterns as (a) domain adaptation is applied, and (b) a better domain adaptation technique is applied. We addressed the regularity found in Figure 4 in appendix section 6.2.3.
Q 2. Figure 1(c) looks very suspicious. I can hardly believe t-SNE could produce this _very_ regular structure for non-degenerate (non-synthetic, real-world) data.
A: Inspired by the feature representation visualization shown in [Ganin et. al, JMLR 2016], we also use t-SNE plots to show the domain invariant feature representations learned by various domain adaptation approaches. In Figure 1, we showed the t-SNE results which we consistently obtained for the source-target pairs from Adult-AHRF to Child-AHRF data. It was surprising for us to see a regular t-SNE plot but that is the plot we obtained for the 3-1 source-target pair. We also varied perplexity (from 5 to 100) and obtained similar t-SNE plots from our VRADA model. We will release our codes to public for reproducibility.
References:
[Ganin et. al, JMLR 2016] Ganin, Yaroslav, et al. "Domain-adversarial training of neural networks." Journal of Machine Learning Research 17.59 (2016): 1-35.
[Ben-David et. al, MLJ 2010] Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. A theory of learning from different domains. Machine Learning, 9(1-2):151–175, 2010.
[Louizos C. et. al, ICLR 2016] Louizos, Christos, et al. "The variational fair autoencoder." ICLR (2016).
The work combines variational recurrent neural networks, and adversarial neural networks to handle domain adaptation for time series data. The proposed method, along with several competing algorithms are compared on two healthcare datasets constructed from MIMIC-III in domain adaptation settings.
The new contribution of the work is relatively small. It extends VRNN with adversarial training for learning domain agnostic representations. From the experimental results, the proposed method clearly out-performs competing algorithms.
However, it is not clear where the advantage is coming from. The only difference between the proposed method and R-DANN is using variational RNN vs RNN. Little insights were provided on how this could bring such a big difference in terms of performance and the drastic difference in the temporal dependencies captured by these two methods in Figure 4.
Detailed comments:
1. Please provide more details on what is plotted in Figure 1. Is 1 (b) is the t-sne projection of representations learned by DANN or R-DANN? The text in section 4.4 suggests it’s the later case. It is surprising to see such a regular plot for VRADA.  What do you think are the two dominant latent factors encoded in figure 1 (c)?
2. In Table 2, the two baselines have quite significant difference in performance testing on the entire target (including validation set) vs on the test set only. VRADA, on the other hand, performs almost identical in these two settings. Could you please offer some explanation on this?
3. Please explain figure 3 and 4 in more details. how to interpret the x-axis of figure 3, and the x and y axes of figure 4. Again the right two plots in figure 4 are extremely regular comparing to the ones on the left.
Thank you for your comments and questions. We have revised the paper, added experiments providing additional insights. In the above 'Response to all reviewers' comment, we have clarified our contributions and below we provide detailed response to your questions. We believe these address your concerns and hope you will consider updating your review.
Our framework VRADA is inspired by the recent advances in deep domain adaptation approaches and is built upon the deep domain adversarial training method (Ganin et. al, JMLR, 2016). We employed Variational Recurrent Neural Network (VRNN) to capture the temporal dependencies since it has been shown to perform better than the dynamic Bayesian networks (DBNs) and Recurrent Neural Networks (RNN) is the earlier works (J. Chung, et. al, NIPS 2015).
Also, note that VRNN is better than standard RNNs since the complex temporal dependencies present in the data cannot be modelled efficiently by the output probability models used in the RNNs (J. Chung, et. al, NIPS 2015, Boulanger. et. al, ICML 2012).
Q: However, it is not clear where the advantage is coming from?
Now addressing the source of our stronger performance. Using a VRNN in VRADA, adds two capabilities: capture complex temporal dependencies present in the data and learns to create the data’s reconstruction.  Both of these help our model learn more significant patterns within the data.
Earlier when asked about the impact of reconstructing the data, we mistakenly answered that the improvement was marginal. We apologize for this blunder. We ran experiments (results are shown in the appendix section 6.2.2) and found non-trivial increases in performance when including reconstruction loss in our joint optimization vs not. This empirically demonstrates that learning how to reconstruct the data helps VRADA learn structural and temporal dependencies present in the data.
Clarifications:
Q: Please provide more details on what is plotted in Figure 1. Is 1 (b) is the t-sne projection of representations learned by DANN or R-DANN? The text in section 4.4 suggests it’s the later case. It is surprising to see such a regular plot for VRADA.  What do you think are the two dominant latent factors encoded in figure 1 (c)?
A: Figure 1 b) is the t-SNE for R-DANN. This typo has been fixed in the revised draft. In Figure 1, we are interested in showcasing the effect of adaptation on the distribution of the latent features. From this figure, we see that our method has much better mixing of the source and target domain latent features. It was surprising for us to see a regular t-SNE plot but this is the plot we obtained for the 3-1 source-target pair. We will release our codes to public for reproducibility. We are working with medical experts to interpret the two dominant latent factors used in figure 1.
Q.In Table 2, the two baselines have quite significant difference in performance testing on the entire target (including validation set) vs on the test set only. VRADA, on the other hand, performs almost identical in these two settings. Could you please offer some explanation on this?
A:Table 2 has been updated. In our earlier draft, AUC for each source-target pair was incorrectly calculated. We sincerely apologize for this error. In the revised draft, we see that in table 2, VRADA obtains slightly better overall AUC scores than R-DANN  (~1.5 to 2% performance gain).
Q. Please explain figure 3 and 4 in more details. how to interpret the x-axis of figure 3, and the x and y axes of figure 4. Again the right two plots in figure 4 are extremely regular comparing to the ones on the left.
A: x-axis of Figure 3 is the time steps of the activation.
For example, in AHRF dataset, we have 4 time steps (as discussed in dataset description Section 4.1), thus x-axis of Fig. 3 has 4 time units.
Similarly, the x-axis of Figure 4 is cell state activations unrolled over time (here the length is 480), while the y-axis is all data points for the source (group 2) and target domain (group 5) pairs. The right two plots in figure 4 correspond to the cell state activations of our VRADA for all the samples of the source and target domains (for the ICD9 code prediction task). We expect these plots to be similar and this indicates that the temporal dependencies (which appear as regular structural patterns in the plots) captured in the source domain (group 2) are transferred efficiently to the target domain (group 5). We further analyze and discuss the regularity found in Figure 4 in appendix section 6.2.3.
References:
[Boulanger. et. al, ICML 2012] Boulanger-Lewandowski, Nicolas, Yoshua Bengio, and Pascal Vincent. "Modeling temporal dependencies in high-dimensional sequences: Application to polyphonic music generation and transcription." ICML (2012).
[J. Chung. et. al, NIPS 2015] Chung, J., Kastner, K., Dinh, L., Goel, K., Courville, A. C., & Bengio, Y. (2015). A recurrent latent variable model for sequential data. In Advances in neural information processing systems (pp. 2980-2988).
Thank you all for your time and thoughtful comments! We have incorporated your suggestions to improve our paper.
First, we would like to clarify the key contributions of our paper. Motivated by challenges within the healthcare domain, we are proposing a novel unsupervised deep domain adaptation framework for knowledge transfer within healthcare time series data. The framework 1) captures temporal dependencies present in the time series data, and 2) transfers these dependencies from  the source to target domain via adversarial training, and 3) is able to obtain state-of-the-art results, outperforming previous domain adaptation approaches. To the best of our knowledge, this is the first time that complex temporal latent dependencies have been used to transfer knowledge in healthcare multivariate time series data.
It is worth noting that our domain adaptation framework is general and is suitable for applications where dependencies are present in the multivariate time series data.
The other major contribution of our work is moreso from an applications perspective. Healthcare data is a new frontier for machine learning. The data we worked with is complex: it’s temporal, it's multivariate, and it’s noisy,. Healthcare data is also quite different from the data that machine learning scientists are accustomed to working with. The dataset we used (which is the largest publicly available medical data healthset) only have a few thousand training examples (which we then further subdivided into multiple domains). This is considered a small dataset.
Further, domain adaptation is typically done when there are ample unlabeled target examples which can be used to learn some mutual representation. We did not have this luxury.
Despite, our model outperformed all other models.
We have uploaded a revised draft to address the questions and comments of the reviewers. In the revised draft, we have added an appendix section with the following new results:
- Discussion on different training methods for our VRADA model and their results
- Results on variations of VRADA model including
--- Adversarial training at each time step
--- Impact of adversarial training on the memory cell state visualizations
--- Study on the effect of reconstruction loss on the prediction performance
- R-DANN model architecture block diagram

Meta-review:
role: PC
data: comment: The paper offers a contribution to domain adaptation. The novelty with respect to methodology is modest, utilizing an existing variational RNN formulation and adversarial training method in this setting. But the application is important and results are strong. Improving the analysis of the results, and studying variants of the approach to understand the contribution of each component, will make this paper considerably stronnger. We encourage the authors to revise accordingly.

Decision: Accept (Poster)