Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors present a GAN-based architecture, called GT-GAN, for generating synthetic samples for regular and irregular time series. It comprises of two main components: (1) Autoencoder(AE) - encoder (Neural CDE) encodes time series into a latent space and decoder (GRU-ODE) recovers a continuous path from which synthetic time series is sampled. It is pre-trained using standard reconstruction loss. Decoder is trained additionally with GAN's discriminator loss. (2) Generator(CTFP) and Discriminator(GRU-ODE) networks which are trained using adversarial loss and maximum likelihood training with the log-density. Generator synthesize a time series latent vectors which is used by AE decoder to generate the time-series. This fake time series is then passed to discriminator along with true samples. The authors evaluate their method on 2 simulated and 2 real-world datasets and compare it against 7 related methods. While GT-GAN doesn’t always have a better performance for regular time series, however, it outperforms all other methods for irregular time series. The authors also present an ablation study to understand the role of each loss term and show that each loss component is required for the superior performance of the model.
strengths_and_weaknesses: 1. First of all, the paper is very well-written and provide good amount of details to understand the motivation of the work, and the proposed method.
2. The papers presents a through comparison with several related methods. The authors made extra efforts to modify competing methods for making compatible with irregular time series, and therefore, ensuring a fair comparison.
3. The evaluation shows the superiority of the method both quantitatively and qualitatively, especially for irregular time series.


questions: Good work. I’m just asking this out of curiosity. Since GT-GAN learns the temporal dynamics of irregular time series well, do you think if the model can be modified to do forecasting (which is history conditioned time series generation) for irregular time series?
limitations: As the authors mentioned in the limitations, the proposed method has many hyperparameters and it often becomes difficult to tune with standard methods like Grid Search. One way to deal with it is tying AutoML libraries like SyneTune which provides several state-of-the-art distributed hyperparameter optimizers (HPO).
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper presents a time series GAN model that uses neural ODEs and continues time-flow processes to generate synthetic time series from regular and irregular data (irregular means time series data having missing observations). Specifically, the authors use neural ordinary differential equations, neural controlled differential equations, and continuous time-flow processes. The authors evaluate their approach against standard baselines and datasets and conduct an ablation study of different components. In particular, their method outperforms baselines in the irregular data setting with missing observations. 
strengths_and_weaknesses: The main contribution in this paper is the use of ODE/continuous time-flow processes into a GAN framework for time series synthesis. ODEs and CTFPs are a natural choice when learning from data with missing observations. The overall architecture is close to TimeGAN and uses an autoencoder path (with reconstruction loss) and an adversarial path with a discriminator that is trained on a standard adversarial loss (the individual components are continuous units, though). An interesting addition is the use of the invertible CTFP to use the hidden representation from the autoencoder to train the "log-density" path by converting the hidden representation into noise vectors and then again into a hidden representation by the CTFP and computing its log probability.

The overall architecture of the model is reasonable and the evaluation is sound overall. However, since the main contribution of this paper is using continuous layers for irregular time series synthesis, I would like to propose additional experiments to strengthen the paper. 


questions: 1. ODEs (and their variants) are a natural choice for modeling data with missing observations and this paper provides further evidence for this. However, the overall architecture is very similar to TimeGAN and therefore it is hard to judge how much gain is coming from the architecture itself or by replacing the standard RNN units with continuous counterparts. In particular, I one ablation that would be interesting is to replace the continuous units with GRU-$\Delta$-t or GRU-D. How much performance does the model loose when using these units in GT-GAN? Likewise, how would the baselines perform (in particular TimeGAN) if the corresponding ODE units are used instead of the GRU-$\Delta$-t or GRU-D? This would help to judge how much performance actually stems from the use of continuous units. 

2. The overall architecture is very similar to TimeGAN in the sense that it uses an autoencoder path, a GAN path and a way to link the two (in this case, the inversion/recovery and log density of the hidden representation from the autoencoder). However, this component is not analyzed in the ablation study. How does the model perform when removing this log-density path? Could this path be replaced with a supervised loss between the hidden (fake) representation of the CTFP and the autoencoder? Since this component is the biggest differentiator between the proposed architecture and TimeGAN (on top of the continuous units, of course) it would be interesting to measure this effect. 

3. The authors do not provide details on how the hyperparameters of GT-GAN are tuned. I would kindly ask the authors to discuss their tuning method in the paper. 
limitations: I appreciate that the authors include a limitations statement and discusses that the model contains many hyperparameters and is difficult to train because of that. While this might be the case it is a fairly general statement. It would be more interesting for the reader to understand what specific limitations arise from using continuous units (if there are any). Also, the authors discuss the training time in the Appendix and I think this could be moved to the main text.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes GT-GAN, a framework for synthesizing regular and irregular time series with various techniques such as generative adversarial networks, auto-encoders, neural ordinary differential equations, neural controlled differential equations, and continuous time-flow processes. Experiments show that this method can outperform baselines in regular and irregular time series synthesis. Ablation studies show that each part of the model is critical to the synthesis performance.
strengths_and_weaknesses: Strengths
+ The studied problem is valuable for the field of machine learning.
+ The proposed method outperforms all baselines across regular and irregular time series synthesis qualitatively and quantitatively according to the conducted experiments.
+ The experiments are abundant, and the experimental settings are clearly stated.

Weaknesses
- The technical contributions are limited to this conference. The method proposed appears to be a combination of existing models. And some combinations of these models have been studied, e.g., Flow-GAN [Grover et al., 2017] combines maximum likelihood and GAN training. 
- The proposed method lacks motivation. It is unclear to me that why we need so many technologies together to achieve the goal.
- The description of the training method is not clear to me. It would be better to give an algorithm pseudocode.
questions: What are the difficulties of synthesizing regular and irregular time series in a single framework? How does the proposed method solve the difficulties? In other words, what parts of the method enable it to synthesize both regular and irregular time series? Since the author regards the ability to synthesize both regular and irregular time series as the core contribution of this work, it is better to clarify it more clearly.

In Table 5, GT-GAN without pre-training performs better than GT-GAN in terms of the predictive score, which contradicts the discriminative score. Can the authors explain this issue?
limitations: The authors have discussed the limitations and potential negative societal impact of their work.
review_title: null

Meta-review: metareview: This paper presents a new model for time series data that can handle data sampled at irregular time intervals.  The proposed model makes extensive use of continuous time processes in a GAN framework.  Experiment results show that the proposed model consistently outperforms existing approaches.  All reviewers lean on the accept side and I support that consensus.
confidence: Certain
Decision: Accept