Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper introduces a novel model of Variational Autoencoder that deals with irregularly sampled time series with a probabilistic approach to do time series interpolation.

The main contribution is the architecture by itself, its components, and the training process. 
The model was evaluated on both real-world data sets from the medical and climate domain and synthetic data.

main_review: + 
Novel method
Well written
Notation is clear and organized
Well evaluated on multiple datasets
Extensive ablation tests and discussion about the model components
Code available at review

-
Big novelty overlap with [1]

[1] Satya Narayan Shukla and Benjamin Marlin. Multi-time attention networks for irregularly sampled time series. In International Conference on Learning Representations, 2021

summary_of_the_review: Very good paper with correct methodology. relevant results and a concerning overlap with a recent ICLR paper
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a novel model, HeTVAE, for probabilistic interpolation of time series that are irregularly sampled. HeTVAE builds on prior work by complementing it with a learned time-dependent output variance in the VAE and architectural improvements. The latter include a new branch accounting for the distribution of the sampled timestamps in the series and the addition of a deterministic branch bypassing the stochastic variational latent variable. The performance of HeTVAE is evaluated on multiple datasets against various baselines and via ablation studies.
main_review: ## Contribution

### Advantages

The tackled problem of probabilistic interpolation of time series is relevant and especially valuable as it has received less attention than the forecasting task, at least in the neural networks community. This is especially the case in the considered setting where observations are not synchronized between dimensions.

The proposed model is for the most part interesting and well motivated, justifying the additions that are made on top of the prior mTAN. These contributions are, to the best of my knowledge, novel. The so-called heteroscedasticity of the output variance answers a crucial issue of standard VAEs with constant output variance in this context of application - even though this has already been considered elsewhere. The intensity branch of the introduced network clearly addresses the shortcomings of prior state-of-the-art methods. Accordingly, the paper is mostly well written and easy to read.

Experimentally, the performance of HeTVAE is state-of-the-art by a large margin, showing the benefits of the approach. Ablation studies show the individual contribution of each model component; I suggest that the authors include the full ablation of the appendix in the main paper. The experiments are sufficiently well designed and diverse on real-world applications in order to correctly assess this performance. Furthermore, qualitative experiments with examples of interpolation are appealing and highlight the impact of the model and its different components.

For all these reasons, I think that this paper is interesting and might be ready for publication at ICLR after the revision during the rebuttal. Indeed, I would express reservations, that I detail below.

### Limitations and Potential Improvements

A first limitation of the current version of the paper deals with the significance of the introduced heteroscedasticity. While very few models that are able tackle the same task as HeTVAE seem to leverage temporally varying output variance, it is unclear whether this is an inherent advantage of the described model, or if it is orthogonal to the other architectural considerations. In other words, could heteroscedasticity be reasonably applied to the considered RNN/ODE-based baselines? If it does, I recommend that the authors include these augmented baselines in their experiments to better contextualize the performance of HeTVAE. In any case, a more detailed discussion of the related work is necessary in this regard. In particular, I would advise the authors to consider other works which, to my understanding, could be included in the related work and considered baselines with heteroscedasticity [1, 2].

A second limitation is a partial lack of specification or explication of the modeling choices. In particular, the motivation and intuition behind the deterministic path is missing from the paper, to my understanding; I am wondering about the impact of its introduction in parallel of the usual variational latent variable, since the deterministic variable is unconstrained whereas the stochastic one is constrained by the KL term in the loss. An additional ablation showing the performance of HeTVAE without the stochastic branch would thus be interesting. Moreover, the role of the reference timestamps $\mathbf{r}$ is unclear and they are only specified in the appendix; could the authors discuss their necessity in the model and the relevance of their choice? Further discussion about how the baselines are adapted for the considered task should also be included. Finally, it is unclear until Section 4 that the learned VAE also learns to predict interpolations besides reconstructing its inputs: this information should be clearly stated earlier, probably in the description of the ELBO.

## Other Remarks and Questions

### Scalability of the Intensity Pathway

Could the authors comment on the scalability of the implementation of the intensity pathway as described in Equation (1)? To my understanding, the numerator pools over all elements of the dataset and it would seem that a large-scale dataset would prevent an efficient computation of this branch.

### Nature of the Augmented Learning Objective

There seems to be a typo in the Equation (11): the augmented learning objective should be minimized, so it should be negatively added the the ELBO.

### Code and Supplementary Material

Based on my limited review, the experiments of this paper seem to be reproducible. The provided code is appreciated. I recommend the authors to remove the hidden folders in the archive, especially the `.git` which is irrelevant and makes the archive heavier than needed.


## References

[1] X. Li et al. Scalable Gradients for Stochastic Differential Equations. AISTATS 2020.\
[2] A. Norcliffe et al. Neural ODE Processes. ICLR 2020.
summary_of_the_review: Given the relevance and experimental results of the proposed model, I think that this paper may be valuable to the audience of ICLR. Nonetheless, some substantial limitations about the nature of the heteroscedasticity contribution and unclear points in the paper prevent me from giving a firm positive score, thus choosing a score marginally below the acceptance threshold.

I believe that this paper may be improved in the course of the discussion phase and would be pleased to raise my score, should my concerns be addressed during this time.

### Post-Rebuttal Update

I acknowledge the authors' response and thank them for their detailed answer. As explained in my follow-up response, the rebuttal partly addressed my concerns. While I still find that it could be improved on some aspects, I believe that this paper may be accepted to the conference. Therefore, before discussing with the other reviewers, I choose to raise my score from 5 to 6.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper introduces a VAE-based model for interpolation of irregularly sampled time series. The temporal input data is mapped to a latent representation over fixed reference points with an attention mechanism, using an intensity network that allows to encode data sparsity information. This latent representation can then be used to interpolate points at new time steps. Thanks to the intensity network and the heteroscedastic output layer, the proposed HeTVAE model can capture uncertainty estimates over the interpolated points. 
The model is tested on a number of datasets containing irregularly sampled points, and outperforms competing methods in the interpolation task.
main_review: STRENGTHS

1. A great deal of modern machine learning literature for time series data often assumes regularly sampled time series, with no missing data and fixed size outputs. This is however rarely the case in many real world applications, which introduces many technical challenges that are not obvious to overcome with more standard architectures. The model presented in this paper is a possible way to tackle these challenges, and as such, I found this paper a very interesting read.
1. Despite building heavily on the mTAN model, the new ideas introduced in the paper are novel and well motivated.
2. Empirically, the HeTVAE outperforms competing methods by a large margin, and seem to be able to correctly capture uncertainty over time (as seen for example in figures 2 and 3)
3. The paper has extensive ablation studies that justify all the new components of the model  

WEAKNESSES
1. I found the theoretical exposition in section 3 somewhat confusing in its current form, I could only follow it after reading details in the appendix and reading section 3 once again.
    1. Reference points play a key role in the HeTVAE, but from section 3 it is not clear what their role is as well as how they are chosen. I was only able to really grasp their role after reading the appendix and looking at the code, which should not be the case. Only in appendix A4 I could understand that they are regularly spaced in [0,1], and only in A.6.1 that the time is scaled between 0 and 1 in all datasets (after which the choice of reference points makes sense)    
    2. related to the above, "reference points" are mentioned in the "intensity encoding" and "model output" paragraphs. But for a reader not familiar with the mTAN it is not obvious why we are interested in them.    
    3. The prior over z is not defined in the paper 
2. The "augmented learning objective" seems quite hacky to me, and I wonder if there are better ways to achieve the same (e.g. better initializations, KL annealing, ..).     
    1. Learning output variances is normally not a problem in VAEs, why is it a problem in this case?
    2. How robust is the choice of the scaling factor lambda across different datasets?
3. Choosing equally spaced reference points means that if one uses as a test point a sample temporally close to the input data (or even a sample from the input data), the imputation of the model might be unnecessarily poor. Could this be improved?
4. As stated by the authors, the model is only able to provide a marginal distribution at each time step, which practically means that the sampled trajectories might look inconsistent (non-smooth). Exploring the usage of sequential latent variable models for this would be an interesting future research direction. 
5. The baselines for HVAE RNN  and HVAE RNN-ODE are much worse than forward imputation, which makes me question the implementation of the models.
summary_of_the_review: This is a paper that can have an impact in real world applications, and as such I think it should be accepted. However, I did not find the paper easy to follow in this form.
For this, for now I have set "marginally above the acceptance threshold", but I would be happy to revise the score upwards if I see that the authors improve the exposition in the paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper introduces several improvements over the previous work mTAN to better support probabilistic interpolation. Specifically, intensity encoding is introduced to make the model be aware of information about input sparsity. Also, the homoscedastic output distribution used by previous work is replaced by a heteroscedastic distribution. Experiments results show that this improved model (HeTVAE) achieves both better likelihood estimation and mean prediction compared to previous works.
main_review: Clarity: I am confused about the differences between heteroscedastic and homoscedastic after reading the paper. Does the heteroscedastic output means that you learn the variance and the homoscedastic output means you fix the variance? What is the shape of w and v in equation 3? what is g in equation 7?

Originality: I think the intensity encoding is novel and makes sense to make the model be aware of input sparsity. For the heteroscedastic output layer, based on my understanding (see Clarity above), it just allows the model to learn the variance rather than using a fixed variance. If my understanding is correct, I don't this heteroscedastic output layer is novel because there are lots of works that learn a variance for maximum likelihood estimation. Combining a deterministic path and a probabilistic path is not novel, which has been proposed in [1, 2]. The augmented training objective in equation 11 that encourages the predicted mean to be close to samples x makes sense, but I think there should be some existing works that also use this trick.

Experiments: I am satisfied with the performance this model achieves. However, I think the authors should also compare to Neural Process [1] and Attentive Neural Process [2], which are mentioned in the related work. [1,2] also use attention mechanism and do probabilistic interpolation. This paper also misses citation and comparison to a previous work NRTSI [3] that can also impute irregularly-sampled time series. I suggest the author compare to NRTSI on the irregularly-sampled Billiard dataset introduced in NRTSI.  Also, I am not clear about what the baseline HTVAE mTAN means? Does it mean "homoscedastic temporal VAE mTAN"? If my understanding is correct, I think the author should also compare to HeTVAE mTAN ("heteroscedastic temporal VAE mTAN") that allows mTAN to learn a variance rather than using a fixed variance. For the ablation study model HeTVAE - DET, because the deterministic path is removed, the model capacity decreases and makes the comparison unfair. For a fair comparison, I suggest increasing the capacity of the probabilistic path after removing the deterministic path. 



[1] Garnelo, Marta, et al. "Neural processes." arXiv preprint arXiv:1807.01622 (2018).

[2] Kim, Hyunjik, et al. "Attentive neural processes." arXiv preprint arXiv:1901.05761 (2019).

[3] Shan, Siyuan, and Junier B. Oliva. "NRTSI: Non-Recurrent Time Series Imputation." arXiv preprint arXiv:2102.03340 (2021).


summary_of_the_review: Given the limited novelty, some unclarities, and lacks of some baselines, I give 5 to this paper now. If the authors can clarify and add these baseline results, I may consider increasing my score.
review_title: null

Meta-review: The submission proposes a model to handle uncertainties in an irregularly sampling time series setting (HetVAE), built on the VAE framework and the previous work on mTAN (multi-time attention networks), and introduces components to encode sparsity information and heteroscedastic output uncertainty. The paper is clear, well-motivated, and contains extensive ablation studies showing the effect of eaach added components.
I recommend this submission for acceptance.
Decision: Accept (Poster)