Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This article tackles the analysis of irregular samples time series. The approach is mainly based on interpolation. Thus, the authors can del with both supervised and unsupervised problems.
The architecture is made of a sinusoid attention layer, a VAE layer that lead to a fixed size set of landmark in the latent space and a RNN decoder.
For the supervised task, the authors add a classification loss.

They obtain impressive results on the interpolation task and interesting results on the classification task.

* In an interpolation problem, we would like to consider a robust baseline as a linear interpolation or an AR-like modeling. Even if I must admit that the authors already propose a lot of comparisons with models from the -recent- litterature, this would give us a meaningful MSE result to compare other approaches.

* notations should be improved (and/or completed with a schema). The model is really very difficult to understand in this version of the article.

* Results are impressive but I don't get which part of the architecture lead to such a performance

review_title: Results push us to accept this article but the lack of clarity in the model description is a strong weak point

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes a novel approach to learn an embedding of continuous time values and use an attention mechanism to produce a fixed-length representation of a time series containing a variable number of observations. In particular, it proposes an mTAN network to leverage the mTAN module in an encoder-decoder framework for both unsupervised and supervised Learning.  The main contribution of this paper is the introduction of Multi-Time Attention Networks to learns a time representation and learns to attend to observations at different time points by computing a similarity weighting by the learning time embedding.  Empirical studies are performed to show the superiority of the proposed model mTANs over several baseline approaches on the tasks unsupervised and supervised learning. 

Pros: 
 
1. The paper proposes a novel model to learns a continuous time representation and adapt to fixed dimensional vectors or discrete sequences. For me, the problem itself is real and practical. 
2. The proposed mTAN is novel for capturing the time dependencies time-series, sparse, irregularly sampled, and multivariate data. 
3. This paper provides comprehensive experiments, including both unsupervised and supervised learning results, to show the effectiveness of the proposed framework.  

Cons: 
 
1.	The paper uses a lot of notations in equations and descriptions, which cause a little bit confusion to follow the authors’ idea. Please consider providing a table to list all the import symbols.
2.	It is better to depict this proposed model structure give the audience a main picture to model.
3.	As the model is attention-based, it has the ability to find the relationships among sequential events. Is it possible to provide one or two case studies to demonstrate the dependencies between time points?
 

Questions during rebuttal period: 
 
Please address and clarify the cons above 

Minor comments: 
1.A few references only list authors, title, and year, but miss publisher or conference, such as “ Michael Mozer, Denis Kazakov, and Robert Lindsey. Discrete event, continuous time rnns. 2017“

review_title: Multi-Time Attention Networks for Irregularly Sampled Time Series

final_score: rating: 7: Good paper, accept
review_content: The paper proposes a novel deep learning framework for handling irregularly sampled time series. 
The paper is well-written and easy to follow. 
The key idea of the paper is to learn embeddings for continuous time values and leveraging a time attention mechanism to learn temporal similarity from data instead of using fixed kernels. 
The time embedding component takes a continuous time point and embeds it into multiple fixed-dimensional spaces. The multi-time attention mechanism takes a query time t and a multivariate sparse and irregularly sampled time series, and returns a fixed dimensional embedding for the query time t. This mechanism is used twice in an encoder-decoder VAE framework with varying reference (input) and query (output) time points.

The idea of the paper is novel, impactful, and well-explained. 
The evaluation and benchmarking is proper with significant improvements over the baselines for classification and interpolation tasks. The approach has significant computational gains over the best performing baseline making it more useful in practice while achieving better or similar classification/interpolation performance.

One query I had is regarding the application of the proposed framework for extrapolation or forecasting tasks. Is the framework directly applicable to such tasks given the way time is handled to get the embeddings as the 0th dimension would keep growing with time (Eqn. 1)? This can have practical implications or challenges even in variable length classification tasks where longer duration time series can be present at test time.

Typos:
time series data data
with mroe than one
In sec 4, input and the generated time series are both denoted by vector s.

review_title: Official Blind Review

final_score: rating: 7: Good paper, accept
review_content: This paper proposed a new model (mTANs) for sparse and irregularly sampled multivariate time series. It incorporates the time attention mechanism to learn embedding for continuous time-series based on a kernel smoothing method. Results on real-world dataset such as EHR data has outperformed other baselines.

I have the following comments that I think is worthwhile to consider to improve the paper:

1. I didn't see much on the main method sections that specifically address/discuss sparsity and irregular sample problems, it seems like the proposed method also works for general time-series data.

2. In the discretization part, it would be better to make it clear how to find the set of reference time points r in practice. It may not be efficient enough if one needs to track each time stamp for each variable to obtain this set $r$.

3. How does an extra module of mTAND in encoding and decoding procedure help intuitively? Given that RNN has already captured temporal information.

4. The noise distribution for sparse / irregularly sampled data can often be heavily skewed, so the Gaussian noise assumption in this paper may not hold.

Additional questions/suggestions:
1. It is better to specify how to compute the gradient for ELBO, e.g., are there any approximations used, are methods like REINFORCE or Gumbel softmax implemented for non-continuous cases, as well as how does the parameter initialize for unsupervised learning problems.

2. Does the unified supervised and unsupervised objective in Eq (15) mean the proposed method combines imputation/interpolation with learning/inference for time series data?

3. From the time series perspective, the positional encoding in Eq (1) is more like a season and trend decomposition, not sure if this plays a similar role here.

4. It would be very helpful for me to understand the paper if the authors could add a schematic figure to demonstrate the complete structure of the model.
review_title: A novel approach and interesting problem, but paper can be further polished.

Meta-review: The work proposed a new approach to encode time series that are irregularly sampled and multivariate using time attention module and an encoder-decoder framework based on VAE. All the reviewers find the approach novel and the experiments extensive with encouraging results. Please continue to improve the presentation of the paper. I would  suggest to move the diagram showing the overall architecture to the main text to assist the explanation. Reviewers also would like to see more explanation on the experimental results and some ablation studies to show the importance of each component of the proposed architecture. 
Decision: Accept (Poster)