Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This manuscript describes a method that models multiple time series data (sensors) across different individuals (samples). A graph is used to model the dependency between the sensors, such that the data of one sensor could be used to infer another, which is the key point of this ms. How one sensor affects another is modelled as a message passing problem on the graph. At each time point, an observation embedding is generated, either from a sensor or its neighbours in the graph. Observation embeddings of a sensor is then aggregated into a fixed-length sensor embedding, using temporal attention. These sensor embeddings are concatenated into a sample embedding for an individual for downstream analysis such as classification.
main_review: I think the model design is reasonable and the results are impressive. It is interesting to see that the dependency between different sensors really helps the predictions in Setting 2,3,4, which will be very useful for many applications. Only thing is that the learned graph (Figure 4) are very dense and it will be interesting to see what happens by making it sparse.
summary_of_the_review: I recommend this ms to be accepted.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a GNN based method for classifying irregular multivariate time series (MTS). The proposed method explicitly models the dependency between sensors, and propagates impacts of intermittent time series observations from sensors to related sensors for learning MTS embeddings. It jointly learns embeddings and dependency graphs through several attentive mechanisms. The paper provides a good perspective of applying GNN methods on MTS related tasks.
main_review: In this paper, the authors developed a GNN based methods for modeling irregular multivariate time series (MTS) and their classification. The proposed method draws an analogy between the ripple effects of raindrops and the propagation of intermittent observations of irregular MTS, and learns individual sensor embeddings through a neural message passing paradigm on the dependency graph between sensors. The goal is to learn MTS embeddings, which build upon sensor embeddings, which in turn build upon observation embeddings using temporal attention. During training, the embeddings and the dependency graphs were learned jointly, through the GNN related parameters. The dependency graph was inferred through observation-level attention across sensors. Thus, the whole model embeds the MTS in a hierarchical manner. In the experiments, the proposed method was compared with five related methods on irregular MTS, and was evaluated in several settings of missing patterns. The results indicate the effectiveness of the method. The paper provides a good perspective of applying GNN methods on MTS related tasks.

The concerns of the paper are summarized as below.
1. The introduction section claims "none of the existing methods on irregular MTS considered relational structures in representation learning ... irregular MTS" (the end of the second paragraph), which may not be true. The following several papers have taken into account of learning correlation parameters between sensors/variables in learning embeddings of irregular MTS.
	* Shukla and Marlin. "Interpolation-Prediction Networks for Irregularly Sampled Time Series." In ICLR. 2018. (cited in the paper)
	* Wu, et al. "Dynamic Gaussian Mixture based Deep Generative Model For Robust Forecasting on Sparse Multivariate Time Series." In AAAI. 2021.

	Although these papers didn't use GNN, but it is not rigorous to say relational structures have not been considered in the existing methods.

2. From the paper, it is unclear about why assuming inter-sensor dependency is specifically beneficial for dealing the irregularity of MTS, from an intuitive perspective. In other words, it is obscure on how important message passing is for dealing with the irregularity. Considering the dependency is useful for modeling the impacts between sensors, but what's more important to model irregular MTS, such as missing patterns and temporal impact decays (as discussed in many works in this area), are not well elaborated w.r.t. the proposed method.

3. In terms of the technique, it is unclear how could embedding and dependency graph be effectively learned in a joint manner. On one hand, the learning of embedding in Eq (2) depends on meaningful dependency graphs and pairwise attention weights. On the other hand, the learning of dependency graphs in Eq (3) depends on meaningful embeddings and attention weights. This forms a dilemma as neither of the embeddings nor the dependency graphs are available at the beginning. Moreover, the key component, inter-sensor attention in Eq (1) again depends on the embedding of the active sensor, but has little to do with the non-active sensors except for some variable parameters. From the current description, it is hard to understand how could the model training effectively learn both meaningful embeddings and dependency graphs.

4. The proposed method is sort of overcomplicated by introducing many learnable parameters, some are not well justified. For example, in Eq. (4), adding parameter s is for making the output a length-T vector, but what is the meaning of s, and its multiplication with the T-by-T self-attention matrix is unclear.

5. Since Eq (5) doesn't consider temporal order of MTS by using temporal attention, it is better to discuss if there is any impacts by neglecting the temporal structure during the aggregation.

6. The sample embedding is generated by concatenating all sensor embeddings. Since this may generate a long vector, will it limit the application of the proposed method when the number of sensors is large?

7. In the proposed method, many model parameters are shared across all samples for leveraging sample similarities, and there are few parameters for distinguishing different samples. Also, the pair-wise regularization in the loss function encourages similar dependency graphs among all samples. These designs may not be reasonable since not all samples are similar. It may be possible that samples form clusters or groups, some are similar while others are not. Encouraging similarity among all samples may be an unreasonable simplification.

8. In the experiment setting, it is better to summarize the missing data ratio in different datasets to highlight the irregularity in the these datasets. For the compared methods, many important ones were omitted, such as ODE based methods, the above two mentioned papers, GAN-based generative imputation methods (listed below), a memory-based method (listed below), and some GNN-based MTS method (listed below).
	* Luo, et al. "Multivariate time series imputation with generative adversarial networks." In NeurIPS. 2018.
	* Tang, et al. "Joint modeling of local and global temporal dynamics for multivariate time series forecasting with missing values." In AAAI. 2020.
	* Hu, et al. "Time-Series Event Prediction with Evolutionary State Graph." In WSDM. 2021.
	* Wu, et al. "Connecting the dots: Multivariate time series forecasting with graph neural networks." In KDD. 2020.

	As for the GNN-based MTS methods, although they don't directly model irregular MTS, it is feasible to first do statistics based imputation (mean imputation) and then apply them. This comparison helps understand how could the proposed method better address irregularity. If some of those methods have been compared on the same datasets in the same settings in other papers, it is better to report the numbers from there for a comprehensive understanding.

9. In the results of "Setting 4", it is unclear why the proposed method performs better than other methods when transferring from one group of samples to another, and what technical design helps the proposed method be effective in domain adaptation, which seems to be another topic different from modeling irregular MTS.

10. In the experiments, the visualization of the dependency graph cannot support the conclusion that the model learns dependencies for distinguishing classes. To this end, the visualization should demonstrate similarity of dependency graphs within the same class, and the dissimilarity between dependency graphs from different classes. Only observing dissimilarity across different classes provides few information. It is possible that the dependency graphs within the same class are different from each other, but the paper doesn't visualize nor discuss it. Also, considering samples with the same label as similar samples may not be reasonable in medical data because patients from the same class may have different data distribution due to different demographics, regions and population.

It is good to see the consideration of using visualization, as it facilitates the understanding of what have been really learned by the proposed model, especially considering the complexity of the model design. However, the current results make it difficult to judge whether the performance gains in Table 1, 2, 3 are from the correct learning of the attention and dependency graphs or other factors. Thus it is better to see a thorough visualization and discussion on inter-sensor attention, temporal attention, and the dependency graphs.
summary_of_the_review: The paper is meaningful in exploring the new application of GNNs in irregular MTS related tasks. However, the paper is concerned w.r.t. its unclear description in some important concepts, the technical limitations, the lack of discussion and comparison of some related methods, and the non-comprehensive experiments.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The core idea of the paper is transforming multivariate time series data into a vector z and using z for classification.
The transformation has two steps. Firstly, each time record of each sensor is mapped into a  space of 20 dimensions. The first 4 dimensions are used for feature extraction, and the last 16 dimensions are used for positional encoding. Secondly, self-attention is used to transform the embedding into a vector z and use it for classification.

main_review: It is hard to understand the operation for the equation, and the figures in the paper are confusing, and the author does not provide any code to help us to check it.

For example, 
1) The dot product in Figure 3 a is ambiguous.
Based on Appendix A.4, it seems the attention α and edge e are all real numbers.
Thus, in Figure 3 a, the dot product for α and e are multiple of the two real numbers without extra weight. However, the dot product of r||p and h is a dot product with matrix D between them. What’s more, the dot product between u and v is a dot product with two vectors between them.


What’s more, based on equation 3, e is calculated by α. Therefore, it seems there is no need to link u and e in Figure 3 a.

2) The form of β is confusing.
In Figure 3 b, the attention weight β seems to be a vector, but it is annotated as a real number instead of a vector symbol.  What’s more, the size of s in Equation 4 is not given; thus, it is hard to say the β is a vector or a real number.

3) The packed observation embedding in Figure 3 b is not mentioned in the paper.

 I tried to search the word “packed”. It is only shown in Figure 3 without explanation in any other places.

summary_of_the_review: The presentation of the paper has big space to improve. The source codes are unavailable, and the main results cannot be confidently reproduced.
review_title: null

Meta-review: The authors introduce a GNN based method for classifying irregular multivariate timeseries.
They represent the dependencies among sensors using a graph structure and deploy message passing to  
model the effect of a sensor on another)s=. The approach jointly learns embeddings and the dependency graph. 

The manuscript gathered a clear accept (8) and two marginal below the threshold scores (5). 
I want to accept this work and I explain why. 

The reviews and the ongoing discussion during the rebuttal showed that the work is interesting 
with its main strength being the novel exploration of GNNs application on irregularly samples multivariate 
time series.

There were many concerns raised by a reviewer regarding important  theoretical and methodological issues in the paper. 
During the rebuttal phase, the authors clarified and resolved the majority of the concerns and there was an ongoing discussion among the two sides, authors and reviewer (which I have to admit was a pleasure to watch researchers communicating). 
The authors took into account the feedback and revised the manuscript accordingly. Having read the edits myself, I believe the submission is substantially improved and addressed the concerns sufficiently. 

I expect that this work will stimulate further research in the community and I would like to accept this.
Decision: Accept (Poster)