Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Authors proposed the method that could represent the sparse sequential highly-multivariate data, which could be trivially represented as neither 1D nor dense multivariate series data. Authors proposed the 1.5D representation which is composed of token-value pair and proposed the value-aware transformer that is able to use the representation. Experimental results on in-patient laboratory data showed the effectiveness of their data representation and algorithms (ie. value-aware transformer).
main_review: > Good motivation: The motivation of solving the representing sparse sequential highly-multivariate data is novel. 

> Authors proposed the good idea in overall; however there are few heuristics involved: I agree that proposing the 1.5D representation and value-aware transformer are novel and quite good idea. However, when proposing the 1.5D representation, I could not find the mathematical grounds for few factors: For example, there is neither clear reason nor ablative studies for selecting 5 degreed values (ie. XLow to XHIGH). 

> Good performance while few ablative studies are missing: I could find that 1D representation have been explicitly experimented during the experiments; however I could not find the baseline that represents signals in the form of multivariate series data and applies corresponding models. I think this baseline is required as the evidence for authors' insist that the sparse sequential highly-multivariate data could be better represented in 1.5D compared to multivariate representation.

summary_of_the_review: Due to the reasons I stated in the main review, I am currently in the borderline accept; however effectively rebutting my comments will make me raise my voting towards higher score.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work proposes an approach for modeling sparse and multivariate sequential data which are commonly found in the healthcare domain. The paper presents and compares different data representations and introduces a 1.5d representation and outlines principles for modeling this type of data. The paper presents the value-aware transformer, a transformer decoder-only architecture for the 1.5d data representation. Based on the experiments on the MIMIC-III dataset, the proposed model and the data representation achieve better performance than several other standard data representations. 
main_review: Positives:
1. The paper focuses on the task of modeling sparse and multivariate sequential data which is an important and practical problem.
2. The paper is easy to follow.

Concerns:
1. The authors are missing several important related works and comparisons with recent and SOTA methods for sparse and multivariate sequential data [1, 2, 3].
2. The main contribution of the paper is not clear. 
3. The authors focus on the data representation for the sparse and multivariate sequential data but fail to compare it with some of the similar data representations introduced in [3] and [4].
4. The proposed framework has marginal novelty in terms of the model architecture as it is a fairly standard transformer architecture. 
5. The authors should provide experiments to show if the design principles hold in practice, in particular, provide a comparison with strong coupling.
6. The authors should perform experiments with more datasets to show the effectiveness of the model.

Additional Comments:
1. How did the authors choose the dimension of the token and value embedding?
2. Why did the authors choose to not consider the temporal information such as the time of the lab events which is fairly common to consider in several works in this domain?
3. What happens if multiple lab events are observed at the same time? How is the order decided in that case? 
4. Could the authors clarify more about the difference between CLS and EOS token and why it has to be added in the end? Based on my understanding of transformer architectures it doesn't really matter where the token is added. 
5. Is the padding used because of the variable length of sequence from different examples?


References:
1. Zhengping Che, Sanjay Purushotham, Kyunghyun Cho, David Sontag, and Yan Liu. Recurrent neural
networks for multivariate time series with missing values. Scientific Reports, 8(1):6085, 2018.
2. S. N. Shukla and B. Marlin. Multi-time attention networks for irregularly sampled time series. In International Conference on Learning Representations, 2021.
3. Max Horn, Michael Moor, Christian Bock, Bastian Rieck, and Karsten Borgwardt. Set functions for
time series. In Proceedings of the 25th International Conference on Machine Learning, 2020.
4. S. N. Shukla and B. M. Marlin. A Survey on Principles, Models, and Methods for Learning from Irregularly Sampled Time Series. CoRR, abs/2012.00168, 2020.

summary_of_the_review: The contributions of the paper are not clear. The paper is also missing important related works and comparisons with recent approaches in this domain. The paper has only marginal novelty both in terms of the data representation and the proposed approach.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes an approach for learning representations of spare sequential data, commonly found in healthcare settings. The work challenges issues with sequential data in electronic health records such as sparsity of events, unequal lengths, which also present imputation challenges. A transformer-based approach is presented which incorporates information about the different sequential events (labs) along with their values which is termed as a 1.5-dimensional representation. The transformer is trained to obtain the representation of sequences of events, labs conducted for in-hospital patients, along with the values of the labs which are categorized based on the quantile range obtained from the training data. The representations from the transformer embeddings are used for predicting two outcomes, 1) in-hospital mortality, and 2) length of stay at 48 hours after admission. The value- aware transformer performs marginally better than a value-unaware transformer with respect to $\mathcal{L_{\text{\[TOK\]}}}$. There is also an improvement in the balanced-accuracy and ROC-AUC for in-hospital mortality when compared with feed-froward neural network and value-unaware transformer. Similar performance characteristics are also observed for length-of-stay regression. 
main_review: The paper tackles an important challenge with modeling sequential data in healthcare, sparsity of events-- labs are generally ordered based on the patient context and may be ordered multiple times and it is important to account for the sequential nature of the event as well as the value for each event. The concept that is termed as 1.5-dimensional setting proposes to account for the value when learning the embeddings for the token which are the lab events with their values, an interesting approach for using transformers. However, there are certain concerns with respect to the design choices regarding the architecture as well as the training process such as the decay rate, the dimensions of the hidden layer, $\gamma$. Moreover, the performance that is achieved for say in-hospital mortality is only sub-par with respect to other approaches for predicting in-hospital mortality that are largely neglected in this work. This creates another problem to assess the merits of the proposed method. While there has been considerable work (see 1-5 below), for predicting in-hospital mortality using MIMIC data, comparison with such baselines needs to be performed. 

The paper can also be improved to make it better for the reader. Adding why the value-aware representations would be beneficial for downstream tasks in the introduction would help to situate the need for this work. Similarly, adding context about the heterogeneity of the data for patients' individual health can help. The choice for the number of dimensions for the token and value representation needs to be supported. While that for the values can be justified based on the quantization, there are concerns with the generalizability of the quantization process as well. I would encourage the authors to think about the sensitivity of the quantization to the population incorporated in the study. The eICU dataset provides a richer ground to handle these generalizability issues and could be checked. 


1. Song, Huan, et al. "Attend and diagnose: Clinical time series analysis using attention models." Thirty-second AAAI conference on artificial intelligence. 2018.
2. Purushotham, Sanjay, et al. "Benchmark of deep learning models on large healthcare mimic datasets." arXiv preprint arXiv:1710.08531 (2017).
3. Che, Zhengping, et al. "Recurrent neural networks for multivariate time series with missing values." Scientific reports 8.1 (2018): 1-12.
4. Jarrett, Daniel, et al. "Clairvoyance: A pipeline toolkit for medical time series." International Conference on Learning Representations. 2020.
5. Li, Ke, et al. "Predicting in-hospital mortality in ICU patients with sepsis using gradient boosting decision tree." Medicine 100.19 (2021).
summary_of_the_review: In general, the paper suffers from clarity as well as comparison with existing baseline other than transformer-based approaches and feed-forward networks. Assessing the performance of the model on different subpopulations based on demographics would be helpful to identify how the method performs subgroups rather than on average since it is known that there are differences in lab-orderings based on demographic attributes such as gender and self-reported race. This could strengthen the contributions.
review_title: null

Meta-review: The paper propose a value-aware transformer for sparse multivariate time series data. While to approach is well motivated and the problem well-motivated from a clinical viewpoint, the comparison with related work brought up by reviewer qFRi and  reviewer ph4X would really make it clear where this paper stands. The authors attempt to diffuse this issue in their replies, but empirical comparisons in the paper would guide practitioners more. This is especially important as the paper is motivated by a real-world problem.
Decision: Reject