Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper considers the problem of supervised classification of time-series data that are irregularly sampled and asynchronous, with a special focus on the healthcare applications in the experiments. Inspired by the recent progress on differentiable set function learning, the paper proposes an approach called Set Functions for Time Series (SEFT), which views the time series as sets, and use a parametrized sum-decomposing function f as the model for representing the probabilities of different classes, with the sets as the inputs. The problem then reduces to learning the finite dimensional parametrization of the function f under a given loss, which is a differentiable optimization problem that can be learned via standard optimization methods. Together with a positional embedding of the timestamps and an attention-based aggregation, the paper reports improved performance of the proposed approach on a few healthcare time series with asynchronous and irregularly sampled data. In particular, the runtime is largely shortened, while the final accuracy remains competitive to other methods compared in the paper. 

The idea of SEFT is novel and the results are also showing its promise. In addition, the interpretability shown in section 4.3 is also attractive. However, there are several issues that limit the contribution and maturity of this paper. 

Firstly, the paper proposes to model time series as a set. But this loses the information of the order of the time series, which can be extremely important in those datasets with long history dependence. In such cases, I'm not convinced that the set modeling would work. The authors should double check the characteristics of the datasets that are used, and see if they lack long history dependence properties in intuition. If so, this should be mentioned clearly. The authors should also make a more fair comparison with other approaches (like those based on RNN) on datasets with strong history dependence, e.g., Memetracker datasets of web postings and limit-order books datasets. Otherwise, it would be not clear whether this set modeling is generally applicable for general time series data.

Secondly, the authors missed a large amount of related literature for approaching asynchronous and irregularly sampled time series, namely (marked) point-process based approaches. See papers like [1, 2, 3], to name just a few. The authors should at least include some of the recent approaches in this direction for comparison before claiming the superiority of SEFT.

Thirdly, there are a few parts that are not very clear. 1) The discussion about complexity (order m and m\log m) at the bottom of page 1 is weird -- what does this complexity refer to? Does it include the learning of the unknown parameters in the models (like training of the neural networks in this paper)? 2) The loss function in formula (5) is not specified later in the paper (at least hard to find). 3) The Table 1 should be explained in much more details. In particular, why don't we include SEFT-ATTN for H-MNIST? The comment after * is also not clear to me -- is it relevant to why SEFT-ATTN is not included? And what are MICRO/MACRO/WEIGHTED AUC? And why are we using different sets of performance criteria for the first two and last two datasets?

Finally, some minor comments: 1) On page 2, "the following methods" should be "the above methods"; 2) on page 3, the meaning of "channels" should be specified clearer; 3) on page 4, in formulae (3) and (4), should there be \pi or 2\pi in the formula?

[1] Mei, Hongyuan, and Jason M. Eisner. "The neural hawkes process: A neurally self-modulating multivariate point process." Advances in Neural Information Processing Systems. 2017.
[2] Xiao, Shuai, et al. "Joint modeling of event sequence and time series with attentional twin recurrent neural networks." arXiv preprint arXiv:1703.08524 (2017).
[3] Yang, Yingxiang, et al. "Online learning for multivariate Hawkes processes." Advances in Neural Information Processing Systems. 2017.

############## post rebuttal ###############
After reading the authors' rebuttal, I decide to improve the rating to 5 (reflected as 6 due to the ICLR rating system limitation this year).

review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: Summary:
The work is focused on classification of irregularly sampled and unaligned multi-modal time series. Prior work has primarily focused on imputation methods, either end-to-end or otherwise. This paper approaches the problem as a set function mapping between the time-series tuples to the class label. The proposed method is uses a set encoding of a multi-modal time series input, followed by mode-specific encoding of the tuples which are then aggregated in multiple ways prior to classification. An attention mechanism is attached in order for the model to automatically weigh the relevance of tuples for the classification. The model is compared to imputation based baselines on clinical ICU time series classification tasks. The performance mostly appears comparable across baselines but the proposed method has much better run-times. 

The paper is for the most part well written, and related work well characterized. The formulation is interesting and clinically relevant as well so the choice of data-sets makes some sense. I have a few concerns about the architecture formulation and lack of clarification and intuition in what appears to be the main contribution of the paper (Sec 3.2 and 3.3) which I will detail below:

a. In the evaluation, I really want to see a decoupling between the "time encoding step" and "attention based aggregation" on the performance to figure out to isolate different sources of performance improvements. That is can there be a SEFT without time encoding? If not, why not? I encourage more ablation like studies that look at different sources of performance gains and demonstrate them in experiments.

b. The description of Sec 3.3. is really missing key motivation for the choices made around how the attention formulation is designed. For example why does the dot produce include the set elements? What if it doesn't? What is Q supposed to capture? 

c. Is a_{j,i} shared across instances? Then irrespective of the number of observations per instance, the $j^{th}$ tuple gets similar weights? If not appropriate indexing will help clarify this.

d. It would be useful to provide how exactly a label is inferred for a *new* test instance. 

I have some minor additional feedback (just for presentation and motivation purposes):

1. Authors make a claim in the introduction which should likely be qualified with a citation - "Furthermore, even though a decoupled imputation scheme followed by classification is generally more scalable, it may lose information that is relevant for prediction tasks". How does decoupled imputation imply loss of relevant information? By losing information about which observations are missing and relying on that for prediction? Does this clinically make sense? Or even generally? 

2. In Sec 3.3, you probably mean $W_i \in R^{(im(f') + |s_j|) \times d}$. That is parenthesis are missing?

3. What are the +- std errors indicating? Is it cross validation error on a held-out test set? 

4. Initially $i$ is indexing samples and by equation (3), (4) $i$ indexes time(?) and in Sec 3.3 $i$ indexes observations? How are observations defined here? is it measurement of specific modality at a specific time instance? Can you clear this in the introduction itself? 

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
I have read the authors updated draft and response. The experiments section looks much better now. 

1. The overall contribution has less clinical utility in my opinion as generally a patient likely deteriorates over time before an adverse outcome and therefore -- to give the model too much flexibility w.r.t. time ordering doesn't make quite as much sense. This is reflected in the fact that experimental results are not drastically better than other baselines. The authors might be able to show the utility of the method on other time series classification datasets where this is not a limitation of the data itself. However in those settings, it may be a bit hard to beat transformers. Do the authors have a sense of where the benefits of this method really are?

2. Mortality tasks are generally on the simpler side of clinical prediction problems as well. Nonetheless I think the contribution has some utility to the community. I do encourage the authors to try non--clinical datasets for a comparison

3. Please have a discussion that includes limitations and to discuss where the benefits of your methods really lie. A clear and thoughtful discussion is currently missing in your conclusions. 

With that said, I am updating my score to a 6.
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: The paper idea of this paper is straightforward and clear: treat the irregular time series as a bag of events, augment them with time information using positional encoding, and process the events in parallel. The idea is certainly faster than the sequential algorithms such as RNNs and their extensions. However, as shown in the experiments, because it does not encode the "sequential-ness prior" to the model, it is less accurate. Compared to RNNs, the proposed model has better access to the entire length of sequences and does not suffer from the limited memory issues of RNNs and variants.

The proposed idea in this paper can be considered a simplified version of the Transformers. Like transformers, the time and order are only provided to the model using the positional encoding and attention is central to aggregation over the sequence. Realizing the relationship with the Transformers not only decreases the novelty degree for this paper but also requires the authors to include the Transformers in the baselines.

Finally, the results reported in the experiments are nice, especially for the baseline GRU-D! However, the MIMIC-III Mortality benchmark has a lot more than 21,000 stays to the best of my recollection. Can you please elaborate on how the number of data points has decreased?
review_title: Official Blind Review #1

Meta-review: The paper investigates a new approach to classification of irregularly sampled and unaligned multi-modal time series via set function mapping. Experiment results on health care datasets are reported to demonstrate the effectiveness of the proposed approach. 

The idea of extending set functions to address missing value in time series is interesting and novel. The paper does a good job at motivating the methods and describing the proposed solution. The authors did a good job at addressing the concerns of the reviewers. 

During the discussion, some reviewers are still concerned about the empirical results, which do not match well with published results (even though the authors provided an explanation for it). In addition, the proposed method is only tested on the health care datasets, but the improvement is limited. Therefore it would be worthwhile investigating other time series datasets, and most important answering the important question in terms of what datasets/applications the proposed method works well. 

The paper is one step away for being a strong publication. We hope the reviews can help improve the paper for a strong publication in the future. 
Decision: Reject