Unified recurrent network for many feature types

Abstract: There are time series that are amenable to recurrent neural network (RNN) solutions when treated as sequences, but some series, e.g. asynchronous time series, provide a richer variation of feature types than current RNN cells take into account. In order to address such situations, we introduce a unified RNN that handles five different feature types, each in a different manner. Our RNN framework separates sequential features into two groups dependent on their frequency, which we call sparse and dense features, and which affect cell updates differently. Further, we also incorporate time features at the sequential level that relate to the time between specified events in the sequence and are used to modify the cell's memory state. We also include two types of static (whole sequence level) features, one related to time and one not, which are combined with the encoder output. The experiments show that the proposed modeling framework does increase performance compared to standard cells.

Reviews and author responses:

Thanks for your comments, our responses to each are below.

 - If I understand correctly, each sparse feature has its own memory state. This will pose a scalability issue when the number of features are large, e.g., those in electronic medical records where the number of features can go to tens of thousands, and each feature can be very sparse in its own way (e.g., diseases progress differently).
 
In our experiments the memory states of each sparse feature were of the same size as the dense state, but it would be possible to make these states smaller as the number of sparse features grows. We did not need to do this for our experiments, but it would probably be necessary in the mentioned medical domain. As for the features being sparse in their own way, STLSTM cells are designed with this in mind.

 - As for the ICLR audience, I think it will be informative to explain design choices. For example, maintaining the memory when sparse feature is not observed may implicitly assume that the missingness is uninformative. In medicine, on the other hand, missingness can be informative, e.g., when doctor decides that a measurement is not needed.

For our test cases, the missingness was uninformative in that the value simply did not change since the last measurement. While we see the issue with informative missingness, it seems like distinguishing this from all other sources is a problem for any model. In the medical domain specifically, it may be possible to indicate from the data whether a test was purposefully omitted, in which case this could be treated as a sparse input.


 - For evaluation, I had a hard time decoding Figure 5, especially the legend of the lines and the small fonts. May be synthetic datasets will help to shed the light on the performance of the STLSTM better. Also, putting results of two experiments in the same figure in this case makes it harder to read.

The figures have been updated with a larger font as can be seen in the revision, and while we agree that larger figures would be preferable, we are constrained by the page limits.

Thanks for the authors' detailed responses, those proposed updates address most of my concerns regarding presentations. However, I agree with other reviewers that this work relies quite a bit on existing work, and I'll maintain my score. 

If I understand correctly, each sparse feature has its own memory state. This will pose a scalability issue when the number of features are large, e.g., those in electronic medical records where the number of features can go to tens of thousands, and each feature can be very sparse in its own way (e.g., diseases progress differently).

As for the ICLR audience, I think it will be informative to explain design choices. For example, maintaining the memory when sparse feature is not observed may implicitly assume that the missingness is uninformative. In medicine, on the other hand, missingness can be informative, e.g., when doctor decides that a measurement is not needed.

For evaluation, I had a hard time decoding Figure 5, especially the legend of the lines and the small fonts. May be synthetic datasets will help to shed the light on the performance of the STLSTM better. Also, putting results of two experiments in the same figure in this case makes it harder to read.

I have read other reviews and authors' response and I agree that the work has practical merits and is worth pursuing further. However, at this stage, I can only maintain my score.


Hi,

I have read the authors' response to my review, as well as the other reviews and responses. The responses to my comments were largely as expected, so they do not change my view of the paper.

I agree with Reviewer 2 that including one or two concrete running examples through the paper would make the semantics of the different types of features more clear.

However, I mentioned that the work relies on quite a bit on existing work, and several of the other reviewers also had questions about the novelty of this work. I believe that pulling the existing material together into a unified framework, as well as explicitly distinguishing the different groups of features, is nevertheless a worthwhile contribution.

Thanks for the review and helpful comments. We have responded to your concerns below.

Con 1: TLSTM accounts more for irregular time between samples as opposed to asynchronous feature sampling, where different features are sampled at every time step.  The modification of TLSTM updates is an expansion on that work, but the inclusion of sparse features tackles a completely different problem than TLSTM and in our opinion is novel.

Con 2: The plots do show the percentage gains in score as you guessed. We can modify the figures to make this clearer. We could also include a legend showing which line corresponds to which feature, but it’s not clear how much value this would provide. While there is a disparity in the number of parameters between the models, additional experiments showed that the base model did not improve by adding more parameters. We will add this information to the text. All information on sequence processing for the power consumption data set can be found in the Appendix.


Con 3: The quoted part should read “the number of sparse features grows” and will be updated. Sparse features are used in TLSTM but are simply treated as dense features. We can clarify this in the text. In Section 4.1 we mention that static features are present for all experiments, with the exception of the Table 1b which investigates their effectiveness.

Con 4: Static decay and decay features are very similar, however as stated in Section 3.5 “the static decay features apply to the sequence as a whole rather than an individual time step.” A static decay feature does vary with time and could be the time elapsed between the final time step of the sequence and the prediction time being used for that sequence.

Minor comment 1: We can make the figure/fonts bigger as the length restrictions allow.

Minor comment 2: There are labels for groups of equations on Page 4.

Minor comment 3: More details can be found in the Appendix.

Minor comment 4: We restrict this with \alpha \geq 0 so that the function is non-increasing. This restraint will be added to the text.


Thank you for the comments, we are glad that you found the paper thought provoking. In light of your comments about the motivation, we will add explicit examples for each subsection in Section 3.5 to make this clearer, but our churn dataset did in fact support all five data types. Examples of each type for the power consumption dataset can be found in the Appendix. Although some of these features were synthetically created to mirror our churn dataset, they can still give some insight to the need for each feature. A decay feature such as the time between timesteps allows us to take into account the nonuniformity between events. For a static decay feature we used the time between the last event and the prediction time. The intention is to remove short term contributions in the prediction when there is a long-time gap after the final event. Static decay features for the power consumption data set are day of week, day of month, and time of day as these feature are informative but are relevant on a sequence level but not on a time step level.

Thank you for the review and your detailed comments. Below are our responses to some of your concerns.

Comment: However, I had difficult coming up with an example of a “static decay feature”. It would be helpful to give a concrete example of one of these in the main text. (It is also not clear to me why the difference in time between the time of the last event in a sequence and the prediction time for that sequence would be considered a “static decay” feature rather than just a “static” feature.)

It seems that giving examples of each feature type in the subsections of Section 3 would help explain the motivation for incorporating each type of a feature. There is one ‘static decay’ feature mentioned in the Appendix, and as you mentioned it is the time between the last event and the prediction time. This is treated differently from ‘static’ features for the same reason ‘decay’ features are not ‘dense’ features, i.e., they are used to decompose the output of the RNN network (or the memory state for ‘decay’ features) into short and long-term components. The intention is to remove short term contributions in the prediction when there is a long-time gap.

Comment: First, the sparsity mechanism is rather simple. In many domains (e.g., the medical domain considered in several of the cited papers), missingness is non-uniform and is often meaningful. While “meaningfulness” may be difficult to simulate, burstiness (non-uniformity) could be simulated.

There are actually two sampling mechanisms for the power consumption dataset, random and group sampling, which are detailed in the appendix. We agree that the random method is rather simple and so we also included group sampling to simulate the ‘burstiness’ you mention. While this may still be missing some information that cannot be simulated easily, we can say that our churn dataset did exhibit natural non-uniformity.

Comment: Second, for the groups, it is not clear whether all combinations of, e.g., 2 (informative) feature were sparsely sampled or if only one group of 2 was chosen. If the former, then some measure of variance should be given to help estimate statistical significance. 

It was only one subset of the two, determined by which features most improved predictions individually that are treated as sparse. This clarification will be added to the manuscript. 

Comment: Third, the particular classification task here is, essentially, forecasting one of the input variables. While that is certainly a relevant problem, many other time series classification or regression problems are not tied so directly to observations. It is not clear if these results are relevant in that setting.

While we cannot share much detail on the churn dataset, we can reveal that the labels are not tied directly to the input features. In churn, the prediction is if a customer is likely to churn while the features are for example customer interactions with the company. So the predictions are completely decoupled from input features.  Despite this we still saw performance gains for our model over the base model.  The tie mentioned in your comment is present in the power dataset since the data does not include other possible labels. 




We would appreciate receiving further comments and details regarding why the proposed solution is not convincing, and why the main results are not informative? Especially since we see our model performing better under most circumstances.

This paper proposes a new type of recurrent neural network which takes into account five different features: in addition to the prevalent dense features, the author(s) also consider(s) sparse features, time features, global static and changing features. The differences in feature types are reflected in the cell state or output state update rules. Experiments on a modified UCI dataset and a proprietary dataset show that the proposed model outperforms the time-variant LSTM (TLSTM).

Pros:
1. By decomposing the cell state into different components for different feature types (dense vs sparse, short term vs long term) and update them in different manners, the model takes advantage of the feature type information.
2. By updating sparse feature related cell states only when sparse features are present, it could be potentially computationally cheaper than treating everything as dense (although in the paper due to more parameters the proposed model is actually slower).

Cons:
1. The contributions are not significant. It seems that TLSTM already "accounts for asynchronous feature sampling" (Sec 1) and the novelty here lies most in how sparse features are treated.
2. The presentation is sometimes confusing. For example, in Figure 5 which presents the main results, what's the "relative change in F1 score" and what's the unit in the plot? If it's percentage the gains seem to be too small and could be potentially due to the additional parameters. Besides, what does "group sampling" mean exactly? Furthermore, legend seems to be missing.
3. Crucial implementation details are missing. The paper mentions that "the number of features grows" in the proposed model. Are sparse features and static features used or not in TLSTM? 
4. What's the difference between a static decay feature (Sec 3.5) and decay features (Sec 3.2)? Isn't the static decay feature varying with time as well?

Minor comments:
1. Figure 1 and 5 are too small and hard to read.
2. Sec 3.3, "updated based on Equation 1 and Equation 2", but none of the equations are numbered in this paper.
3. Some discussions on the proprietary dataset seem to be irrelevant. I'd rather see how are sparse features generated for the UCI dataset.
4. The decay function $g= 1 / log (e + \alpha^T x_t^{\delta})$, how can we make sure that as time passes it decreases as time passes?


Overall, I think explicitly taking into account different feature types in the LSTM cell update rules is interesting, but the contributions of this paper compared to TLSTM are not significant enough for acceptance, and the presentation can be made more clear.

The paper addresses limitations of the LTSM method for modeling time series. The work is motivated by applications where multiple time series need to be combined while they may get updated in an asynchronous fashion. Authors mention IoT applications in the Intro and give examples from a power consumption data set and a churn prediction application in their numerical experiments sections. 
The paper's main contribution is to shrink down time series into 5 different categories which have different sampling schemes: (1) dense (2) sparse (3) decay (4) static decay (3) static standard, and proposing building blocks to incorporate these in a unified recurrent mechanism. 
The paper's motivation for introducing sparse input to LTSM was rather straightforward and convincing, and the churn prediction application is an excellent motivation for this. I had a harder time following why the 3 other types of features needed to be included as well at this point. Perhaps a more problem oriented explanation with concrete examples could have helped. Or those features should not yet be used as generalizations but kept for future work.
Overall I think the paper has several interesting ideas. Even though not all are as convincing, I think the paper is thought provoking and may interest the ICLR community.

Summary
========
The paper addresses the problem of irregular spacing in sequential data with five different irregularity types. The solution is based on modifying LSTM cell.

Comment
========
Irregular and multiple spacing presents in many real world applications with time-stamped events. The problem is therefore important to address properly. However, I found the evaluation of the proposed solution is less convincing. The main results in Figure 5 are not informative. 

There have been related works addressing irregular and multiple spacing (not just missing data as cited in the paper):

- Pham, T., Tran, T., Phung, D., & Venkatesh, S. (2016, April). DeepCare: A deep dynamic memory model for predictive medicine. In Pacific-Asia Conference on Knowledge Discovery and Data Mining (pp. 30-41). Springer, Cham.
- Koutnik, J., Greff, K., Gomez, F., & Schmidhuber, J. (2014). A clockwork RNN. arXiv preprint arXiv:1402.3511.
- Chen, C., Kim, S., Bui, H., Rossi, R., Koh, E., Kveton, B., & Bunescu, R. (2018, October). Predictive Analysis by Leveraging Temporal User Behavior and User Embeddings. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management (pp. 2175-2182). ACM.





[Relevance] Is this paper relevant to the ICLR audience? yes

[Significance] Are the results significant? somewhat

[Novelty] Are the problems or approaches novel? reasonably

[Soundness] Is the paper technically sound? yes, I think. I did not thoroughly check the equations.

[Evaluation] Are claims well-supported by theoretical analysis or experimental results? somewhat

[Clarity] Is the paper well-organized and clearly written? yes, except the experiments

Confidence: 2/5

Seen submission posted elsewhere: No (but I did find it on arXiv after writing the review)

Detailed comments:

In this work, the authors propose a new type of memory cell for RNNs which account for multiple types of time series. In particular, the work combines uniformly sampled observations (“normal” RNN input), time-decaying observations, non-decaying observations which may change, and static features which are not time-dependent. Empirically, the proposed approach outperforms an RNN with TLSTM cells in some cases.

=== Comments

I found the proposed approach for incorporating the different types of time series reasonable. This work definitely leans heavily on ideas from TLSTM and others, but, to the best of my knowledge, the specific combination and formulation is novel, especially concerning the “non-decaying time series” observations.

 However, I had difficult coming up with an example of a “static decay feature”. It would be helpful to give a concrete example of one of these in the main text. (It is also not clear to me why the difference in time between the time of the last event in a sequence and the prediction time for that sequence would be considered a “static decay” feature rather than just a “static” feature.)

My main concern with the paper is that the experimental design and results are not especially easy to follow; consequently, they are not as convincing as they might be. First, the sparsity mechanism is rather simple. In many domains (e.g., the medical domain considered in several of the cited papers), missingness is non-uniform and is often meaningful. While “meaningfulness” may be difficult to simulate, burstiness (non-uniformity) could be simulated. Second, for the groups, it is not clear whether all combinations of, e.g., 2 (informative) feature were sparsely sampled or if only one group of 2 was chosen. If the former, then some measure of variance should be given to help estimate statistical significance. Third, the particular classification task here is, essentially, forecasting one of the input variables. While that is certainly a relevant problem, many other time series classification or regression problems are not tied so directly to observations. It is not clear if these results are relevant in that setting.

=== Typos, etc.

The plots and figures in the paper are very difficult to read. Larger versions, or at least versions with increased fonts, should be used.


Meta-review: This paper presents an algorithm for combining various feature types when training recurrent networks. The features are handled by modifying the update rules and cell states based on the features' type -- dense, sparse, static, w/ decay, etc.

Strengths
- The model handles each feature according to its type and handles cell state and transitions appropriately. 
- Extends earlier work to handle more feature types, like sparse features.

Weaknesses
- Limited novelty. Models similar to various aspects of the proposed system have been presented in prior works. For example: TLSTM, which the authors use as a baseline. Although some components are novel, like the treatment of sparse features, contributions, in my opinion, are not sufficient to be accepted at ICLR.
- Presentation: Confusing and not enough information for reproducing results; multiple reviewers raised concerns about presentation of the feature types and experimental results. There were suggestions to improve, which the authors did consider during revision, but some concerns still remain.

In the end, the reviewers agreed about the limited novelty of this work, given existing literature. The recommendation, therefore, is to reject the paper. 


Decision: rejected-papers