Unified recurrent network for many feature types
Authors: Alexander Stec
Diego Klabjan
Jean Utke
Venue:  (2020)
Keywords: sparse
recurrent
asynchronous
time
series

Abstract: There are time series that are amenable to recurrent neural network (RNN) solutions when treated as sequences, but some series, e.g. asynchronous time series, provide a richer variation of feature types than current RNN cells take into account. In order to address such situations, we introduce a unified RNN that handles five different feature types, each in a different manner. Our RNN framework separates sequential features into two groups dependent on their frequency, which we call sparse and dense features, and which affect cell updates differently. Further, we also incorporate time features at the sequential level that relate to the time between specified events in the sequence and are used to modify the cell's memory state. We also include two types of static (whole sequence level) features, one related to time and one not, which are combined with the encoder output. The experiments show that the proposed modeling framework does increase performance compared to standard cells.

Under review as a conference paper at ICLR 2020
UNIFIED RECURRENT NEURAL NETWORK FOR MANY FEATURE TYPES
Anonymous authors Paper under double-blind review
ABSTRACT
There are time series that are amenable to recurrent neural network (RNN) solu- tions when treated sequences, but some series, e.g. asynchronous time series, provide a richer variation of feature types than current RNN cells take into ac- count. In order to address such situations, we introduce a unified RNN that han- dles five different feature types, each in different manner. Our RNN framework separates sequential features into two groups dependent on their frequency, which we call sparse and dense features, and which affect cell updates differently. Fur- ther, we also incorporate time features at the sequential level that relate the time between specified events in the sequence and are used to modify the cell's mem- ory state. We also include two types of static (whole sequence level) features, one related to time and one not, which are combined with the encoder output. The ex- periments show that the proposed modeling framework does increase performance compared to standard cells.
INTRODUCTION
The study of time series has a long history and the literature for it covers many different methods (Hamilton (1994)). The study of asynchronous time series is an important subset of this. Asyn- chronous time series are series for which features are sampled at irregular time intervals, and at any given time step new values of any subset of features may present. When a feature does change values often it can be treated as being present only at times of change. For example, in industrial loT several sensors can be monitored, each one with its own sampling rate. We can have a feature that corresponds to the production rate that seldom changes and it impacts predictions. This clearly presents difficulties from both a modeling and data input perspective. Fixed sampling, repeating missing values, and other types of data imputation have all been used to varying degrees of success. More recent attempts using machine learning for asynchronous time series use Gaussian processes Cunningham al. (2012): Li & Marlin (2016)), which are hard to scale in the presence of many features.
Deep learning is much more recent development, and it is achieving great success in many fields, e.g. machine vision (Krizhevsky et al. (2012); Szegedy et al. (2015); He et al. (2016)) and natural language processing (Wu al. (2016); Yin al. (2017)). Recurrent neural networks (RNNs) are a type of a neural network that use shared weights applied to sequences. These types of networks achieve state of the performance for sequential data. Time series can be thought of as sequences in the RNN context, and thus they are amenable to
solutions (Malhotra et al. (2015;; Che et al. 2018): Längkvist et al.] (2014). However. many of the most widely used RNN cells, e.g. the Long Short Term Memory (LSTM) cell (Hochreiter & Schmidhuber (1997)), are built for problems where time either does not matter or there is constant time step size.
The adaptation of these cells for asynchronous sequences is more recent development, varying from combining traditional methods with deep learning (Binkowski et al. 2018); Wen et al. to pure neural network approaches (Neil al. (2016); Baytas et al. (017)). These attempts may take into account time between samples and the patterns of missing values, but they do not fundamentally treat features that are present with varying levels of frequency any differently. Thus, feature that occurs at every time step and a feature that almost never occurs at any given time step update the cell in the same manner.

---

Under review as a conference paper at ICLR 2020
We address this shortcoming by introducing recurrent cell that uses features that are present frequently, called dense features, differently from features that are rarely present, termed sparse features, when updating the cell''s hidden and memory states. In particular, split into two parts, one part for dense features and one part for sparse features. Additionally, each sparse feature maintains its own hidden and memory state that are updated only time steps when that feature is present. Thus, the update for the sparse part of the overall hidden state depends on the subset of sparse features present at any given time step, while the update for the dense part of the overall hidden state happens in the standard LSTM manner.
cell's hidden state is
In addition handling these two feature types, our cell also accounts for the irregular time between time steps, expanding on the work of Baytas et al. (2017) by allowing more flexibility for ingesting time features. Specifically, we modify the decay function to handle an arbitrary number of decay features, not just the time elapsed since the last time step. The model also permits sequence level features, distinguishing time related features, called static decay features, which are handled in the same manner within the recurrent cell, from the non-time related features, called static standard features, which are embedded and concatenated with the sequence output in the standard manner. Overall, support five feature types.
The main contributions of this work are as follows. First, the introduction of new recurrent cell that accounts for asynchronous feature sampling and treats features in variable manner dependent on the frequency with which their values are present. Second, an expansion of the time-aware LSTM (TLSTM) to handle multiple time features, and the extension of this method applied to the output of the recurrent cell itself. Third and finally, a general recurrent framework to incorporate the five aforementioned feature types.
2 RELATED WORK
When dealing with an irregularly sampled multivariate sequence, one can consider the variables not present any given time step to be missing. Prior work done on recurrent networks with missing data includes (Bengio & Gingras (996);; Tresp & Briegel (1998)). Lipton et al. (2016) test the performance of different types of imputation and missing value indicator input features. This work inputs the missing values patterns into the recurrent network by concatenating with the feature vectors while using a standard LSTM network. They find zero filling missing features and using indicator features outperforms data imputation, which may actually interfere with the network's ability to pick out missing values.
Che et al. (2018) investigate a similar approach and improve upon this by modifying GRU (Cho) itt al. (2014)) cell to separately incorporate the missing value patterns as well as the time between successive measurements for each feature. It does this by introducing two decay mechanisms. The first decay acts to bring a missing value toward the baseline mean value from its last measured value so the old value is not input repeatedly. This work treats all features in the same manner, regardless of the frequency a missing value is present for any given feature. The work in Che et al. (2018) does not differentiate between features that never have missing values and features where almost all values are missing. The second decay mechanism uses the time feature to decay the hidden state before calculating the new hidden state. We note that missing data time is not the same as features not being present (sampled) or a feature rarely changing value. We focus on the latter while Che et al. 2018); Bengio & Gingras 1996): Tresp & Briegel 19988 focus on the former.
Baytas et al. (017)) incorporate the time between events for event based sequences with irregular intervals similar fashion the second decay mechanism in Che et al. (2018). This is done by decomposing the memory state into short and long term components, and then multiplying the short term component by a decay factor that decreases as the time between events increases. We use this in our own recurrent cell, and also expand upon the idea by incorporating multiple time features, not just limited to the time between adjacent events. We also use the same decomposition and decay method outside of our recurrent cell since the prediction time occurs some time after the final event. Their work does not consider sparse features.
Neil et al. 2016) modify the standard LSTM cell to address its issues with asynchronous time series. They do this by incorporating a time gate into the network that controls when and to what extent the hidden and memory states can be updated. This means that there time steps where updates
2

---

Under review as a conference paper at ICLR 2020
made, and so the network accumulates changes in the feature vector before updating the hidden and memory states. While the time gate has learnable parameters, they still do not differentiate between features that occur at every time step versus those that rarely occur in the cell updates.
In summary, there is no prior work that considers the notion of sparse features and a single model for all five feature types.
3 MODELS
The main contribution of this paper is the development of a recurrent cell which handles five distinct feature types. The following subsections review each of the feature types and describes how they are handled in the Sparse Time LSTM (STLSTM model). The five types are referred to as the following: dense features, sparse features, decay features, static standard features, and static decay features. The dense features are the standard RNN sequence input, such as the embedding of an event type or the sequence state features a given time step, and are present at any given time in a sequence. The sparse features change values infrequently, and thus they can be considered as being present only at select times. Stating that the value is not present means the feature value has not changed from the previous time. A further assumption is that these feature values do not decay with time. The decay features must be related to time measured with respect to some condition, such as time since the previous event in event driven sequence. These features are particularly useful when events in the sequence are clustered, meaning that we observe many events in close proximity alternating with long gaps. Both the static standard and the static decay features give information about the sequence as a whole, rather than being tied to a specific time step. Static standard features are time invariant, e.g. the gender of sequence's object, whereas static decay features are time related, e.g. the time since last event in the sequence was observed. Figure depicts these concepts.
09
Tanhi
x9
h4
taal erry beters
x1
Dear
Figure The memory state is decomposed into short and long term components, and then the standard LSTM updates are performed.
Figure I: The framework of the five feature types.
We consider a neural network to be mapping f(X), where f represents the network and X is the input to the network. We split X first into two parts, X = (æ.tattic, xx) where static represents the sequence level static features and represents the sequence features. The static features can be split into two parts staatic = (zstatic, stattcc,A), ), for static standard and static decay features respectively. The sequence features can further be broken down into three parts, that represent the dense, decay, and sparse features respectively. For notational convenience, each of these features has vector at each time step of the sequence (even though in our implementation the sparse features would be used only when the value changes). The dense features are denoted as ad (1r, .rd x4) for sequence with T time steps. The decay features are represented as - (If 22 + T.e sparse features, however, are represented slightly differently than the previous two types of features, because their changes are not always present. We use the term present for a sparse feature when its value changes. A single sparse feature k can be represented as sequence of tuples, 2t ((MIR, xff) (MTK, s77k)) )), where m4k (0, 1) is mask value that when the feature is present and the feature missing. If m& new feature value, and otherwise the value of zik does not impact the feed forward computation. If a sparse feature is present at a time, then this input is passed into and used in the cell updates. there is no sparse input at given time step, the sparse part of the cell is untouched at that time step.
(x x .PP)
then aft is the actual
Our recurrent cell built off of the commonly used LSTM Hochreiter & Schmidhuber (197), well as its time-aware variant (TLSTM) Baytas et al. (017)). For LSTM-type cells, there is memory state, C. and a hidden state, h. which pass from the cell at the previous time step and into the cell along with the features at the current time step. The memory state boosts performance for
3

---

Under review as a conference paper at ICLR 2020
long term dependencies, and the gate structure of the LSTM allows selective parts of the memory to be forgotten and updated. The standard LSTM architecture does not address irregularly sampled events, however, and the TLSTM variant addresses this by using the time elapsed between events to modify the memory state.
In the TLSTM cell, at each time step, first fully connected layer is used to decompose the cell's memory state into short and long term components. The time elapsed is passed as the input to a non-increasing decay function that maps to a scalar decay factor. The short term component of the memory state is multiplied by the decay factor that decreases as more time passes. The equations for single decay feature x, for this process are below on the left, where g a decay function, 2? is the decay input, and Ct-1 is the previous cell memory state.
After the memory state modified, the cell update is performed in the standard LSTM manner, with the exception that the modified cell memory state calculated in Equation I is used to update the current memory state. The equations are as follows on the right where 2y is the feature vector the current time step. A diagram of this base cell is given in Figure 2
f6 a ( + W Worg + +b9)
c = tanh C-1++
d = o (W 4- + Win + +b,))
Ō = Ch-1 (1
C (We, h4-1 We Wort bg)
Ch = Ct-1 - c.51
C f Ci-1 + it Cit.
C7- Ct-1 + C5-11
of (Woh 14-1 + vo + 1%)
1 = of * tanh((C))
3.1 DENSE FEATURES
The dense features are the standard features used for recurrent models. It is assumed that the features are not time decay related, c.g. they encode attributes of events, but not times between events. At each time step, the hidden state from the previous time step, the memory vector, and the feature vector for the current time step are input into the cell, and the hidden state and memory vector are updated for the next time step.
3.2 DECAY FEATURES
Decay features are not relevant or applicable for some RNN applications such as text or video, but for a sequence with measurements or events that occur with varying frequency, time information can add predictive power.
In the beginning of this section, model expressed by Equation is presented that allows a single decay feature. Here extend this to multiple decay features. Incorporating multiple decays quires a simple modification to the decay function g. Instead of mapping from a scalar to a scalar, maps from vector scalar. The proposed g function is g 22( where 0s
trainable vector.
3.3 SPARSE FEATURES
Sparse features are similar in nature to the dense features, with the exception that their values change very rarely throughout the sequence. Like the dense features, the sparse features describe the state of the sequence at particular time step. However, since they rarely change from one step to another, when treated in the same manner as dense features, the sparse features would just repeat inputs the vast majority of time steps. There is clear cutoff point in new value frequency for which dense feature turns into a sparse feature.
We propose a new recurrent cell, called the Sparse Time Long Short Term Memory (STLSTM) cell, that attempts to more strongly capture the change in sparse features. The STLSTM cell has a hidden state split into two parts, the sparse features, h2 = (hg, heP). Further, each sparse feature has its own memory state, and this memory state is only updated when there is change in the feature value. At each time step, the
corresponding the dense features, and the other corresponding to
4

---

Under review as a conference paper at ICLR 2020
dense part of the hidden state is updated based Equation [ and Equation 2. For the sparse part of the hidden state, there is a proposed hidden state from each sparse feature, and these proposed hidden states are aggregated together.
Let us denote the memory state sparse feature & as C. and the corresponding hidden state as hik- The main idea is to not change Ctk if mik 0, i.e. to simply carry it over, and otherwise if Mek to use the feature Iik within LSTM--lkee update equations. Formally, if mak = 0, then there is no change the memory state or the hidden state, that is,
Otk = 1-1,k* 1ig = h
If there is change in the feature value (mix 1), the new memory state is determined by the following equations:
lik
(W + f= +,7))
isp (W 14-1 + W Jit + (rth)
Cap = o(W
Ch 14-1 + W. ti bg)
sp C STh 1-1,k "k
Once the memory state calculated, the hidden state parts are calculated using standard output gates. The equations for this as follows:
o oW W wit + 6,5)
hi of tanh((Ck)
he [MC C(h (1) pm W ah
Ot [og, ((o Om W no)
Here C an aggregation function that yields a vector of the same dimension as hi and Wah Wa. are trainable parameters. For example, L can be simply averaging the hidden state proposals we can place a fully connected layer over all of the sparse hidden states, and use this layer to compute single hidden state for all sparse features. A diagram for the STLSTM cell is depicted in Figure
by foco
Merrg
Figure 3: Full wiring for the STLSTM cell. The sparse feature update follows one of two possible paths dependent on the masking.
Figure Static features are embedded and then concatenated to RNN output.
3.4 STATIC STANDARD FEATURES
All previous discussions dealt with features that can assigned to specific time step sequence, but we can also have features that are sequence level instead of time step level. These features, which are not time related, give information about the sequence as a whole, so they should not serve as inputs the recursive part.
The incorporation these features is fairly simple. First the features are passed through fully connected layer to get embedding, and then this embedding is concatenated with the output, hr, of the recurrent cell at the final time steps. A diagram this can he seen in Figure 4
3.5 STATIC DECAY FEATURES
Similar to the static standard features, the static decay features apply to the sequence whole rather than an individual time step. Their values are related to the time since the last observed event, e.g. the time between the final time step and the prediction time.
5

---

Under review as a conference paper at ICLR 2020
These features used in the same manner the decay features within the STLSTM cell. The out- put of the recurrent network, hy, is decomposed into short and long term components, and the decay factor is applied to the short term component before being added back to the long term component. Formally, this is given as
h = tanh (Watatic, hT + (otatic,
h = l (Istatic,A)
by lT h
hy =
h hig
This modified output is then concatenated with the embedding of the static standard features before input to the decoder.
COMPUTATIONAL STUDY
For all experiments in this section we use Keras with Tensorflow backend single GeForce GTX 1080 GPU card. We use ADAM as the optimization algorithm, and keep track of the validation FI score control the number of training epochs. If the validation F1 score does not increase for 15 epochs, then training is terminated. We use standard weight initialization, Glorot uniform for weights connected to the inputs and orthogonal initialization for the recurrent weights. To test out the STLSTM cell with all five feature types, we use two data sets, one public and one proprietary.
4.1 POWER CONSUMPTION DATA SET
The public data set is a modified version of the UCI household electric power consumption dataset (Dheeru & Karra Taniskidou (2017)). This data contains measurements for seven different elec- trical quantities and sub-metering values with sampling rate of one minute taken over the course of nearly four years. All of the features are sampled every minute, except for approximately 1.25% of the records which have no measurements. We simply fill in these missing values by repeating the record that immediately precedes it. Since the data set does not have natural sparse and decay features, we have to artificially create them, which is described in the Appendix.
For the experiments described in this section, we use sequences covering two hours worth of power data. The recurrent part of our network is composed of two stacked RNN cells. We compare using two architectures: one has STL.TMM at the bottom layer and LSTT at the second layer, and the other has TLSTM at the bottom layer and LSTT at the second layer. Experiments have shown that using two layers is optimal. We use a standard LSTM cell as the second layer instead of STLTSM or TLSTM cells because we assume the time and sparsity information is encoded in the output of the base layer that is fed into the second layer. These all have a hidden dimension of 64, which is also the size of the single embedding layer for the static standard features. Using same hidden layer size means that STLSTM cells have more parameters than the TLSTM cells, but preliminary experiments on TLSTM cells showed this is the optimal architecture for performance during inference. For all results the static features are present, unless it is stated otherwise.
The first metric we investigate is the relative performance of the STLSTM cell versus the standard TLSTM cell versus sparsity of individual sparse features. For this we use a dense layer as the aggregation function in the STLSTM cell. For each of the seven features, we set that feature to be sparse with sparse ratio ranging from 0.01 to 0.15 and the remaining six to be dense. For the TLSTM model, the sparse features are input at every time step as dense features. Figure 5 shows the relative performance on F1 scores between the TLSTM and STLSTM models for both types of record subsampling, with percentages greater than meaning the STLSTM performs better. The Fl score of STLSTM, which corresponds to the denominator, is approximately 0.7 but depends which feature is made to be sparse.
A common characteristic for all series is that the relative performance of the STLSTM cell increases as the sparsity increases, but only to a certain point where it then maintains or slightly drops. Further, STLSTM offers larger improvement in the case of the group sampling in most cases. As can be seen in Figure [5] the relative performance of each model depends on the feature we are treating as sparse. It is not surprising that the feature with the largest dependence sparsity is the voltage

---

Under review as a conference paper at ICLR 2020
Falativ
Spite 1930
Figure 5: Relative change in Fl score between STLSTM and TLSTM cell. For single sparse fea- tures, random sampling is first and group sampling second. Third is 2, 3, and 4 sparse features using group sampling. Fourth is relative performance different sparse subsets of features for the churn data set.
variable itself (represented by blue), since this is the feature we are making predictions on. The features with the smallest relative performance change are found have the smallest response to sparsity as well, which may indicate that they have less predictive power and thus it is not surprising that the difference between the TLSTM and STLSTM is also small.
Further than looking at individual sparse features, we also investigate groups of sparse features with different levels of sparsity. We report the results on groups sparse features with size 2, 3, and since it was shown that of the features are not affected by sparsity. Figure shows the relative performance for STLSTM and TLSTM models for groups of sparse features that were found to have the largest effect at a given group size and sparsity. Unsurprisingly, the characteristics of the series are the same as the analysis of single features. Using more sparse features does increase the relative performance of the STLSTM cell to point, but for this dataset where the features are closely related to one another the effectiveness is limited.
In addition to focusing the relative performance between the TLSTM and STLSTM cells, also consider the performance of different aggregation methods within the STLSTM cell. In particular, compare using dense layer, averaging aggregation, and maximum aggregation at four different levels of sparsity. The aggregation method clearly does not matter there only one sparse feature, and for this comparison we must use sets of sparse features. In particular, report results using the set of 3 sparse features that was found to give the largest relative improvement for STLSTM. The results are summarized in Table la based on group sampling datasets.
As can be seen in Table Ia the dense layer aggregation performs as well or better than the other two methods at every level of sparsity. The average and maximum aggregation have similar performance, perhaps slightly leaning toward the average method at low sparsity and the maximum method for higher sparsity.
Table 1: Aggregation and static feature analysis.
(a) F1 scores for STLSTM model with different aggregation methods for different levels of spar- sity.
(h) Averaged relative F1 score change after incor- porating one both types of static features on group sampling datasets.
Sparse ratio
Dense layer
Average
Max
Sparse ratio
Static standard
Static decay
Both
0.03
0.668
0.658
0.659
0.03
1.56
0.67
2.01
0.07
0.675
0.664
0.666
0.07
1.48
0.58
1.92
0.11
0.683
0.674
0.672
0.11
1.45
0.51
1.76
0.15
0.693
0.685
0.681
0.15
1.44
0.48
1.69
Finally, we study the effect of adding both types of static features to the model. Table Tb shows the average relative improvements in F1 score for adding in one or both types of static features for different subsets of sparse features with varying levels of sparsity. is clear from Table that both types of features improve the performance of the model, with the static standard features having a larger effect. This larger effect may be specific to the features of the power consumption data, but generally both types of features improve performance when properly integrated into the model. With respect to sparsity, both types of static features improve the model more as the features become more sparse, with the static decay features being more dependent sparsity.
7

---

Under review as a conference paper at ICLR 2020
It is worth discussing the impact of the added complexity of the STLSTM cell on training time. With STLSTM, as the number of sparse features grows, so too does the number of parameters in the model. This makes training slower per epoch, but it is mitigated in wall-clock training time by the fact that it takes fewer epochs to reach convergence. Using standard TLSTM, an epoch of training time takes about 10 seconds with the given sequence parameters for the datasets. When using four sparse features, this increases to about 18 seconds per epoch. However, the STLSTM cell generally reaches convergence in 40-50 epochs while the TLSTM cell requires 70-80 epochs.
4.2 REAL WORLD CHURN DATA SET
In addition to this constructed dataset, also use real world proprictary dataset with naturally sparse features for churn prediction within a future time span. At each time step, there is an event embedding is treated a dense feature. In addition to this, there are approximately 60 additional sequence state features. These state features have varying levels of sparse ratios, ranging from 0.03 to 0.6. We experiment with different subsets of dense and sparse features using the sequence state features. Additionally, we use two decay features, one static standard feature, and one static decay feature. For this dataset we use a training set with 700,000 samples and validation//est sets with 200,000 samples each. Sequence length ranges from I to 150, but the samples are not evenly distributed by sequence length. The distribution exponential, with large portion of sequences on the lowest end of that range and the mean is approximately 40.
For this data set, we compare two stacked architectures composed of three stacked RNN cells. As before, the bottom layer uses either STLSTM or TLSTM, and the higher levels use standard LSTM cells. When multiple sparse features are present, we use dense layer as the aggregation method since it performs best. All results are given relative to the standard TLSTM model with all 60 additional sequence state features treated as dense features, as this was found to outperform an all LSTM model by approximately 2% under the all dense setting.
Since this data set is naturally sparse, we do not tune the sparsity, but instead investigate the relative performance for different subsets of sparse and dense features. There four qualitative feature subsets we primarily study. The first sparse subset contains features that relate to relative changes, e.g. a customer can upgrade/dowwgrrade his service to different tier. The second sparse subset includes true counts for feature values, e.g. the total number of people associated with an account. These two subsets the most sparse and have average sparsity of 0.065. The other two feature subsets contain features relating customer specific (c.g. a customer relocates) and sequence specific information (e.g. the number of emails to customer service). The relative performance for these sparse feature subsets is found in Figure 5 We observe that using only relative change features gives the smallest performance gain. This is not surprising because in this case a repeated value of has meaning and the features are specifically designed to capture changes, somewhat mitigating the intended effect of the STLSTM cell. Using the raw counts does give a larger performance increase.
Both customer and sequence specific subsets show the largest improvement, indicating that these sub groups are responsive to STLSTM architecture. It is interesting that these two subgroups do not necessarily contain only the most sparse features, but mix between sparsity ratios roughly from 0.03 to 0.3. This suggests that in real applications with naturally sparse features there is a large range of sparsity that can occur in the sparse sub group of features.
4.3 FUTURE WORK
For future work, it would be interesting to incorporate even more feature types than the five covered in this work. One in particular is a feature type that gives time information looking forward in the sequence. All features in this work use time information related to past events, but there are cases that can benefit from the utility of incorporating future knowledge when available. One example of this is the time to the prediction from the current time step so the network can have direct knowledge of its absolute time location in the sequence.
REFERENCES
Inci M Baytas, Cao Xiao, Xi Zhang, Fei Wang, Anil K Jain, and Jiyy Zhou. Patient subtyping via time-aware LSTM networks. In Proceedings of the 23rd ACM SIGKDD international conference
8

---

Under review as a conference paper at ICLR 2020
on knowledge discovery and data mining, pp. 65-74,2011
Yoshua Bengio and Francois Gingras. Recurrent neural networks for missing or asynchronous data. In Advances in neural information processing systems, pp. 395 401, 1996.
Mikolaj Binkowski, Gautier Marti, and Philippe Donnat. Autoregressive convolutional neural net- works for asynchronous time series. In Proceedings of the 35th international conference on machine learning, volume 80, pp. 579-588, 2018.
Zhengping Che, Sanjay Purushotham, Kyunghyun Cho, David Sontag, and Yan Liu. Recurrent neural networks for multivariate time series with missing values. Scientific reports, 8(1):6085, 2018.
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation.
2014.
John Cunningham, Zoubin Ghahramani, and Carl Rasmussen. Gaussian processes for time-marked time-series data. In Artificial intelligence and statistics, pp. 255 263, 2012.
Dua Dheeru and En Karra Taniskidou. 7/archive. ics... uuw..ed/mml
UCI machine learning repository, 2017. URL http:
James Douglas Hamilton. Time series analysis, volume 2. Princeton university press, Princeton, NJ, 1994.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Martin Längkvist, Lars Karlsson, and Amy Loutfi. A review of unsupervised feature learning and deep learning for time-series modeling. Pattern recognition letters, 42:11-24 2014.
Steven Cheng-Xian Xian Li and Benjamin M Marlin. A scalable end-to-end Gaussian process adapter for irregularly sampled time series classification. In Advances in neural information processing systems, pp. 1804-1812, 2016.
Zachary C Lipton, David Kale, and Randall Wetzel. Directly modeling missing data in sequences with RNNs: Improved classification of clinical time series. In Machine learning for healthcare conference, pp. 253-270, 2016.
Pankaj Malhotra, Lovekesh Vig, Gautam Shroff, and Puneet Agarwal. Long short term memory networks for anomaly detection time series. In European symposium on artificial networks, computational intelligence, and machine learning, pp. 89-93 2015.
Daniel Neil, Michael Pfeiffer, and Shih-Chii Liu. Phased LSTM: Accelerating recurrent network training for long event-based sequences. Advances in neural information processing systems, pp. 3882-3890, 2016
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1-9, 2015.
Volker Tresp and Thomas Briegel. A solution for missing data in recurrent neural networks with an application to blood glucose prediction. In Advances in neural information processing systems, pp. 971-977,777 1998.

---

Under review as a conference paper at ICLR 2020
Ruofeng Wen, Kari Torkkola, and Balakrishnan Narayanaswamy, A multi-horizon quantlle recurrent
forecaster. arXiv:1711.1105.. 2017.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, and Klaus Macherey. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv: 2016.
Wenpeng Yin, Katharina Kann, Mo Yu, and Hinrich Schütze. Comparative study of CNN and RNN for natural language processing. 77X.h:/702.0/9900 2017.
10

---

Under review as a conference paper at ICLR 2020
5 APPENDIX
The power consumption dataset one long sequence, and so we break this down into many se- quences. We create the sequences based on three parameters: sample time T,, shift window, and sequence length T'. To create the first sequence we first set the start index to the first record. A full sequence is created by selecting Tg records beginning at the start index. We further subsample from the T, records in order to obtain final sequences with decay features in two ways. In the first way we randomly sample individual records. The first record in the full sequence is always kept, but after the first record we randomly drop records based on Bernoulli until there are T remaining. In the second way, we create groups of records. We start with the first records from the start index. Next focus records from start index 8 to start index T. We select random record and the surrounding 4 records. We then repeat the procedure until we have T selected records. We finish with groups of 5 consecutive records that do not overlap. In either way we sample, this means the sampling rate is no longer a uniform one minute, and we use the time between two adjacent events to be the decay feature. The seven electrical features are dense features present every time step in the sequence. The start index is then increased by the shift window and the process continues until we run through the entire dataset. If the shift window is less than the sample time, then there can be some overlap in the sequences. We allow these overlaps in the training set, but remove any sequences the validation and test sets with subsequences duplicated in the training set.
Once the sequences are all created, we can create sparse features on any subset of the seven dense features. This is done by choosing sparse ratio for each feature in the sparse subset, where the sparse ratio is the ratio of feature values that are kept in each sequence. We keep feature values present with uniform probability equal to the sparse ratio for that feature. If a feature value is not kept at a time step, its mask value is set to 0 at the time step. This allows us to easily experiment with different numbers of sparse features and the extent of the sparsity.
Both types of static features are also used for this dataset. There is one static decay feature, and it the time elapsed between the last event of the sequence and the prediction time for that sequence, which is T3 events from the time of the first event. There are three categorical static standard features, and these are the day of the week, day of the month, and time of day (morning, afternoon, evening, spans two hours less, and assign the static standard
night). Each of the sequences feature values based on the first event in the sequence. Figure depicts these concepts.
T time steps
Sampled sequence
A1
Al
At
à static
Full sequence
T, time steps
Main sequence
Start
Index
Figure 6: The creation of sequences from the electric power consumption dataset.
The training target for the power consumption dataset is multi-class classification target. We predict whether the average voltage over some time interval after the prediction time of a sequence stays within half a standard deviation, increases by more than half a standard deviation, or decreases by more than half a standard deviation when compared to the value averaged over the time span of the sequence measured from the first event to T, time steps after the first event. The standard deviation value here is taken over the entire training set. This is an unbalanced classification task. While the exact majority class percentage varies depending on the way the sequences are created, falls between 65% and 80% for our experiments. For the experiments described in Section 4.1 we use two hours of data corresponding to T, 120. We also use shift window of 30 and set T 50.
11


Reviews:
[{"review_id": "rJgzT_0HqS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This submitted manuscript is exactly the paper (bearing no difference) that was submitted to ICLR 2019 and also rejected. \nThis submitted manuscript is exactly the paper (bearing no difference) that was submitted to ICLR 2019 and also rejected. \nThis submitted manuscript is exactly the paper (bearing no difference) that was submitted to ICLR 2019 and also rejected. \nThis submitted manuscript is exactly the paper (bearing no difference) that was submitted to ICLR 2019 and also rejected. \nThis submitted manuscript is exactly the paper (bearing no difference) that was submitted to ICLR 2019 and also rejected. \nThis submitted manuscript is exactly the paper (bearing no difference) that was submitted to ICLR 2019 and also rejected. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "r1eA6IK99H", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes Sparse Time LSTM, an extension of TLSTM to handle (the most importantly) the sparse features which is updated infrequently, and also the static features which does not change across the time. For sparse features, authors proposed an lazy update mechanism where the memory state is updated only at the time of a sparse feature update (i.e. when $m_{tk} = 1$), and then the hidden state / output gate are updated by aggregating over the hidden state proposals for all sparse features (using either average-/max-pooling or a MLP). For static features, the authors proposed inputing the feature embeddings only at the final time steps. The proposed method can also handle decay features by using the decay mechanism that already exists in TLSTM (although the authors proposed an extension to generalize the mechanism to vector input). The authors tested the proposed method on a synthetic dataset based on UCI electronic power consumption data, and a proprietary dataset for churn prediction.\n\nI vote to reject the paper in its current form. I believe the paper is tackling an important task (extending neural network methods for sparse and missing time-series data), and I find the proposed methods intuitive and the experiment evaluations interesting. However, I find the technical description of the proposed method insufficient, in the sense that (1) there does not seem to be explanation about how the proposed mechanism is better than the previous method (e.g. intuition or theoretical justification of why handling sparse update using the TLSTM approach is inferior), or (2) no justification about the particular choice of the model (e.g. in Section 3.2, why choose this particular form of g)? Also, the experiment section does not seem to contain sufficient detail to be reproducible (e.g., how many simulation runs were conducted?), and the reported results do not contain standard error information which makes it difficult to decide the significance of the improvement. The organization of the paper can also be improved (see Major comments). In summary, the paper in its current form does not yet reach the ICLR standard due to the lack of clarity in its technical description, and lack of sufficient detail in experiment results to be convincing. This paper can benefit greatly with sufficient justification of its choice of model mechanisms, a more careful explanation of the advantage of its approach to sparse feature relative to previous approaches, and also a self-contained description of the experiment procedures and standard error information for the experiments.\n\nAlthough the proposed method is of limited novelty (a straightforward extension to TLSTM) and the proposed methods for static/decay features feels somehow trivial, I feel this work is still interesting since it discusses an interesting topic. For future improvements, in addition to incorporating reviewers' comments from past submissions,  authors can consider shaping it into an in-depth study of how LSTM approaches should best handle sparse features. \n\n\n\nMajor Comments:\n\nThe organization of the method section can be improved. Earlier descriptions of the TLSTM should become a separate section called Related Work or Background, or at least being clearly marked as previous work. The proposed STLSTM method, which is the major innovation of this paper, should be introduced in the beginning of the Method section (rather than being delayed to Section 3.3).\n\n\nA more detailed explanation of experiment protocol is needed. What is group sampling, and how many times were the experiments repeated? On a related note, in Table 1a-1b, please also report the standard error of F1 scores across multiple runs. This is needed to access whether the improvements brought by dense layer / additional features are significant.\n\nMinor Comments:\n\nPlease add equation numbers to all equations.\n\nEquation on page 5, please explain explicitly what is being aggregated over by the aggregation function L. Is the hidden states for all sparse features?\n\nThe titles and labels for the experiment result figure (Figure 5) are too small to be readable. Also there's no legend for the first two figures. What does each colored line mean? My guess is they are corresponding to the seven features, however author should explain this clearly in the figure caption or in the text description.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "H1lDRVew5B", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I have checked the paper that AnonReviewer1 pointed out and verified that the two papers are identical. It would be better to reject the paper. \n\nI would also appreciate authors to consider the comments from the previous ICLR-19 reviews and improve the paper. \n\nI also think that the desk rejection would be an appropriate decision for this paper. \n\nAt least it would be good to add the following papers as pointed by one of review last year.\n- Pham, T., Tran, T., Phung, D., & Venkatesh, S. (2016, April). DeepCare: A deep dynamic memory model for predictive medicine. In Pacific-Asia Conference on Knowledge Discovery and Data Mining (pp. 30-41). Springer, Cham.\n- Koutnik, J., Greff, K., Gomez, F., & Schmidhuber, J. (2014). A clockwork RNN. arXiv preprint arXiv:1402.3511.\n- Chen, C., Kim, S., Bui, H., Rossi, R., Koh, E., Kveton, B., & Bunescu, R. (2018, October). Predictive Analysis by Leveraging Temporal User Behavior and User Embeddings. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management (pp. 2175-2182). ACM.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I did not assess the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: main summary: sparse time LSTM

discussions;
reviewer 4: technical description of the proposed method insufficient,
reviewer 2, 3: same paper sent to ICLR 2019 and rejected
recommendation: rejected, based on all reviewers comments
Decision: Reject