Unified recurrent network for many feature types

There are time series that are amenable to recurrent neural network ( RNN ) solutions when treated as sequences , but some series , e.g . asynchronous time series , provide a richer variation of feature types than current RNN cells take into account . In order to address such situations , we introduce a unified RNN that handles five different feature types , each in a different manner . Our RNN framework separates sequential features into two groups dependent on their frequency , which we call sparse and dense features , and which affect cell updates differently . Further , we also incorporate time features at the sequential level that relate to the time between specified events in the sequence and are used to modify the cell ’ s memory state . We also include two types of static ( whole sequence level ) features , one related to time and one not , which are combined with the encoder output . The experiments show that the proposed modeling framework does increase performance compared to standard cells . 1 INTRODUCTION . The study of time series has a long history and the literature for it covers many different methods ( Hamilton ( 1994 ) ) . The study of asynchronous time series is an important subset of this . Asynchronous time series are series for which features are sampled at irregular time intervals , and at any given time step new values of any subset of features may be present . When a feature does not change values often it can be treated as being present only at times of change . For example , in industrial IoT several sensors can be monitored , each one with its own sampling rate . We can have a feature that corresponds to the production rate that seldom changes and it impacts predictions . This clearly presents difficulties from both a modeling and data input perspective . Fixed sampling , repeating missing values , and other types of data imputation have all been used to varying degrees of success . More recent attempts using machine learning for asynchronous time series use Gaussian processes ( Cunningham et al . ( 2012 ) ; Li & Marlin ( 2016 ) ) , which are hard to scale in the presence of many features . Deep learning is a much more recent development , and it is achieving great success in many fields , e.g . machine vision ( Krizhevsky et al . ( 2012 ) ; Szegedy et al . ( 2015 ) ; He et al . ( 2016 ) ) and natural language processing ( Wu et al . ( 2016 ) ; Yin et al . ( 2017 ) ) . Recurrent neural networks ( RNNs ) are a type of a neural network that use shared weights applied to sequences . These types of networks achieve state of the art performance for sequential data . Time series can be thought of as sequences in the RNN context , and thus they are amenable to RNN-based solutions ( Malhotra et al . ( 2015 ) ; Che et al . ( 2018 ) ; Längkvist et al . ( 2014 ) ) . However , many of the most widely used RNN cells , e.g . the Long Short Term Memory ( LSTM ) cell ( Hochreiter & Schmidhuber ( 1997 ) ) , are built for problems where time either does not matter or there is a constant time step size . The adaptation of these cells for asynchronous sequences is a more recent development , varying from combining traditional methods with deep learning ( Binkowski et al . ( 2018 ) ; Wen et al . ( 2017 ) ) to pure neural network approaches ( Neil et al . ( 2016 ) ; Baytas et al . ( 2017 ) ) . These attempts may take into account time between samples and the patterns of missing values , but they do not fundamentally treat features that are present with varying levels of frequency any differently . Thus , a feature that occurs at every time step and a feature that almost never occurs at any given time step update the cell in the same manner . We address this shortcoming by introducing a new recurrent cell that uses features that are present frequently , called dense features , differently from features that are rarely present , termed sparse features , when updating the cell ’ s hidden and memory states . In particular , our cell ’ s hidden state is split into two parts , one part for dense features and one part for sparse features . Additionally , each sparse feature maintains its own hidden and memory state that are updated only at time steps when that feature is present . Thus , the update for the sparse part of the overall hidden state depends on the subset of sparse features present at any given time step , while the update for the dense part of the overall hidden state happens in the standard LSTM manner . In addition to handling these two feature types , our cell also accounts for the irregular time between time steps , expanding on the work of Baytas et al . ( 2017 ) by allowing more flexibility for ingesting time features . Specifically , we modify the decay function to handle an arbitrary number of decay features , not just the time elapsed since the last time step . The model also permits sequence level features , distinguishing time related features , called static decay features , which are handled in the same manner as within the recurrent cell , from the non-time related features , called static standard features , which are embedded and concatenated with the sequence output in the standard manner . Overall , we support five feature types . The main contributions of this work are as follows . First , the introduction of a new recurrent cell that accounts for asynchronous feature sampling and treats features in a variable manner dependent on the frequency with which their values are present . Second , an expansion of the time-aware LSTM ( TLSTM ) to handle multiple time features , and the extension of this method applied to the output of the recurrent cell itself . Third and finally , a general recurrent framework to incorporate the five aforementioned feature types . 2 RELATED WORK . When dealing with an irregularly sampled multivariate sequence , one can consider the variables not present at any given time step to be missing . Prior work done on recurrent networks with missing data includes ( Bengio & Gingras ( 1996 ) ; Tresp & Briegel ( 1998 ) ) . Lipton et al . ( 2016 ) test the performance of different types of imputation and missing value indicator input features . This work inputs the missing values patterns into the recurrent network by concatenating with the feature vectors while using a standard LSTM network . They find zero filling missing features and using indicator features outperforms data imputation , which may actually interfere with the network ’ s ability to pick out missing values . Che et al . ( 2018 ) investigate a similar approach and improve upon this by modifying a GRU ( Cho et al . ( 2014 ) ) cell to separately incorporate the missing value patterns as well as the time between successive measurements for each feature . It does this by introducing two decay mechanisms . The first decay acts to bring a missing value toward the baseline mean value from its last measured value so the old value is not input repeatedly . This work treats all features in the same manner , regardless of the frequency a missing value is present for any given feature . The work in Che et al . ( 2018 ) does not differentiate between features that never have missing values and features where almost all values are missing . The second decay mechanism uses the time feature to decay the hidden state before calculating the new hidden state . We note that missing data at a time is not the same as features not being present ( sampled ) or a feature rarely changing value . We focus on the latter while Che et al . ( 2018 ) ; Bengio & Gingras ( 1996 ) ; Tresp & Briegel ( 1998 ) focus on the former . Baytas et al . ( 2017 ) incorporate the time between events for event based sequences with irregular intervals in a similar fashion to the second decay mechanism in Che et al . ( 2018 ) . This is done by decomposing the memory state into short and long term components , and then multiplying the short term component by a decay factor that decreases as the time between events increases . We use this in our own recurrent cell , and also expand upon the idea by incorporating multiple time features , not just limited to the time between adjacent events . We also use the same decomposition and decay method outside of our recurrent cell since the prediction time occurs some time after the final event . Their work does not consider sparse features . Neil et al . ( 2016 ) modify the standard LSTM cell to address its issues with asynchronous time series . They do this by incorporating a time gate into the network that controls when and to what extent the hidden and memory states can be updated . This means that there are time steps where no updates are made , and so the network accumulates changes in the feature vector before updating the hidden and memory states . While the time gate has learnable parameters , they still do not differentiate between features that occur at every time step versus those that rarely occur in the cell updates . In summary , there is no prior work that considers the notion of sparse features and a single model for all five feature types . 3 MODELS . The main contribution of this paper is the development of a recurrent cell which handles five distinct feature types . The following subsections review each of the feature types and describes how they are handled in the Sparse Time LSTM ( STLSTM model ) . The five types are referred to as the following : dense features , sparse features , decay features , static standard features , and static decay features . The dense features are the standard RNN sequence input , such as the embedding of an event type or the sequence state features at a given time step , and are present at any given time in a sequence . The sparse features change values infrequently , and thus they can be considered as being present only at select times . Stating that the value is not present means the feature value has not changed from the previous time . A further assumption is that these feature values do not decay with time . The decay features must be related to time measured with respect to some condition , such as time since the previous event in an event driven sequence . These features are particularly useful when events in the sequence are clustered , meaning that we observe many events in close proximity alternating with long gaps . Both the static standard and the static decay features give information about the sequence as a whole , rather than being tied to a specific time step . Static standard features are time invariant , e.g . the gender of a sequence ’ s object , whereas static decay features are time related , e.g . the time since last event in the sequence was observed . Figure 1 depicts these concepts . We consider a neural network to be a mapping f ( X ) , where f represents the network and X is the input to the network . We split X first into two parts , X = ( xstatic , x ) where xstatic represents the sequence level static features and x represents the sequence features . The static features can be split into two parts xstatic = ( xstatic , s , xstatic , ∆ ) , for static standard and static decay features respectively . The sequence features can further be broken down into three parts , x = ( xd , x∆ , xsp ) that represent the dense , decay , and sparse features respectively . For notational convenience , each of these features has a vector at each time step of the sequence ( even though in our implementation the sparse features would be used only when the value changes ) . The dense features are denoted as xd = ( xd1 , x d 2 , ... , x d T ) for a sequence with T time steps . The decay features are represented as x∆ = ( x∆1 , x ∆ 2 , ... , x ∆ T ) .The sparse features , however , are represented slightly differently than the previous two types of features , because their changes are not always present . We use the term present for a sparse feature when its value changes . A single sparse feature k can be represented as a sequence of tuples , xspk = ( ( m1k , x sv 1k ) , ... , ( mTk , x sv Tk ) ) , where mtk ∈ { 0 , 1 } is a mask value that is 1 when the feature is present and 0 if the feature is missing . If mtk = 1 , then xsvtk is the actual new feature value , and otherwise the value of xsvtk does not impact the feed forward computation . If a sparse feature is present at a time , then this input is passed into and used in the cell updates . If there is no sparse input at a given time step , the sparse part of the cell is untouched at that time step . Our recurrent cell is built off of the commonly used LSTM Hochreiter & Schmidhuber ( 1997 ) , as well as its time-aware variant ( TLSTM ) Baytas et al . ( 2017 ) . For LSTM-type cells , there is a memory state , C , and a hidden state , h , which pass from the cell at the previous time step and into the cell along with the features at the current time step . The memory state boosts performance for long term dependencies , and the gate structure of the LSTM allows selective parts of the memory to be forgotten and updated . The standard LSTM architecture does not address irregularly sampled events , however , and the TLSTM variant addresses this by using the time elapsed between events to modify the memory state . In the TLSTM cell , at each time step , first a fully connected layer is used to decompose the cell ’ s memory state into short and long term components . The time elapsed is passed as the input to a non-increasing decay function that maps to a scalar decay factor . The short term component of the memory state is multiplied by the decay factor that decreases as more time passes . The equations for a single decay feature x∆t for this process are below on the left , where g is a decay function , x ∆ t is the decay input , and Ct−1 is the previous cell memory state . After the memory state is modified , the cell update is performed in the standard LSTM manner , with the exception that the modified cell memory state calculated in Equation 1 is used to update the current memory state . The equations are as follows on the right where xdt is the feature vector to the current time step . A diagram of this base cell is given in Figure 2 . CSt−1 = tanh ( W∆Ct−1 + b ∆ ) ĈSt−1 = C S t−1 · g ( x∆t ) CLt−1 = Ct−1 − CSt−1 C∗t−1 = C L t−1 + Ĉ S t−1 , fdt = σ ( W d fhht−1 +W d fxx d t + b d f ) idt = σ ( W d ihht−1 +W d ixx d t + b d i ) C̃dt = σ ( W d Chht−1 +W d Cxx d t + b d C ) Cdt = f d t ∗ C∗t−1 + idt ∗ C̃dt , odt = σ ( W d ohht−1 +W d oxx d t + b d o ) hdt = o d t ∗ tanh ( Cdt )

Summary: This paper proposes Sparse Time LSTM, an extension of TLSTM to handle (the most importantly) the sparse features which is updated infrequently, and also the static features which does not change across the time. For sparse features, authors proposed an lazy update mechanism where the memory state is updated only at the time of a sparse feature update (i.e. when $m_{tk} = 1$), and then the hidden state / output gate are updated by aggregating over the hidden state proposals for all sparse features (using either average-/max-pooling or a MLP). For static features, the authors proposed inputing the feature embeddings only at the final time steps. The proposed method can also handle decay features by using the decay mechanism that already exists in TLSTM (although the authors proposed an extension to generalize the mechanism to vector input). The authors tested the proposed method on a synthetic dataset based on UCI electronic power consumption data, and a proprietary dataset for churn prediction.