Time2Vec: Learning a Vector Representation of Time

1 INTRODUCTION . In building machine learning models , “ time ” is often an important feature . Examples include predicting daily sales for a company based on the date ( and other available features ) , predicting the time for a patient ’ s next health event based on their medical history , and predicting the song a person is interested in listening to based on their listening history . The input for problems involving time can be considered as a sequence where , rather than being identically and independently distributed ( iid ) , there exists a dependence across time ( and/or space ) among the data points . The sequence can be either synchronous , i.e . sampled at regular intervals , or asynchronous , i.e . sampled at different points in time . In both cases , time may be an important feature . For predicting daily sales , for instance , it may be useful to know if it is a holiday or not . For predicting the time for a patient ’ s next encounter , it is important to know the ( asynchronous ) times of their previous visits . Recurrent neural networks ( RNNs ) do not typically treat time itself as a feature , typically assuming that inputs are synchronous . When time is known to be a relevant feature , it is often fed in as yet another input dimension ( Choi et al. , 2016 ; Du et al. , 2016 ; Li et al. , 2018b ) . In practice , RNNs often fail at effectively making use of time as a feature . To help the RNN make better use of time , several researchers design hand-crafted features of time that suit their specific problem and feed those features into the RNN ( Choi et al. , 2016 ; Baytas et al. , 2017 ; Kwon et al. , 2019 ) . Hand-crafting features , however , can be expensive and requires domain expertise about the problem . Many recent studies aim at obviating the need for hand-crafting features by proposing generalpurpose—as opposed to problem specific—architectures that better handle time ( Neil et al. , 2016 ; Zhu et al. , 2017 ; Mei & Eisner , 2017 ; Hu & Qi , 2017 ; Upadhyay et al. , 2018 ; Li et al. , 2018a ) . We follow an orthogonal but complementary approach to these recent studies by developing a generalpurpose model-agnostic representation for time that can be potentially used in any architecture . In particular , we develop a learnable vector representation ( or embedding ) for time as a vector representation can be easily combined with many models or architectures . We call this vector representation Time2Vec . To validate the effectiveness of Time2Vec , we conduct experiments on several ( synthesized and real-world ) datasets and integrate it with several architectures . Our main result is to show that on a range of problems and architectures that consume time , using Time2Vec instead of the time itself offers a boost in performance . 2 RELATED WORK . There is a long history of algorithms for predictive modeling in time series analysis . They include auto-regressive techniques ( Akaike , 1969 ) that predict future measurements in a sequence based on a window of past measurements . Since it is not always clear how long the window of past measurements should be , hidden Markov models ( Rabiner & Juang , 1986 ) , dynamic Bayesian networks ( Murphy & Russell , 2002 ) , and dynamic conditional random fields ( Sutton et al. , 2007 ) use hidden states as a finite memory that can remember information arbitrarily far in the past . These models can be seen as special cases of recurrent neural networks ( Hochreiter & Schmidhuber , 1997 ) . They typically assume that inputs are synchronous , i.e . arrive at regular time intervals , and that the underlying process is stationary with respect to time . It is possible to aggregate asynchronous events into time-bins and to use synchronous models over the bins ( Lipton et al. , 2016 ; Anumula et al. , 2018 ) . Asynchronous events can also be directly modeled with point processes ( e.g. , Poisson , Cox , and Hawkes point processes ) ( Daley & Vere-Jones , 2007 ; Laub et al. , 2015 ; Xiao et al. , 2017 ; Li et al. , 2018a ; Xiao et al. , 2018 ) and continuous time normalizing flows ( Chen et al. , 2018 ) . Alternatively , one can also interpolate or make predictions at arbitrary time stamps with Gaussian processes ( Rasmussen , 2004 ) or support vector regression ( Drucker et al. , 1997 ) . Our goal is not to propose a new model for time series analysis , but instead to propose a representation of time in the form of a vector embedding that can be used by many models . Vector embedding has been previously successfully used for other domains such as text ( Mikolov et al. , 2013 ; Pennington et al. , 2014 ) , ( knowledge ) graphs ( Grover & Leskovec , 2016 ; Nickel et al. , 2016 ; Kazemi & Poole , 2018 ) , and positions ( Vaswani et al. , 2017 ; Gehring et al. , 2017 ) . Our approach is related to time decomposition techniques that encode a temporal signal into a set of frequencies ( Cohen , 1995 ) . However , instead of using a fixed set of frequencies as in Fourier transforms ( Bracewell & Bracewell , 1986 ) , we allow the frequencies to be learned . We take inspiration from the neural decomposition of Godfrey & Gashler ( 2018 ) ( and similarly ( Gashler & Ashmore , 2016 ) ) . For time-series analysis , Godfrey & Gashler ( 2018 ) decompose a 1D signal of time into several sine functions and a linear function to extrapolate ( or interpolate ) the given signal . We follow a similar intuition but instead of decomposing a 1D signal of time into its components , we transform the time itself and feed its transformation into the model that is to consume the time information . Our approach corresponds to the technique of Godfrey & Gashler ( 2018 ) when applied to regression tasks in 1D signals , but it is more general since we learn a representation that can be shared across many signals and can be fed to many models for tasks beyond regression . While there is a body of literature on designing neural networks with sine activations ( Lapedes & Farber , 1987 ; Sopena et al. , 1999 ; Wong et al. , 2002 ; Mingo et al. , 2004 ; Liu et al. , 2016 ) , our work uses sine only for transforming time ; the rest of the network uses other activations . There is also a set of techniques that consider time as yet another feature and concatenate time ( or some hand designed features of time such as log and/or inverse of delta time ) with the input ( Choi et al. , 2016 ; Li et al. , 2017 ; Du et al. , 2016 ; Baytas et al. , 2017 ; Kwon et al. , 2019 ; Trivedi et al. , 2017 ; Kumar et al. , 2018 ; Ma et al. , 2018 ) . Kazemi et al . ( 2019 ) survey several such approaches for dynamic ( knowledge ) graphs . These models can directly benefit from our proposed vector embedding , Time2Vec , by concatenating Time2Vec with the input instead of their time features . Other works ( Neil et al. , 2016 ; Zhu et al. , 2017 ; Mei & Eisner , 2017 ; Hu & Qi , 2017 ; Upadhyay et al. , 2018 ; Li et al. , 2018a ) propose new neural architectures that take into account time ( or some features of time ) . We show how Time2Vec can be used in one of these architectures to better exploit temporal information ; it can be potentially used in other architectures as well . 3 BACKGROUND & NOTATION . We use lower-case letters to denote scalars , bold lower-case letters to denote vectors , and bold upper-case letters to denote matrices . We represent the ith element of the vector r as r [ i ] . For two vectors r and s , we use [ r ; s ] to represent their concatenation and r s to represent element-wise ( Hadamard ) multiplication of the two vectors . Throughout the paper , we use τ to represent a scalar notion of time ( e.g. , absolute time or time from the last event ) and τ for a vector of time features . Long Short Term Memory ( LSTM ) ( Hochreiter & Schmidhuber , 1997 ) is considered one of the most successful RNN architectures for sequence modeling . A formulation of the original LSTM model and a variant of it based on peepholes ( Gers & Schmidhuber , 2000 ) is presented in Appendix C. When time is a relevant feature , the easiest way to handle time is to consider it as just another feature ( or extract some engineered features from it ) , concatenate the time features with the input , and use the standard LSTM model ( or some other sequence model ) ( Choi et al. , 2016 ; Du et al. , 2016 ; Li et al. , 2018b ) . In this paper , we call this model LSTM+T . Another way of handling time is by changing the formulation of the standard LSTM . Zhu et al . ( 2017 ) developed one such formulation , named TimeLSTM , by adding time gates to the architecture of the LSTM with peepholes . They proposed three architectures namely TLSTM1 , TLSTM2 , TLSTM3 . A description of TLSTM1 and TLSTM3 can be found in Appendix C ( we skipped TLSTM2 as it is quite similar to TLSTM3 ) . 4 TIME2VEC . A common approach to deal with time in different applications is to apply some hand-crafted function ( s ) f1 , . . . , fm to τ ( τ can be absolute time , time from last event , etc . ) , concatenate the outputs f1 ( τ ) , . . . , fm ( τ ) with the rest of the input features x , and feed the resulting vector [ x ; f1 ( τ ) ; . . . ; fm ( τ ) ] to a sequence model ( see Section 2 for references ) . This approach requires hand-crafting useful functions of time which may be difficult ( or impossible ) in several applications , and the hand-crafted functions may not be optimal for the task at hand . Instead of hand-crafting functions of time , we devise a representation of time which can be used to approximate any function through learnable parameters . Such a representation offers two advantages : 1- it obviates the need for hand-crafting functions of time and 2- it provides the grounds for learning suitable functions of time based on the data . As vector representations can be efficiently integrated with the current deep learning architectures , we employ a vector representation for time . Our proposed representation leverages the Fourier sine series ( Arfken & Weber , 1999 ) according to which any 1D function can be approximated in a given interval using a weighted sum of sinusoids with appropriate frequencies ( and phase-shifts ) . We include k sinusoids of the form sin ( ωiτ + ϕi ) in our vector representation where ωi and ϕi are learnable parameters1 . That is , we concatenate the input features x with k sinusoids and feed the concatenation [ x ; sin ( ω1τ +ϕ1 ) ; . . . ; sin ( ωkτ +ϕk ) ] into a sequence model . Different functions of time can be created using these sinusoids by taking a weighted sum of them with different weights . We allow the weights of the sequence model to combine the sinusoids and create functions of time suitable for the task . If we expand the output of the first layer of a sequence model ( before applying an activation function ) , it has the form : a ( τ , k ) [ j ] = γj + ∑k i=1 θj , i sin ( ωiτ + ϕi ) , where θj , is are the first layer weights and γj is the part of output which depends on the input features x ( not on the temporal features ) . Each a ( τ , k ) [ j ] operates on the input features x as well as a learned function fj ( τ ) = ∑k i=1 θj , i sin ( ωiτ + ϕi ) of time , as opposed to a hand-crafted function2 . Following Godfrey & Gashler ( 2018 ) , to facilitate approximating functions with non-periodic patterns and help with generalization , we also include a linear projection of time in our vector representation . We name our vector representation of time Time2Vec . Time2Vec of τ , denoted as t2v ( τ ) , is a vector of size k + 1 defined as follows : t2v ( τ ) [ i ] = { ωiτ + ϕi , if i = 0. sin ( ωiτ + ϕi ) , if 1 ≤ i ≤ k. ( 1 ) where t2v ( τ ) [ i ] is the ith element of t2v ( τ ) and ωis and ϕis are learnable parameters . The use of sine functions is inspired in part by Vaswani et al . ( 2017 ) ’ s positional encoding . Consider a sequence of items ( e.g. , a sequence of words ) { I1 , I2 , . . . , IN } and a vector representation vIj ∈ Rd for the jth item Ij in the sequence . Vaswani et al . ( 2017 ) added sin ( j/10000k/d ) to vIj [ k ] if k is even and sin ( j/10000k/d + π/2 ) if k is odd so that the resulting vector includes information about the position of the item in the sequence . These sine functions are called the positional encoding . Intuitively , positions can be considered as the times and the items can be considered as the events happening at that time . Thus , Time2Vec can be considered as representing continuous time , instead of discrete positions , using sine functions . The sine functions in Time2Vec also enable capturing periodic behaviors which is not a goal in positional encoding . We feed Time2Vec as an input to the model ( or to some gate in the model ) instead of adding it to other vector representations . Unlike positional encoding , we show in our experiments that learning the frequencies and phase-shifts of sine functions in Time2Vec result in better performance compared to fixing them . 1k can be treated as a hyper-parameter . 2The function can model a real Fourier signal when the frequencies ωi of the sine functions are integer multiples of a base ( first harmonic ) frequency . However , we show in Section 5.2 that learning the frequencies results in better generalization .

Summary: This paper introduces a particular learnable vector representation of time which is applicable across problems without the use of a hand-crafted time representation. Their representation makes use of a feed-forward layer with sine activations which operates on time data. As it is a vector representation, it combines well with other deep neural network methods. They motivate their problem well, explaining why time data is important to a variety of problems and situate their solution as an orthogonal approach to many current solutions in the literature. They make reference to fourier analysis as motivation for their representation. Finally, they provide experimental results to support their claims using fabricated and real-world time series datasets, as well as ablation studies to support their design decisions.