Time2Vec: Learning a Vector Representation of Time
Authors: Seyed Mehran Kazemi
Rishab Goel
Sepehr Eghbali
Janahan Ramanan
Jaspreet Sahota
Sanjay Thakur
Stella Wu
Cathal Smyth
Pascal Poupart
Marcus Brubaker
Venue:  (2020)

Abstract: Time is an important feature in many applications involving events that occur synchronously and/or asynchronously. To effectively consume time information, recent studies have focused on designing new architectures. In this paper, we take an orthogonal but complementary approach by providing a model-agnostic vector representation for time, called Time2Vec, that can be easily imported into many existing and future architectures and improve their performances. We show on a range of models and problems that replacing the notion of time with its Time2Vec representation improves the performance of the final model.

Under review as a conference paper at ICLR 2020
TIME2VEC: LEARNING A VECTOR REPRESENTATION OF TIME
Anonymous authors Paper under double-blind review
ABSTRACT
Time is an important feature in many applications involving events that occur synchronously and/or asynchronously. To effectively consume time information, recent studies have focused designing new architectures. In this paper, we take an orthogonal but complementary approach by providing a modellaggnsstc vector representation for time, called Time2 Vec, that can be easily imported into many existing and future architectures and improve their performances. We show on a range of models and problems that replacing the notion of time with its Time2 representation improves the performance of the final model.
INTRODUCTION
In building machine learning models, "time" is often an important feature. Examples include predicting daily sales for a company based on the date (and other available features), predicting the time for a patient's 's next health event based on their medical history, and predicting the song person interested listening to based on their listening history. The input for problems involving time can be considered as sequence where, rather than being identically and independently distributed (iid), there exists a dependence across time (and/or space) among the data points. The sequence can be either synchronous, i.e. sampled at regular intervals, asynchronous, i.e. sampled at different points in time. In both cases, time may be an important feature. For predicting daily sales, for instance, may be useful know if it is holiday not. For predicting the time for a patient's next encounter, is important to know the (asynchronous) times of their previous visits.
Recurrent neural networks (RNNs) do not typically treat time itself a feature, typically assuming that inputs are synchronous. When time is known to be a relevant feature, it is often fed in as yet another input dimension (Choi al. 2016 Du et al. 2016; Li al. 2018b). In practice, RNNs often fail at effectively making use of time as feature. To help the RNN make better use of time, several researchers design hand-crafted features of time that suit their specific problem and feed those features into the RNN (Choi et al., [016 Baytas et al.) 2017: Kwon et al. (019). Hand- Hand-crafting features, however, can be expensive and requires domain expertise about the problem.
Many recent studies aim at obviating the need for hand-crafting features by proposing general- purpose-st as opposed to problem specific architectures that better handle time (Neil et al., 2016) Zhu et al. 2017 Mei & Eisner 2017 Hu & Qi, 2017 Upadhyay et al. 2018 Li al. 2018a). We follow an orthogonal but complementary approach these recent studies by developing a general- purpose model-agnostic representation for time that be potentially used in any architecture. In particular, we develop a learnable vector representation (or embedding) for time as a vector representation be casily combined with many models architectures. We call this vector representation Time2 Vec. To validate the effectiveness of Time2Vec, we conduct experiments on several (synthesized and real-world) datasets and integrate it with several architectures. Our main result is to show that on a range of problems and architectures that consume time, using Time2Ve Vec instead of the time itself offers boost in performance.
RELATED WORK
There is long history of algorithms for predictive modeling in time series analysis. They include auto-regressive techniques (Akaike 1969) that predict future measurements in sequence based

---

Under review as a conference paper at ICLR 2020
window of past measurements. Since it is not always clear how long the window of past measurements should be, hidden Markov models (Rabiner & Juang/ (986). dynamic Bayesian networks (Murphy & Russell, 2002), and dynamic conditional random fields (Sutton et al., 2007) use hidden states as finite memory that can remember information arbitrarily far in the past. These models can be seen as special cases of recurrent neural networks (Hochreiter & Schmidhuber 1997). They typically assume that inputs are synchronous, i.e. arrive at regular time intervals, and that the underlying process is stationary with respect time. It is possible to aggregate asynchronous events into time-bins and to use synchronous models over the bins (Lipton et al., 2016 Anumula et al., 2018). Asynchronous events can also be directly modeled with point processes (e.g.. Poisson, Cox, and Hawkes point processes) (Daley & Vere-Jones] 2007: Laub et al., 2015 Xiao et al., 2017 Li et al.) 20188a Xiao ct al. 2018) and continuous time normalizing flows (Chen et al. 2018). Alternatively, one can also interpolate or make predictions at arbitrary time stamps with Gaussian processes (Rasmussen 2004)
or support vector regression Drucker et al. 1997).
Our goal is not to propose new model for time series analysis, but instead to propose representation
of time in the form of a vector embedding that can be used by many models. Vector embedding has
been previously successfully used for other domains such as text (Mikolov et al. 2013 Pennington
iet al. 2014). (knowledge) graphs (Grover & Leskovee| [0166] Nickel et al. 2016 Kazemi & Poole
2018). and positions (Vaswani 2017]
2017: Gehring al. 2017). Our approach is related time
decomposition techniques that encode a temporal signal into a set of frequencies (Cohen 1995).
However, instead of using a fixed set of frequencies as in Fourier transforms (Bracewell & Bracewell 1986), we allow the frequencies to be learned. We take inspiration from the neural decomposition of Godffey & Gashler (2018) (and similarly (Gashler & Ashmore 2016)). For time-series analysis, Godfrey& & Gashler (2018) decompose ID signal of time into several sine functions and linear function to extrapolate (or interpolate) the given signal. We follow a similar intuition but instead of decomposing a ID signal of time into its components, we transform the time itself and feed its transformation into the model that is to consume the time information. Our approach corresponds to the technique of Godffey & Gashler (2018) when applied to regression tasks in ID signals, but is more general since we learn a representation that can be shared across many signals and can be fed to many models for tasks beyond regression.
While there is a body of literature on designing neural networks with sine activations Lapedes & Farber 1987; (Sopena et al. 1999 Wong al. 2002 Mingo et al. 2004 Liu et al., (2016), our work uses sine only for transforming time; the rest of the network uses other activations. There is also a set of techniques that consider time as yet another feature and concatenate time (or some hand designed features of time such as log and/or inverse of delta time) with the input (Choi et al., 20166] Li et al.
2017 Du et al. 2016 Baytas al. 2017 Kwon et al., 2019 Trivedi al.) [211]] Kumar et al. 2018 Ma et al. 2018). Kazemi et al. (2019) survey several such approaches for dynamic (knowledge) graphs. These models can directly benefit from our proposed vector embedding, Time2 by concatenating Teme2 with the input instead of their time features. Other works (Neil et al. 2016 Zhu et al.] 2017 Mei & Eisner. 2017: Hu & Qi 2017; Upadhyay et al.) 2018 Li et al. 2018a) propose new neural architectures that take into account time (or some features of time). We show how Time2 Vec can be used in one of these architectures to better exploit temporal information; it can be potentially used in other architectures as well.
3 BACKGROUND & NOTATION
We use lower-case letters to denote scalars, bold lower-case letters to denote vectors, and bold upper-case letters denote matrices. We represent the 7th element of the vector r as r(i]. For two vectors r and 8, we use [r: 8) to represent their concatenation and r @ s to represent element- (Hadamard) multiplication of the two vectors. Throughout the paper, we use T to represent a scalar notion of time (e.g., absolute time or time from the last event) and T for a vector of time features.
Long Short Term Memory (LSTM) (Hochreiter & Schmidhuber 1997 is considered one of the most successful RNN architectures for sequence modeling. A formulation of the original LSTM model and a variant of it based on peepholes (Gers & Schmidhuber, 2000) is presented in Appendix C When time is relevant feature, the easiest way to handle time is consider it just another feature (or extract some engineered features from it), concatenate the time features with the input, and use the standard LST model (or some other sequence model) (Choi et al. 2016 Du et al. 2016 Li et al.
2

---

Under review as a conference paper at ICLR 2020
2018b). In this paper, we call this model LSTM+T. Another way of handling time is by changing the formulation of the standard LSTM. Zhu et al. (2017) developed one such formulation, named TImeLSTM, by adding time gates to the architecture of the LSTM with peepholes. They proposed three architectures namely TLSTMI, TLSTM2, TLSTM3. A description of TLSTM/ and 7ISTM3 can be found in Appendix C skipped TLSTM2 as it is quite similar to TLSTM3).
4 TIME2VEC
A common approach to deal with time in different applications is to apply some hand-crafted function(s) fm....;fm to T (T can be absolute time, time from last event, etc.), concatenate the outputs fi(T).... [T) with the rest of the input features and feed the resulting vector [x; (T): Sm (T)) to a sequence model (see Section 2 for references). This approach requires hand-craftinn useful functions of time which may be difficult (or impossible) in several applications, and the hand-crafted functions may not be optimal for the task at hand. Instead of hand- hand-crafting functions of time, we devise a representation of time which can be used to approximate any function through learnable parameters. Such representation offers two advantages: 1- it obviates the need for hand-crafting functions of time and 2- it provides the grounds for learning suitable functions of time based on the data. As vector representations can be efficiently integrated with the current deep learning architectures, we employ vector representation for time.
Our proposed representation leverages the Fourier sine series (Artken & Weber 1999) according to which any ID function can be approximated in given interval using a weighted sum of sinusoids with appropriate frequencies (and phase-shifts). We include l sinusoids of the form sin(w;r + pi) in our vector representation where wi and yo are learnable parameter That is, we concatenate the input features æ with l sinusoids and feed the concatenation [x; sin(w17 + :. + yx))] into a sequence model. Different functions of time can be created using these sinusoids by taking weighted sum of them with different weights. We allow the weights of the sequence model to combine the sinusoids and create functions of time suitable for the task. If we expand the output of the first layer of a sequence model (before applying an activation function), it has the form: a(r, k)[i = 7 + + 0g.1 sin (WIT + sp:), where 05 are the first layer weights and % is the part of output which depends on the input features x (not on the temporal features). Each a(T. k]] operates on the input features x as well as a learned function [5(T) >>> 0j sin (wir spe) of time, opposed to hand-crafted functior Following Godfrey & Gashler (2018), to facilitate approximating functions with non-periodic patterns and help with generalization, we also include a linear projection of time in our vector representation. We name our vector representation of time Time2Vec Time2 Vec of r, denoted as 12v(T), is vector of size k + 1 defined as follows:
WiT + yPs: sin (Wer + pi),
if = 0. if ki
12v (T)
(1)
where (2v is the 7th element of t2v(7) and wis and fo:s are learnable parameters.
The use of sine functions is inspired in part by Vaswani et al. (2017)ss positional encoding. Consider sequence of items (e.g., a sequence of words) {11, 12, .... [N] and a vector representation V1, E Rd for the jth item I in the sequence. Vaswani et al. (2017) added sin (j/ (j/10000*/4) to v1, [k] if A is T/ r/2) if k is odd so that the resulting vector includes information about the position of the item in the sequence. These sine functions are called the positional encoding. Intuitively, positions can be considered as the times and the items be considered as the events happening at that time. Thus, Time2Vec can be considered as representing continuous time, instead of discrete positions, using sine functions. The sine functions in Time2Vec also enable capturing periodic behaviors which is not goal in positional encoding. We feed Time2Vec as an input to the model (or some gate in the model) instead of adding other vector representations. Unlike positional encoding, we show in our experiments that learning the frequencies and phase-shifts of sine functions in Time2Vee result in better performance compared to fixing them.
even and sin (j L
k can he treated as 'The function can model a real Fourier signal when the frequencies wi of the sine functions are integer multiples of a hase (first harmonic) frequency. However, we show in Section 5.2 that learning the frequencies results in better generalization.
3

---

Under review as a conference paper at ICLR 2020
4.1 PROPERTIES OF TIME2
We review some of the interesting and desired properties of Time2Vec.
Periodicity: In many scenarios, some events occur periodically. The amount of sales of store, for instance, may be higher on weekends or holidays. Weather condition usually follows a periodic pattern over different seasons (Gashler Ashmore 2016). Some other events may be non-periodic but only happen after a point in time and/or become more probable as time proceeds. For instance, some diseases are more likely for older ages.
The period of sin (WeT + (Ps) is 2m i.e. it has the same value for T and + 2r, Therefore, the sine functions in Time2Vec help capture periodic behaviors without the need for feature engineering. For instance, a sine function sin (WT ip) with w 2 repeats every 7 days (assuming T indicates days) and can be potentially used to model weekly patterns. Furthermore, unlike other basis functions which may show strange behaviors for extrapolation (see, e.g., (Poole et al. (0144), sine functions are expected to work well for extrapolating to future and out of sample data (Vaswani et al.) 2017). The linear term represents the progression of time and can used for capturing non-periodic patterns in the input that depend on time.
Invariance to Time Rescaling: Since time can be measured in different scales (e.g., days, hours, seconds, etc.), another important property of representation for time is invariance time rescaling (see, e.g., (Tallec & Ollivier 2018).. A class of models is invariant to time rescaling if for any model M1 C and any scalar a >> O, there exists model M2 E C' that behaves on at (r scaled by a)) in the same way M1 behaves on original TS. Proposition T establishes the invariance TimeTee2ee Vec to time rescaling. The proof is in Appendix D
Proposition 1. Time2 is invariant to time rescaling.
A representation for time should be easily consumable by different models and archi- tectures. A matrix representation, for instance, may be difficult to consume as it cannot be easily appended with the other imputs. By selecting a vector representation for time, we ensure easy integration with deep learning architectures.
Simplicity:
EXPERIMENTS & RESULTS
We use the following datasets:
1) Synthesized data: We create a toy dataset to use for explanatory experiments. The inputs in this dataset are the integers between 1 and 365. Input integers that are multiples of 7 belong to class one and the other integers belong to class two. The first 75% is used for training and the last 25% for testing. This dataset is inspired by the periodic patterns (e.g., weekly or monthly) that often exist in daily-collected data; the input integers can be considered as the days.
2) Event-MNIST::
Sequential (event- based) MNIST is a common benchmark in sequence modeling literature (see, e.g., (Bellec et al.) 2018 Campos et al.] 2018 Fatahi et al.) (0166). We create sequential event-based version of MNIST by flattening the images and recording the position of the pixels whose intensities are larger than a threshold (0.9 in our experiment). Following this transformation, each image will be represented
an array of increasing numbers such as [f, f2, f3,..,,, f.]. We consider these values as the event times and use them to classify the images. As in other sequence modeling works, our aim in building this dataset not to beat the otate-of-the-art on the MNIST dataset; our aim is to provide dataset where the only input is time and different representations for time can be compared when extraneous variables (confounders) are eliminated as much as possible.
3) TIDIGITSIS (Anumula et al. 2018): The dataset includes audio spikes of the TIDIGITS spoken digit dataset (Leonard & Doddington, 1993) recorded by the binaural 64-channel silicon cochlea sensor. Each sample is a sequence of (t,c) tuples where ł represents time and c denotes the index of active frequency channel at time t. The labels are sequences of I to 7 connected digits with a vocabulary consisting of 11 digits (i.e. "zero" to "nine" plus "oh") and the goal is to classify the spoken digit based on the given sequence of active channels. We use the reduced version of the dataset where only the single digit samples are used for training and testing. The reduced dataset has total of 2,464 training and 2, 2,486 test samples.
4

---

Under review as a conference paper at ICLR 2020
Tecthen
24.0
170
Tarader
(a) Event-
Event-MNIST MNIST
(b) Raw N. TIDIGIITS
(c) Stack Overflow
622
LSTH Temslee
0.00
0.00
615
022
10
Epech
Opech
(d) LM
(c) CiteULike
Figure 1: Comparing LSTM+T and LSTM+Time2W6 on several datasets.
4) Stack Overflow (SOF): This dataset contains sequences of badges obtained by stack overflow users and the timestamps which the badges were obtained We used the subset released by Du ct al. (2016) containing ~ 6K users, 22 event types (badges), and ~ 480K events. Given sequence [66 ) (6% ( ..., (6%, [ for each user u where 6y is the badge id and E is the timestamp when received this badge id, the task is to predict the badge the user will obtain at time *+11
5) Fas This dataset contains a history of listening habits for LM users (Celma 2010). We used the code released by Zhu et al. (2017) to pre-process the data. The dataset contains 1K users, 5000 event types (songs), and ~ 819K events. The prediction problem is similar to the SOF dataset but with dynamic updating (see, (Zhu et al. 2017) for details).
6) CiteULikee: This dataset contains data about what and when a user posted on citeulike website") iT
The original dataset has about 8000 samples. Similar FM, used the pre-processing used by Zhu et al. (017)) to select ~ 1.6K sequences with 5000 event types (papers) and ~ 36A events. The task for this dataset similar to that for Last.FM.
For classification tasks, we report accuracy corresponding to the percentage of correctly classified examples. For recommendation tasks, we report Recall@ q and MRR@q. Following Zhu et al. (2017), to generate recommendation list, we sample k - 1 random items and add the correct item the sampled list resulting in list of items. Then our model ranks these & items. Looking only at the top ten recommendations, Recall @ q corresponds to the percentage of recommendation lists where the correct item is the top q: MRR @q (reported in Appendix [] corresponds to the mean of the inverses of the rankings of the correct items where the inverse rank is considered 0 if the item does not appear in top q recommendations. For FM and CiteULike, following Zhu et al. (2017) report Recall @ 10 and MRR @ 10. For SOF, report Recall @ 3 and MRR as there only 22 event types and Recall @ 10 and MRR @ 10 are not informative enough. The detail of the implementations is presented in Appendix A
Mcasures:
5.1 ON THE EFFECTIVENESS OF TIMB2VEC
Fig. represents the obtained results of comparing LSTM+Time2Vec with LSTM+T several datasets with different properties and statistics. On all datasets, replacing time with Time2Vec improves the performance in most cases and never deteriorates it; in many cases, LSTM+TTmme2 performs consistently better than LSTM+T. AAnMull Anumula al. (2018) mention that I.STM+T fails on N_TIDIGITSIS 18 the dataset contains very long sequences. By feeding better features to the LSTM rather than relying on the LSTM to extract them, Time2 Vec helps better optimize the LSTM and
5

---

Under review as a conference paper at ICLR 2020
Lain
042
ILa MLL
328
0:13
322
0.31
Expc
Docch
(a) TLSTMI, LastFM
(b) TLSTMI, CiteULike
0.4
3330
2025
2244
30
20
30
40
30
28
20
30
(c) SLMTM3, Last.M
(d) TLSTM3, CiteULike
Figure 2: Comparing TLSTMI and TLSTM3 on Last.FM and CiteULike in terms of Recall @ 10 with and without Teme2Vec.
offers higher accuracy (and lower variance) compared to LSTM+T. Besides N. N.TIDIGITSIS SOF also contains somewhat long sequences and long time horizons. The results on these two datasets indicate that Vec can be effective for datasets with long sequences and time horizons.
To verify if Time2 Vec can be integrated with other architectures and improve their performance, we integrate it with TLSTMI and TLSTM3, two recent and powerful models for handling asynchronous events. We replaced their notion of time with (2v(T) and replaced the vectors getting multiplied to with matrices accordingly. The updated formulations are presented in Appendix C The obtained results in Fig. 2 for TLSTMI and TLSTM3 on FM and CiteULike demonstrates that replacing time with Time2 Vec for both TLSTMI and TLSTM3 improves the performance.
5.2 MODEL VARIANTS & ABLATION STUDY
Other activation functions: Inspired by Fourier sine series and by positional encoding, we used sine activations in Eq. [ To evaluate how sine activations compare to other activation functions for our setting, we repeated the experiment on Event-MNIST in Section 5.1 when using non-periodic activations such as Sigmoid, Tanh, and rectified linear units (ReLU) (Nair & Hinton, 2010). and periodic activations such as mod and triangle. We fixed the length the Time2Vec to 64 1, f.e. 64 units with non-linear transformation and 1 unit with linear transformation. From the results shown in Fig. aia,, it can be observed that the periodic activation functions (sine, mod, and triangle) outperform the non-periodic ones. Other than not being able capture periodic behaviors, we believe one of the main reasons why these non periooii activation functions do not perform well is because as time goes forward and becomes larger, Sigmoid and Tanh saturate and ReLU either goes zero or explodes. Among periodic activation functions, sine outperforms the other two.
Fixed frequencies and phase-shifts: Vaswani et al. (2017) mention that learning sine frequencies and phase-shifte for their positional encoding gives the same performance as fixing frequencies to cenneentiall-deecyyiin
and 2. This raises the question of whether learning the sine frequencies and phase-shifts of Time2 from data offer any advantage compared to fixing them. To answer this question, we compare three models on Event-MNIST when using Teme2Vec of length 16 - 1: 1- fixing (2v(7) to sin 2am for < 16, 2- fixing the frequencies and phase shifts according to Vaswani et al. (201))ss positional encoding, and 3- learning the frequencies and phase-shifts from the data. Fig. 5 b) represents our obtained results. The obtained results in Fig. 5)b) show that learning the frequencies and phase-shift rather than fixing them helps improve the performance of the model.
values and phase-shile shises
16
Modeling Periodic Behaviours:
To measure how well Time2 performs in capturing periodic behaviours, trained model our synthesized dataset where the input integer (day) is used as

---

Under review as a conference paper at ICLR 2020
-25
-20
Test
(a) weighted sum of the sinusoids Time2 Vec oscillating every 7 days.
(b) A weighted sum the sinusoids in
Time2Vec oscillating every 14 days.
The models learned for our synthesized dataset before the final activation. The red dots represent the points to be classified as 1.
Figure 3:
the time for Time2 Vec and fully connected layer is used on top of the Time2 to predict the class. That is, the probability of one of the classes is a sigmoid of a weighted sum of the Time2Vec elements. Fig. (a) shows the learned function for the days in the test set where the weights, frequencies and phase-shifts are learned from the data. The red dots on the figure represent multiples of 7. It can be observed that Time2 Vec successfully learns the correct period and oscillates every 7 days. The phase-shifts have been learned in a way that all multiples of 7 are placed on the positive peaks of the signal facilitate separating them from the other days. Looking the learned frequency and phase-shift for the sine functions across several runs, we observed that in many runs one of the main sine functions has a frequency around 0.898 s ? and a phase-shift around 1.56 g thus learning to oscillate every 7 days and shifting by 2 to make sure multiples of 7 end up at the peaks of the signal. Fig. shoos the initial and learned sine frequencies for one run. It can be viewed that the beginning, the weights and frequencies are random numbers. But after training, only the desired frequency (2)) has a high weight (and the frequency which gets subsumed into the bias). The model perfectly classifies the examples in the test set which represents the sine functions in Time2Vec can be used effectively for extrapolation and out of sample times assuming that the test set follows similar periodic patterns the train sef We added some noise to our labels by flipping 5% of the labels selected at random and observed a similar performance in most runs.
To test invariance time rescaling, multiplied the inputs by 2 and observed that in many runs, the 2T frequency of one of the main sine functions was around 0.448 = 2+7 thus oscillating every 14 days. An example of a combination of signals learned to oscillate every 14 days is in Fig. (b).
The use of periodicity in sine functions: has been argued that when sine activations are used, only a monotonically increasing (or decreasing) part of it is used and the periodic part is ignored (Giambattita Parascandolo) 2017). When we use Teme2 however, the periodicity of the sine functions are also being used and seem to be key the effectiveness of the Time2Vec representation. Fig.5(c) 5(c) shows some statistics on the frequencies learned for Event-MNIST where we count the num- ber of learned frequencies that fall within intervals of lengths 0.1 centered at [0.05, 0.15, .... 0.95] (all learned frequencies are between 0 and 1). The figure contains two peaks at 0.35 and 0.85. Since the input the sine functions for this problem can have a maximum value of 784 (number
$ Replacing sine with a non-periodic activation function resulted always predicting the majority class.
K10
9.N-
23
S.4
06
0.
03
0.5
Figure 4: (a) Initial vs. (b) learned weights and frequencies for our synthesized dataset.
7

---

Under review as a conference paper at ICLR 2020
LIT atheliiin coople
LET alrepood
(a)
30
0.41
TUSTN -l meavee, wi Fner
035
623
Spech
(c)
An ablation study of several components in Time2Vec. (a) Comparing different activation functions for Teme2Vec Event-MNIST. Sigmoid and Tanh almost overlap. (b) Comparing fre- quencies fixed to cqually-spaced values, frequencies fixed according to positional encoding (Vaswani al. 2017), and learned frequencies Event-MNIST. (c) A histogram of the frequencies learned in Time2 Vec for ENNSS. The x-axis represents frequency intervals and the y-axis represents the number of frequencies in that interval. (d) The performance of #LSTM3+TTmm2 on CiteULike in terms of Recall @ 10 with and without the linear term.
Figure 5:
of pixels in an image), sine functions with frequencies around 0.35 and 0.85 finish (almost) 44 and 106 full periods. The smallest learned frequency is 0.029 which finishes (almost) 3.6 full periods. These values indicate that the model indeed using the periodicity of the sine functions, not just monotonically increasing (or decreasing) part of them.
The Linear Term: To see the effect of the linear term in Time2Vec, we repeated the experiment for Event- when the linear term is removed from Time2VecWi observed that the results not affected substantially, thus showing that the linear term may not be helpful for Event-MNIST. This might be due to the simplicity of the Event-MNIST dataset. Then we conducted similar experiment for TLSTM3 on CiteULike (which is more challenging dataset) and obtained the results in Fig, (dd). From these results, can see that the linear term helps facilitate lcarning functions of time that can be effectively consumed by the model.
CONCLUSION & FUTURE WORK
In many tasks for synchronous and asynchronous event predictions, time is important feature. Previous work has mainly resorted to applying hand-crafted functions to time and concatenating these functions with the rest of the input features. In this work, we presented approach that automatically learns these functions from data. In particular, we developed Time2Vec. a vector representation for time, using sine and linear activations and showed the effectiveness of this representation across several datasets and several tasks. In the majority of our experiments, Time2 Vec improved our results, while the remaining results were not hindered by its application. While sine functions have been argued to complicate the optimization (Lapedes & Farber 1987, Giambattista Parascandolo 2017). we did not experience such a complication except for the experiment in Subsection [5.2 on synthesized dataset when using only few sine functions. We hypothesize that the main reasons include combining sine functions with a powerful model (e.g., LSTM) and using many sine functions which reduces the distance to the goal (see, e.g., (Neyshabur et al. 2019)). We leave deeper theoretical analysis of this hypothesis, development of better optimizers, and experimenting with other representations for time future work.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Hirotugu Akaike. Fitting autoregressive models for prediction. Annals of the instutute of Statistical Mathematics, 21(1):243-247, 1969.
Jithendar Anumula, Daniel Neil, Tobi Delbruck, and Shih-Chii Liu. Feature representations for neuromorphic audio spike streams. Frontiers in neuroxcience, 12:23, 2018.
George B Arfken and Hans J Weber. Mathematical methods for physicists, 1999.
Inci M Baytas, Cao Xiao, Xi Zhang, Fei Wang, Anil K Jain, and Jiayu Zhou. Patient subtyping via time-aware lstm networks. In ACM SIGKDD, pp. 65-74, 2017.
Guillaume Bellec, Darjan Salaj, Anand Subramoney, Robert Legenstein, and Wolfgang Maass. Long short-term memory and learning-to-learn in networks of spiking neurons. In NeurlPS, 2018.
Ronald Newbold Bracewell and Ronald N Bracewell. The Fourier transform and its applications. McGraw-Hill New York, 1986.
Víctor Campos, Brendan Jou, Xavier Giró-i Nieto, Jordi Torres, and Shih-Fu Chang. Skip mnn: Learning to skip state updates in recurrent neural networks. ICLR, 2018.
O. Celma. Music Recommendation and Discovery in the Long Tail. Springer, 2010.
Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud, Neural ordinary differential equations. In Neural Information Processing Systems (NeurIPS), 2018.
Edward Choi, Mohammad Taha Bahadori, Andy Schuetz, Walter Stewart, and Jimeng Sun. Doctor
Al: Predicting clinical events via recurrent neural networks. In Machine Learning for Healthcare Conference, pp. 301-318, 2016.
Leon Cohen. fime-frequnncy analysis, volume 778. Prentice hall, 1995.
Daryl Daley and David Vere-Jones. An introduction to the theory of point processes: volume II: general theory and structure. Springer Science & Business Media, 2007.
Harris Drucker, Christopher JC Burges, Linda Kaufman, Alex Smola, and Vladimir Vapnik. Support vector regression machines. In NeurlPS, pp. 155-161, 1997.
Nan Du, Hanjun Dai, Rakshit Trivedi, Utkarsh Upadhyay, Manuel Gomez-Rodrigues and Le Song. Recurrent marked temporal point processes: Embedding event history to vector. In ACM SIGKDD, pp. 1555-1564. ACM, 2016.
Mazdak Fatahi, Mahmood Ahmadi, Mahyar Shahsavari, Arash Ahmadi, and Philippe Devienne. evt.mnist: A spike based version of traditional mnist. arXiv preprint arXiv: 2016.
Michael S Gashler and Stephen Ashmore. Modeling time series data with deep fourier neural networks. Neurocomputing, 188:3-11, 2016.
Jonas Gehring, Michael Auli, David Grangier. Denis Yarats, and Yann N Dauphin. Convolutional sequence to sequence learning. arXiv preprint
2017.
Felix A Gers and Jürgen Schmidhuber. Recurrent nets that time and count. IJCNN, volume 3, pp. 199-194. IEEE, 2000.
Tuomas Virtanen Giambattista Parascandolo. Heikki Huttunen. Taming the waves: sine activa- tion function deep neural networks. 2017. URL https:/ http::::://ppereeee....o wttp://www..//tdd/dddddd Sks32F9ea 9eg
Luke B Godfrey and Michael S Gashler. Neural decomposition of time-series data for effective generalization. IEEE transactions on neural networks and learning systems, (9(7):2973-2983 2018.
Klaus Greff, Rupesh K Srivastava, Jan Koutník, Bas R Steunebrink, and Jürgen Schmidhuber. Lstm: A search space odyssey. IEEE transactions on neural networks and learning systems, 28(10): 0222-2232, 2017.

---

Under review as a conference paper at ICLR 2020
Aditya Grover and Jure Leskovec. node2vec: Scalable feature learning for networks. In ACM SIGKDD, pp. 2016.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Hao Hu and Guo-Jun Qi. State-fre memory recurrent neural networks. In International Conference on Machine Learning, pp. 1568-1577, 2017.
Seyed Mehran Kazemi and David Poole. SimplE embedding for link prediction in knowledge graphs. In NeurlPS, pp. 4289 4300, 2018.
Seyed Mehran Kazemi, Rishab Goel, Kshitij Jain, Ivan Kobyzev, Akshay Sethi, Peter Forsyth, and Pascal Poupart. Relational representation learning for dynamic (knowledge) graphs: survey. arXiv preprint arXiv: 1485, 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Srijan Kumar, Xikun Zhang, and Jure Leskovec. Learning dynamic embedding from temporal interaction networks. arXiv preprint arXiv:/812.02289, 2018.
Bum Chul Kwon, Min-Je Choi, Joanne Taery Kim, Edward Choi, Young Bin Kim, Soonwook Kwon, Jimeng Sun, and Jaegul Choo. Retainvis: Visual analytics with interpretable and interactive recurrent neural networks electronic medical records. IEEE transactions visualization and computer graphics, 25(1):299-3300) 2019.
Alan Lapedes and Robert Farber. Nonlinear signal processing using neural networks: Prediction and system modelling. Technical report, 1987.
Patrick J Laub, Thomas Taimre, and Philip K Pollett. Hawkes processes. arXiv preprint 1rXiv:1507.02822, 2015.
R Gary Leonard and George Doddington. Tidigits. Linguistic Data Consortium, Philadelphia, 1993.
Shuang Li, Shuai Xiao, Shixiang Zhu, Nan Du, Yao Xie, and Le Song. Learning temporal point processes via reinforcement learning. In NeurlPS, pp. 10804-10814, 2018a.
Yang Li, Nan Du, and Samy Bengio. Time-dependent representation for neural event sequence prediction. arXiv preprint arXiv: 1100.00080053 2017.
Yang Li, Nan Du, and Samy Bengio. Time-dependent representation for neural event sequence prediction. 2018b. URL
Zachary C Lipton, David Kale, and Randall Wetzel. Directly modeling missing data in sequences with rnns: Improved classification of clinical time series. Machine Learning for Healthcare Conference, pp. 253-270, 2016.
Peng Liu, Zhigang Zeng, and Jun Wang- Multistability of recurrent neural networks with non- monotonic activation functions and mixed time delays. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 46(4):512-523, 2016.
Yao Ma, Ziyi Guo, Zhaochun Ren, Eric Zhao, Jiliang Tang, and Dawei Yin. Streaming graph neural networks. arXiv preprint arXiv: 18/0. /0627, 2018.
Hongyuan Mei and Jason M Eisner. The neural hawkes process: A neurally self- self-modulating multivariate point process. In NeurIPS, pp. 6754-6764, 2017.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. NewrIPS, 2013.
Luis Mingo, Levon Aslanyan, Juan Castellanos, Miguel Diaz, and Vladimir Riazanov, Fourier neural networks: An approach with sinusoidal activation functions. 2004.
10

---

Under review as a conference paper at ICLR 2020
Kevin Patrick Murphy and Stuart Russell. Dynamic bayesian networks: representation, inference and learning. 2002.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In ICML, pp. 807-814, 2010.
Daniel Neil, Michael Pfeiffer, and Shih-Chii Liu. Phased Istm: Accelerating recurrent network training for long or event-based sequences. In NeurlPS, pp. 3882-3890, 2016.
Behnam Neyshabur, Zhiyuan Li, Srinadh Bhojanapalli, Yann LeCun, and Nathan Srebro. The role of over-paramettrization in generalization of neural networks. In ICLR, 2019.
Maximilian Nickel, Kevin Murphy, Volker Tresp, and Evgeniy Gabrilovich. A review of relational machine learning for knowledge graphs. Proceedings of the IEEE, 104(1):11-33) 2016.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Vetito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In EMNLP, pp. 1532-1543, 2014.
David Poole, David Buchman, Seyed Mehran Kazemi, Kristian Kersting, and Sriraam Natarajan. Population size extrapolation in relational probabilistic modelling. SUM. Springer, 2014.
Lawrence R Rabiner and Biin- Hwang Juang. An introduction to hidden markov models. ieee assp magazine, 3(1):4-16, 1986.
Carl Edward Rasmussen. Gaussian processes in machine learning. In Advanced lectures on machine learning, pp. 63-71. Springer, 2004.
Josep M Sopena, Enrique Romero, and Rene Alquezar. Neural networks with periodic and monotonic activation functions: comparative study in classification problems. 1999.
Charles Sutton, Andrew McCallum, and Khashayar Rohanimanesh. Dynamic conditional random fields: Factorized probabilistic models for labeling and segmenting sequence data. Journal of Machine Learning Research, 8(Mar):693-723, 2007.
Corentin Tallec and Yann Ollivier. Can recurrent neural networks warp time? In International Conference on Learning Representation (ICLR), 2018.
Rakshit Trivedi, Hanjun Dai, Yichen Wang, and Le Song. Know-evolve: Deep temporal reasoning for dynamic knowledge graphs. In ICML, pp. 3462-3471, 2017.
Utkarsh Upadhyay, Abir De, and Manuel Gomez-Rodriguce Deep reinforcement learning of marked temporal point processes. In NeurlPS, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In NeurlPS, 2017.
Kwok-wo Wong, Chi-sing Leung, and Sheng-jiang Chang. Handwritten digit recognition using multilayer feedforward neural networks with periodic and monotonic activation functions. In Pattern Recognition, volume pp. 106-109. IEEE, 2002.
Shuai Xiao, Mehrdad Farajtabar, Xiaojing Ye, Junchi Yan, Le Song, and Hongyuan Zha. Wasserstein learning deep generative point process models. In NeurlPS, 2017.
Shuai Xiao, Hongteng Xu, Junchi Yan, Mehrdad Farajtabar, Xiaokang Yang, Le Song, and Hongyuan Zha. Learning conditional generative models for temporal point processes. AAAI, 2018.
Yu Zhu, Hao Li, Yikang Liao, Beidou Wang, Ziyu Guan, Haifeng Liu, and Deng Cai. What to do next: Modeling user behaviors by time-Istm. IJCAI, pp. 3602-3608, 2017.
11

---

Under review as a conference paper at ICLR 2020
0.2-
LSTM- T0m 2422-21
033
4p-ch
Figure 6: Comparing LSTM+T and Vee on Event-MNIST.
A IMPLEMENTATION DETAIL.
For the experiments on ENenT- N. N.TIDIGITS18 and SOF, we implemented our model in PyTorch Paszke et al. (2017). We used Adam optimizer Kingma & Ba (2014) with learning rate of 0.001. For Event-MNIST and SOF, we fixed the hidden size of the LSTM to 128. For N. TIDIGITS18, due its smaller train set, fixed the hidden size to 64. We allowed each model 200 epochs. We used a batch size of 512 2 for Event-MNIST and 128 for N_ N.TIDIGITSS and SOF. For the experiments on LMst.FM and CiteULike we used the code released by Zhu al. (2017 without any modifications, except replacing T with (2)( The only other thing we changed in their code was to change the SAMPLE.TIME variable from 3 to 20. SAMPLE TIME controls the number of times we sampling to compute Recall @ 10 and MRR @ 10. We experienced high variation when sampling only 3 times so we increased the number of times we sample to 20 to make the results more robust. For both Last.FM and CiteULike, Adagrad optimizer is used with learning rate of 0.01, vocabulary size is 5000, and the maximum length the sequence is 200. For FM, the hidden size of the LSTM 128 and for CiteULike, it is 256. For all except the synthesized dataset, we shifted the event times such that the first event of each sequence starts at time O.
optimizerKingmma
For the fairess of the experiments, have an (almost) equal number of parameters. For instance, since adding Time2Vec as an input to the LSTM inereases the number of model parameters compared to just adding time as feature, we reduced the hidden size of the LSTM for this model to ensure the number of model parameters stays (almost) the same. For the experiments involving Tece2 unless stated otherwise, we tried vectors with 16, 32 and 64 sine functions (and one linear term). We reported the vector length offering the best performance in the main text. The results for other vector lengths can be found in Appendix [ For the synthetic dataset, we use Adam optimizer with learning rate of 0.001 without any regularization. The length of the Time2Vee vector is 32.
made sure the competing models for all our experiments
'Code and datasets available at: https: http:///httt.ommm/ooaaaaaaaaaa/a /https://githhei com htmm/Darroo/tiimm...m 1stm
www.boreelisaila...oooooooaaaaaa /imee/ee
LSOMAT
05-
LETN LON-TTREERE
24-
0-24
LTT 222
92-
04--1
83
63
100
355
375
120
325
200
Spach
Spech
(a) Event-MNIST
(b) Raw N,TIIGGITSIS
Figure 7: Comparing LSTM+T and VeT Event-MNIST and raw NIDIGII
12

---

Under review as a conference paper at ICLR 2020
0.65
0.80
0.60
0.75
0.50
0.65
LSTN- T
0.45
75
125
150
200
125
150
200
Esoch
Epoch
Figure 8: Comparing LSTM+T and LSTM+ on SOF.
0.34
TT 16-1
LSTM+T
0.13
LSTM-T
LSTM 1--
0.32
1STH TimesVao 2201)
LSTM-- 5245
0:30
+
15TH Time2Vao) -5401)
MoRg10
0.12
LST
0.11
026
026
0.10
0.09
20
50
Spoch
Figure 9: Comparing LSTM+T and VeT FM..
B MORE RESULTS
We experiments on other versions of the N. TIDIGITSIS dataset as well. Following Anumula et al. (0018), we converted the raw event data to event-binned features by virtue of aggregating active channels through period of time in which a pre-defined number of events occur. The outcome of binning is thus consecutive frames each with multiple but a fixed number of active channels. In our experiments, we used event-binning with 100 events per frame. For this variant of the dataset, compared LSTM+T and LSTM+Time2 Vec similar to the experiments in Section 5.1 The obtained results were on-par. Then, similar to EveST-MNIST, we only fed as input the times which events occurred (ie we removed the channels from the input). We allowed the models 1000 epochs to make sure they converge. The obtained results are presented Fig. 6 can viewed that Time2 provides an effective representation for time and LSTM+Time2 Vec outperforms LSTM+T on this dataset.
In the main text, for the experiments involving Time2 we tested Time2 vectors with 16, 32 and 64 sinusoids and reported the best one for the clarity of the diagrams. Here, we show the results for all frequencies. Figures 7 8 9 and compare LSTM+T and LSTM+Time2Vec for our datasets. Figures| II and [2] ILSTMI with TLSTMI+Time22e. on Last.FM and CiteULike. Figures [3 and 14 compare TLSTM3 with TLSTM1+TTme2V6 on Last. and CiteULikke Like. In most cases, Time2Vec with 64 sinusoids outperforms (or gives on-par results with) the cases with 32 or 16 sinusoids. An exception is TLSTM3 where 16 sinusoids works best. We believe that is because TLSTM3 has two time gates and adding, e.g., 64 temporal components (corresponding to the sinusoids) to each gate makes it overfit to the temporal signals.
C LSTM ARCHITECTURES
The original LSTM model can be neatly defined with the following equations:
i = a (W ( + U,h + b.)
(2)
f; (W jı, + U by)
(3)
C, = Tanh (W .I +U + b.)
(4)
C fi G cj -
0, = a (W + U, bo)
(6)
h, o, O Tanh (c;)
13

---

Under review as a conference paper at ICLR 2020
LSTM+T
LSTM-T
0.19
1STM=
=2-11
0.08
LSTM-TmsZVecl-321
0.10-
0.07
0.17
6.16
0.08
0.05
30
40
50
50
Epocch
Epoch
Figure 10: Comparing LSTM+T and LSTM+ Vec on CiteULike.
TLSTML
0.15
TLSTMV
0.32
TLS11 ME www.eeeec 10-33
0.14
TLSTIN 212
TLS11 ML TimeeVach-eeee
ILSTIN ma2Vac))
TI. STMI 20m (443-3
M8R-810
0.13
TUSTMM TTTTT--Tmmeeee-eee -0
0.26
0.12-
0.11
0.24
0.10
0.09
10
20
30
40
50
20
40
50
Fpach
Spoch
Figure 11: TLSTMI's performance on Last.FM with and without Vec..
Here i, fr, and o. represent the input, forget and output gates respectively, while c, is the memory cell and h2 is the hidden state. a and Tanh represent the Sigmoid and hyperbolic tangent activation functions respectively. We refer to 2 as the jth event.
Peepholes: [eers& Schmidhuber (2000) introduced a variant of the LSTM architecture where the input, forget, and output gates peek into the memory cell. In this variant, wpi + Cj-1. Wpf C Cj-1+ and w C Cj are added to the linear parts of Eq. (2) and (6) respectively, where wpi Wpf. and Wpo are learnable parameters.
LSTM+T: Let T represent the time features for the jthh event in the input and let t'j = [ T Then LSTM+T uses the exact same equations as the standard LSTM (denoted above) except that æj is replaced with x'j-
TimeLSTM:
We explain TLSTMI and TLSTM3 which have been used in our experiments, For clarity of writing, we do not include the peephole terms in the equations but they are used in the experiments. TLSTM1, a new time gate introduced as Eq. equation and Eq. equa- tion and equation updated to Eq. equation gand equation 10 respectivvl.
t, = (W , (4TT) b,)
(8)
C f, 10 @t, O
(9)
0 = a (Woz, oz, U + U U
(10)
t, controls the influence of the current input on the prediction and makes the required information from timing history get stored on the cell state. TLSTM3 uses two time gates:
1 (W n (u bez) 22 = (W 12 + (u:2T,) + b:2)
(11)
(12)
where the elements of W are constrained to be non-pont non-positive. t1 used for controlling the influence of the last consumed item and t2 stores the thus enabling modeling long range dependencies.
TLSTM3 couples the input and forget gates following Greff et al. (2017) along with the tl and t2
gates and replaces Eq. 55) to 7 with the following:
C = (1 - i, Otl,) O 0i
0tl, OC,
(13)
c, (1 - -
0τ, o, = a (Woz, + +U, UU,, b.)
(14)
(15)
h o, @ Tanh (c,)
(16)
14

---

Under review as a conference paper at ICLR 2020
TI. STMT
0.14
TESTIR TTmedeer 1n-13
11STMI
TLSTML -Tmm22vcc1-222)
0.12
TUSTII To
Nacarga0
0.30
MARazio
0.10
0.2
0.98
9.06
0.10
0.04
20
30
40
50
Epoch
Figure 12: TLSTMI's performance on CiteULike with and without Time2Vec
0.15
0.34
TLSTHG
TLSTMV3
TL51 MI+ (mms2ecc-11-11
0.24
ILSTINS ma2vac]
0.32
TI STS00-Time2ee]]--e- 12+1
TI. (TM00-Tmm220e)--000 (M11)
0.12
0.26
0.11
024
0.09
10
20
30
50
40
50
Epoch
Spoch
Figure 13: TLSTM3's performance on Last.FM with and without Time2 Vec.
0.20
TLSTMG Time2Vaci
TL31M
0.8
0.40
0.18
TLSTMB (2ve-
183015
0.14
0.12
0.25
0.10
0.68
0.06
Figure 14: TLSTM3's performance on CiteULike with and without Time2Vec.
Zhu et al. (2017) use = At in their experiments, where At is the duration between the current and the last event.
MomeLsST+Time2eeVe To replace time in TLSTM1 with Time2Vec, we modify Eq. 8 and (10)
follows:
t, W E + a (U,t2v(T) + b.)
(17)
0 a (Wa L3 + V,t2((7) - +b))
(18)
ie., is replaced with t2v(T), , u, is replaced with U,, and v, is replaced with V... Similarly, for TLSTM3 we modify Eq.
(12 and (15) as follows:
(Un)22v(T)) + b.1) t2, (W wz + a (U.222v(T) + b12)
t1; (W w
(19)
(20)
, ( we + 12 U Uh b..)
(21)
D PROOFS
Proposition Time2 Vec invariant to time rescaling.
Proof. Consider the following Vime2Vec representation M1;
WiT + spss sin (WiT + pP))
if = 0.
(2v(7)
(22)
if Sisk.
15

---

Under review as a conference paper at ICLR 2020
Replacing with (for > 0), the Time2Vec representation updates follows:
12v(a [i] wila 7) Phn sin (a - 7) + p.),
if i 0. if iiiish k.
(23)
Consider another Time2 Vec representation M2 with frequencies w =i Then M2 behaves in the same way as M1. This proves that Vee is invariant time rescaling.
16


Reviews:
[{"review_id": "SyeFixSscr", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a particular learnable vector representation of time which is applicable across problems without the use of a hand-crafted time representation. Their representation makes use of a feed-forward layer with sine activations which operates on time data. As it is a vector representation, it combines well with other deep neural network methods. They motivate their problem well, explaining why time data is important to a variety of problems and situate their solution as an orthogonal approach to many current solutions in the literature. They make reference to fourier analysis as motivation for their representation. Finally, they provide experimental results to support their claims using fabricated and real-world time series datasets, as well as ablation studies to support their design decisions.\n\nWhile I think this work has the potential to be a significant contribution, I rate this a weak reject because the theoretical motivation and analysis of the experimental results are lacking the depth of evidence I would expect for an ICLR paper. If you provide a deeper discussion of the provable claims about the power of your model via Fourier analysis and provide a table of test accuracy/recall@K with/without your representation for more than one other state of the art algorithm for these datasets, I would be convinced to strong accept.\n\nSpecific comments:\n\n* p.3 third paragraph: you repeat yourself in math notation a few times here. Repeated equations usually indicate that there is something new happening, but all of these are just restatements of your theta sin(omega tau + phi) term. I would introduce the notation for t2v(tau) upfront and use that to define a(tau, k)[j] and f_j\n* p.3 A clearer explanation of the theory here would help, as I think Fourier's theorem nicely supports your claims.\n* p.4 first paragraph you claim that this method responds well to data which exhibits seasonality, but none of your datasets deal with data that would exhibit seasonality. There are plenty of simple real-world datasets available which show multi-scale periodic phenomena (activity or location data, weather data, travel data, etc.). In fact, segmentation and recognition of wearable device activity would be a great application for this method.\n* p.4 third paragraph: Your claim of invariance to time rescaling is technically correct, but I am not convinced that a model can learn the correct omega values for an arbitrary rescaling (e.g. if the period is smaller than the time unit). You show that this works for a rescaling from 2pi/7 to 2pi/14, but it would be nice if there was experimental confirmation of this property with frequency > 1.\n* p.6 Showing accuracy/recall across training epochs is not sufficient evidence to show that this is a useful representation. There should be some kind of comparison with test set results from other state-of-the-art work on these datasets. If adding your representation to the SOTA model improved test set performance (or at least sped up training without hurting test set performance), then that would be better evidence. If LSTM+T is the SOTA, say so and restate the author's test performance compared to yours. If this is what these graphs show, consider using a different visualization to make it clearer that you're improving the final performance, not just the training process.\n* p.8 I think sine functions make optimization harder because they make the gradient function periodic with respect to the weights, creating infinitely many local extrema. Historically this may have been an issue, but deep neural networks have so many local minima it might not matter. Still, it would be good to show that trained performance doesn't depend on the initialization values more than a standard LSTM+T model.\n* You have an interesting corner case where your neural network parameters are interpretable: you can interpret the omega values from your model as frequencies and investigate their values to see which kinds of periodicity your model uses. You do something like this on p.7, but it would be neat to see a histogram like the one you have for EventMNIST for one of the real-world datasets to see if it learns the domain-relevant time knowledge you claim that it should learn.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "SyeBhyrCKH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary\n\nThe authors propose a method for encoding time features using a sine function with learned phase and frequency. They apply this method to several synthetic and real-world datasets.\n\nTemporal and positional encoding is important to many applications, including NLP, sound understanding and time series modeling, so the topic is certainly of interest. However, the method they propose offers very little that is new when compared to e.g. Vaswani (https://arxiv.org/pdf/1706.03762.pdf, section 3.5) (the authors acknowledge this work several times). In addition, the authors compare to a baseline that seems to consist of passing time as a float. This seems like a very weak baseline---there are any number of other reasonable ways to encode this.\n\nDue to the incremental nature of the improvement and the weak baseline, I don't think this paper should be accepted to ICLR.\n\n\n## Specific Comments\n\n1. In Section 2, I find the sentence \"We follow a similar intuition but instead of decomposing a 1D signal of time into its components, we transform the time itself and feed its transformation into the model that is to consume the time information\" really unclear. Could you rephrase it?\n\n2. Often, positional encodings are used to encode ordering for a model architecture that is not inherently sequential. This is the case for the positional encodings in the transformer model. Did you try these encodings with non-recurrent architectures?\n\n3. In Section 5.2, did you mean 'fixing t2v(\\tau)[n] = sin(2\\pi n \\tau / 16)'? i.e. I think it's missing a 'tau'\n\n4. In Section 5.2 \"Fixed frequencies and phase shifts\" you compare Time2Vec to a fixed set of frequencies. Since positional encoding with Fourier transforms is well known, this seems like the relevant benchmark but it receives only a brief treatment. The authors compare these methods only on Event-MNIST and only for 16 frequencies. I would like to see this comparison expanded.\n\n5. Could you clarify exactly how time is encoded for LSTM + T? Are you, in fact, just passing a float value? How is this encoded for each data set? For example, the \"times\" for Event-MNIST is always [0, 783] while the SOF data has timestamps. What is the encoding scheme for each?", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "SkgSGYus5H", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors proposed a new embedding for time - Time2Vec. Unlike previous research that is either proposing a new architecture or proposing expensive handcrafted features, this work proposes a model-agnostic learnable time embedding.\n\nI would like to recommend an accept based on the following reasons:\n* Modeling time is crucial for quite a few machine learning tasks. With the two most desired properties, learnable and model-agnostic, this time embedding will be very useful in various applications.\n* The authors are good at story-telling and this makes the paper very readable and approachable. This increases the chance of the contribution made in this paper to be applied in real-world applications.\n* This work did clear and detailed analysis on both the empirical results and the probing experiments.\n\n\n ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "H1gIhzZRtH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Summary\nThis paper proposes a simple representation of time (Time2Vec) for modelling sequential data. The idea is to apply multiple sine functions to the time with trainable period and offset and concatenate them together, which is similar to positional encoding [Vaswani et al.] except that the periods and offsets are learned. The results on several sequential modelling datasets show that Time2Vec performs better than naive representations and alternative baselines. \n\n# Originality\n- Although the proposed representation looks simple and similar to positional encoding [Vaswani et al.], the idea of parameterizing sine functions is novel and interesting.\n\n# Quality\n- The proposed idea is very simple but seems very effective in practice as shown by the empirical results. \n- The paper also provides in-depth analysis and ablation studies showing that each of the proposed component is helpful. \n- Although the results presented in this paper look very promising, it would be much stronger if the paper presented results on other sequential modelling tasks such as machine translation and language modelling that the research community cares about much more. For example, it would be great if the paper showed that replacing the fixed positional encoding with Time2Vec improves the performance on a machine translation dataset.\n- It would be good to show the effect of the number of sine units (k) in Time2Vec. \n- (Minor) Clockwork RNN [Koutn´ık et al.] introduces a nice toy periodic sequential prediction problem, where the model has to recover a mixture of sine/cosine function. It looks like a natural task to show in this paper as well (which can potentially replace the synthesized data experiment). \n\n# Clarity\n- The paper is overall well-written.\n- Figure 4 is not mentioned in the main text.\n- Figure 5 is mentioned earlier than Figure 3. It would be good to swap them. \n\n# Significance\n- This paper proposes a simple but effective idea that can be potentially widely used by the research community. The paper would be stronger if it included more results on high-impact sequential modelling tasks and datasets such as machine translation. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper investigates and evaluates learning high-dimensional embeddings of time, which is useful for a variety of applications. This paper received 4 reviews (due to a missing review, we requested several emergency reviews). R1 recommends Weak Accept, calling the method simple but saying it could be of wide interest and utility in practice. R3 recommends Reject, identifying concerns about the significance of the contribution, caused by the simplicity of the approach, the connection to existing work, and missing comparisons to baselines. In a short review, R4 recommends Accept with several positive comments. In a long, thoughtful review, R5 recommends Weak Reject, due to concerns and questions about the theoretical motivation and depth of experiments. The authors have submitted detailed responses that have addressed many of the questions of the reviewers; however, R3 feels the response does not address their concerns, and R5 is closer to accepting but still feels additional improvements in presentation and experimentation are needed.

Given the split decision, the AC also read the paper. The AC agrees with R1 and R4 that this is an interesting problem and the approach here may be useful in practice, but shares concerns with R3 and R5 about the depth of contribution with respect to existing work, and need for additional experimental validation against stronger baselines. 
Decision: Reject