Time2Vec for Time Series features encoding
By: Marco Cerliani
Tags: Time Series Forecasting, Towards Data Science, Encoding, Data Science, Machine Learning

Time is golden information in every Machine Learning problem which engages Time Series. As Data Scientists, we must do our best to extract time patterns and make our data speak themself. During preprocessing, common procedures are for example standardization (stationarity check, autocorrelation removal…), creating and encoding categorical time features (day, week, month, season…), manual feature engineering (Fourier transformations…). Not always our efforts are repaid because, during modelization, our selected model can fail to treat time itself properly as a feature.

In this post, I try to reproduce the approach proposed in the paper ‘Time2Vec: Learning a Vector Representation of Time’, which final scope is to develop a general-purpose model-agnostic representation for time that can be potentially used in any architecture (I adapt this solution developing a Neural Network in Keras). The authors don’t want to propose a new model for time series analysis, but instead, their goal is to provide a representation of time, in form of vector embedding, in order to automatize the feature engineering process and model time in a better way.

THE DATASET

To give evidence and a concrete utility of the whole solution, we need an adequate dataset. In my case, I need data in time series format without any redundant information in form of extra features. This scenario is typical in autoregressive problems, where we have only one time series and we use it as a feature to forecast the future. In real life this is a common task, so is not difficult to find a dataset. I found a nice one on Kaggle. It stores a great number of historical levels of water in Venice. Forecasting these values is a serious task; daily tourists can access a well detailed and accurate report of the level of the sea in various parts of the city.

I love Venice and my aim is not to compete with them. We also have not enough information to provide truly evident performances (extra regressors such as temperature, moon phases, weather conditions and so on, provide an adequate boost). Here we have to squeeze the only historical data at our disposal to provide the forecasts of the next hour levels of water.

Example of hourly water level (cm) in a week

TIME2VEC IMPLEMENTATION

Mathematically speaking, implement Time2Vec is quite easy:

Where k is the time2vec dimension, tau is a raw time series, F is a periodic activation function, omega and phi are a set of learnable parameters. In my experiment, I set F to be a sin function in order to able a selected algorithm to capture periodic behaviors in data. At the same time, the linear term represents the progression of time and can be used for capturing non-periodic patterns in the input that depend on time.

The simplicity makes this vector representation for time easily consumable by different architectures. In my case, I try to transfer this concept in a Neural Network structure modifying a simple Keras dense layer.

class T2V(Layer):



def __init__(self, output_dim=None, **kwargs):

self.output_dim = output_dim

super(T2V, self).__init__(**kwargs)



def build(self, input_shape): self.W = self.add_weight(name='W',

shape=(input_shape[-1], self.output_dim),

initializer='uniform',

trainable=True) self.P = self.add_weight(name='P',

shape=(input_shape[1], self.output_dim),

initializer='uniform',

trainable=True) self.w = self.add_weight(name='w',

shape=(input_shape[1], 1),

initializer='uniform',

trainable=True) self.p = self.add_weight(name='p',

shape=(input_shape[1], 1),

initializer='uniform',

trainable=True) super(T2V, self).build(input_shape)



def call(self, x):



original = self.w * x + self.p

sin_trans = K.sin(K.dot(x, self.W) + self.P)



return K.concatenate([sin_trans, original], -1)

The output dimension of this custom layer is the hidden dimension specified by the user (1 ≤ i ≤ k), i.e. the learned sinusoids from the network, plus a linear representation of the input (i = 0). With this instrument in our hands, we just have to stack it with other layers and try its power in our case study.

THE MODEL

Is Time2Vec a good representation of time? In order to answer this question, I compare the performances achieved in our forecasting task, building two different Sequential Neural Network models. The first one has as input our custom Time2Vec layer, stacked on a simple LSTM layer. The second one is composed of only the simple LSTM layer used in the previous structure.

def T2V_NN(param, dim):



inp = Input(shape=(dim,1))

x = T2V(param['t2v_dim'], dim)(inp)

x = LSTM(param['unit'], activation=param['act'])(x)

x = Dense(1)(x)



m = Model(inp, x)

m.compile(loss='mse', optimizer=Adam(lr=param['lr']))



return m def NN(param, dim):



inp = Input(shape=(dim,1))

x = LSTM(param['unit'], activation=param['act'])(inp)

x = Dense(1)(x)



m = Model(inp, x)

m.compile(loss='mse', optimizer=Adam(lr=param['lr']))



return m

We carry out the fitting procedures operating hyperparameters optimization. This is done using keras-hypetune. This framework provides hyperparameter optimization of the neural network structures in a very intuitive way. We operate a standard grid search of some parameter combinations. This is done for all two training procedures involved.

RESULTS

I use the first 70% of the data as train set and the remaining 30% as test set. The train is also split into validation, inside the fitting process, in order to operate hyperparameter tuning. The best and optimized T2V + LSTM achieves around 1.67 MAE on the test, while the simple and optimized LSTM achieves around 2.02.

Both the networks seem to be able to identify the pattern in the data really well. The usage of T2V encoding can grant a little boost in performance.

SUMMARY

In this post, I presented an approach that automatically learns the features of the time. In particular, I reproduced Time2Vec, a vector representation for time, adapting it to a Neural Network architecture. In the end, I was able to show the effectiveness, in terms of performances, of this representation on a real task. I want to remark, as the authors of the paper suggest, that T2V is not a new model for time series analysis, but it is a simple vector representation that can be easily imported into many existing and future architectures and improve their performances.