How to use Deep Learning for Time-Series Data | by Zachary Warnes | Towards Data Science

Time series data is often placed in a class of its own regarding machine learning and data science. However, due to the changing distributions over time and the sequential testing, this area of data science can be tricky to manage.
This article will go over three different deep learning models for creating a predictive model based on time-series data. The first will be an LSTM and followed by two CNNs with one-dimensional convolutions and two-dimensional convolutions.
I’ve included code chunks so that you can copy and re-use this code by simply updating the data being used and adjusting parameters.
Feel free to bookmark this page for future reference.
Note that these models can be much more than what you need for a time-series prediction. However, starting with simple models and 2D tabular data, you can benchmark what performance you might expect.
To convert your time series data into 2D data, aggregate over the whole or a subset of the period available. Other options are not discussed here, the most common of which are autoregressive (AR) models.
As any well-versed data scientist knows, data wrangling, manipulation, and all-around data engineering is by far the most time-consuming task for a data scientist. With machine learning and deep learning, in particular, this task is more involved. For example, in contrast to models that run off tabular data (data in a basic pandas dataframe or 2D NumPy array), time-series data requires additional dimensions for models to function correctly.
A straightforward approach to manipulating data into three dimensions or four dimensions is to consider single instances one at a time. In this way, you can look at your data one instance at a time to confirm the correct information (such as standard time intervals) before further transformations.
X = []for i in instances:    # ‘df’ is a 2D pandas dataframe or numpy array    df = generate_instance(i) # Produce a single instance here    X.append(df)X_3D = np.array(X)X_4D = X_3D.reshape((X_3D.shape[0], X_3D.shape[1], X_3D.shape[2], 1))
For each instance, create a 2D dataframe or numpy array. The columns of this dataframe will contain the features, and the rows represent the different time steps for the single instance.
Create each instance one at a time and append them to a list. Once each dataframe is in a list, convert to a numpy array and then easily feed the information into one of the following models.
Note that each of the instances must have the same dimensions. If they differ, you will get some exciting errors.
For 2-Dimensional convolutions, the data must have an additional dimension. For multivariate time series data, this means adding another dimension with the function reshape.
The models use either ‘relu’ or sigmoid activation functions and a sigmoid activation for the final output.
For these models, you can adjust the number of convolutional layers, the kernel size, the number of filters, the learning rate, the number of nodes in the dense layer, the number of LSTM units, the batch size, and the pooling size.
The models here are built to handle a classification problem using multivariate time-series data and use binary cross-entropy as their loss function. Also, these models include proportional class weighting according to their rate of occurrence. Finally, the whole network is optimized with Adam optimization.
The structure of both CNN models follows a similar format:
The data goes through a series of convolutional layers, either one dimensional or two dimensional.The pooling layer. After pooling, the output is flattened and connected to a fully connected layer.After the fully connected layer, the predicted probability is produced.
The data goes through a series of convolutional layers, either one dimensional or two dimensional.
The pooling layer. After pooling, the output is flattened and connected to a fully connected layer.
After the fully connected layer, the predicted probability is produced.
The data goes through an LSTM cell with a variable number of units.The output is flattened and connected to a fully connected layer.After the fully connected layer, the predicted probability is produced.
The data goes through an LSTM cell with a variable number of units.
The output is flattened and connected to a fully connected layer.
After the fully connected layer, the predicted probability is produced.
The structure is kept purposely similar. So it is easy to interchange only the first layers that differ between the CNN and LSTM models.
from tensorflow.keras.models import Sequentialfrom tensorflow.keras.layers import LSTM, Dense, Flatten, Conv1D, Conv2D, MaxPooling1D, MaxPooling2Dfrom tensorflow.keras.optimizers import Adam, schedules, SGDKERNEL_SIZE = 2POOL_SIZE = 2N_FILTERS = 24BATCH_SIZE = 128LEARNING_RATE = 0.00001DENSE_UNITS = 50LSTM_UNITS = 50EPOCHS = 100N_STEPS = X.shape[1]N_FEATURES = X.shape[2]class_weight = {0: 1., 1: 1/np.mean(y_train)}ACTIVATION_FUNC = ‘relu’FINAL_ACTIVATION = ‘sigmoid’LOSS_FUNC = ‘binary_crossentropy’METRIC = ‘accuracy’OPTIMIZER = Adam(learning_rate=LEARNING_RATE)# Split you own data from X_3D or X_4D, and a targetX_train, y_train, X_val, y_val = NoneX_test, y_test = None
There are a couple of things here to make a note of here. I consider the first dimension of my data to be the number of instances, the second the number of time steps, and the third dimension the number of features. With four-dimensional data for the 2D convolutional model, the fourth dimension is a ‘dummy’ dimension to allow for proper processing. Still, the remaining dimensions maintain the same interpretation.
The ordering is important to know depending on the backend you use for these Keras models. Unfortunately, the conventions are not consistent, which may be the source of your errors. If this is the case, consider reshaping your data to change the order of the dimensions.
I have also left the train, calibration, and test data purposely set to ‘None’ so you can adjust your data split how you would like. During fitting, the calibration set is used to evaluate the model.
For more details about how to separate data for different types of problems and training models, you can read this post:
towardsdatascience.com
Long Short-Term Memory Models (LSTM) are a variant of recurrent neural networks (RNN). These models are composed of multiple channels that data can pass through to maintain information from previous steps to be used in future steps.
The LSTM cells that are used in the predictive models shown here follow the same structure. At first glance, they look complicated but can be interpreted intuitively with a few observations.
There are two main paths through an LSTM cell; these can be roughly interpreted as short-term and long-term memory. With the short term being the bottom path through the cell and the long-term is the top path through the cell.
For short-term memory, notice how the input is directed into this path. As one would expect with short-term memory, the model interacts with the new data, Xt, and the output from the previous step. This data then has several possible paths through the cell.
The long term memory is the path along the top of the cell. This path has inputs from the previous long-term memory output. Then through the cell, the long-term memory is updated based on the updates from several different channels.
From short-term to long-term, the different paths with the LSTM cell are most clearly distinguished by the activation function used and the combination of intermediary outputs.
It is crucial to notice that a sigmoid function is used for some paths, and the hyperbolic tangent function is used for other paths. This choice is meaningful. The critical observation here is that the output from a sigmoid function is 0 to 1. In contrast, the output of a hyperbolic tangent function is -1 to 1.
Now, why is there a difference?
In the first channel between the short-term and long-term memory paths, the activation is a sigmoid function, meaning the output is between 0 and 1. Thus, this path dictates whether the input from the current step should be considered more important for long-term memory. If this output is a zero, then the final product between this and the previous long-term memory output results in 0. On the other hand, if this output is closer to 1, then the previous long-term memory persists.
The second and third channels use sigmoid and hyperbolic tangent activations, respectively. This combination acts as a two-step process, determining whether the current information is relevant and whether or not it contributed positively or negatively depending on the -1 to 1 output from the hyperbolic tangent function. This portion of the cell essentially answers these two questions. Then the contribution of the current time step to the long-term memory is added to the long-term memory.
The Final path combines the current time step with the current understanding contained within the long-term memory path. The output from this pathway becomes the output for the following LSTM cell.
model = Sequential()model.add(    LSTM(LSTM_UNITS, activation=ACTIVATION_FUNC,          input_shape=(N_STEPS, N_FEATURES) ))model.add(Flatten())model.add(Dense(DENSE_UNITS, activation=ACTIVATION_FUNC))model.add(Dense(1, activation=FINAL_ACTIVATION))model.compile(    optimizer=OPTIMIZER,    loss=LOSS_FUNC,    metrics=[tf.keras.metrics.AUC(), METRIC])
Convolutional networks are some of the most complicated models in deep learning; they are often made up of many different layers, and the layers can behave differently.
In contrast to a standard neural network with fully connected layers, a convolutional neural network uses convolutional layers, pooling layers, and fully connected layers. Some advanced models even use increasing complex variations of these layers.
Convolutional neural networks CNN are generally associated with image and video machine learning applications due to their ability to effectively capture the relationships between pixels and build open these relationships to model increasingly complex structures.
At its core, these networks rely on their convolutional layers to effectively capture the relationships between close pixels. However, this architecture is not limited to multidimensional images. For example, it is simple to coerce time-series data into a format that aligns well with CNNs.
You can also use convolutions to model the interactions between different steps in a time series. In contrast to images, one-dimensional convolutions can model the behavior from previous and future time steps to capture complex relationships from the series.
Some one-dimensional convolutions are pretty common, and you have likely used them without knowing. For example, a moving average is a convolution type where each step is weighted equally. So, roughly speaking, you can view one-dimensional convolutions as moving averages of different lengths. Each time step in the period is weighted differently.
model = Sequential()# Add one or more convolutional layersmodel.add(Conv1D(filters=N_FILTERS, kernel_size=10, activation=ACTIVATION_FUNC))model.add(MaxPooling1D(pool_size=2))model.add(Flatten())model.add(Dense(DENSE_UNITS, activation=ACTIVATION_FUNC))model.add(Dense(1, activation=FINAL_ACTIVATION))model.compile(    optimizer=OPTIMIZER,     loss=LOSS_FUNC,    metrics=[tf.keras.metrics.AUC(), METRIC])
When you have multiple time-series, one-dimensional convolutions are still possible; however, combining different series using two-dimensional convolutions captures more complex relationships.
These higher dimensional convolutions capture relationships between different times series.
model = Sequential()# Add one or more convolutional layersmodel.add(Conv2D(    filters=N_FILTERS,    padding=”same”,    kernel_size=(2,2),    activation=ACTIVATION_FUNC,    input_shape=(N_STEPS, N_FEATURES, 1)),)model.add(MaxPooling2D(pool_size=2))model.add(Flatten())model.add(Dense(DENSE_UNITS, activation=ACTIVATION_FUNC))model.add(Dense(1, activation=FINAL_ACTIVATION))model.compile(    optimizer=OPTIMIZER,    loss=LOSS_FUNC,    metrics=[tf.keras.metrics.AUC(), METRIC])
To train each of the models, you can use the following same chunk of code. Because the models are assumed to be classifiers, the fitting includes class weights. In most cases, classes are not perfectly balanced, so this addition takes this into account.
model.fit(    X_train, y_train, epochs=EPOCHS,    verbose=1, class_weight=class_weight,    batch_size=BATCH_SIZE,    validation_data=(X_val, y_val),)
Time-series problems have many moving parts. However, many other much simpler models can be used to a reasonable degree of performance.
I have also glossed over some of the more time-involved sections when creating classifiers (or regressors) from multivariate time series data. The two most significant being:
What happens when you have inconsistent time steps and how you can standardize your data and handle gaps.
How to optimize the deep learning model architecture, not all models are created equal. While you may not need a network with 100 layers, you can still benefit from some tuning.
However, building models using multivariate time-series data can be an enjoyable task. It forces you to not treat all your data as tabular data that you can use with a scikit-learn model. These models are flexible and can capture complex time-dependent relationships. You should add them to your existing arsenal of models.
If you’re interested in reading articles about novel data science tools and understanding machine learning algorithms, consider following me on Medium.
If you’re interested in my writing and want to support me directly, please subscribe through the following link. This link ensures that I will receive a portion of your membership fees.