CNN-LSTM: Predicting Daily Hotel Cancellations | by Michael Grogan | Towards Data Science

An LSTM (long-short term memory network) is a type of recurrent neural network that allows for the accounting of sequential dependencies in a time series.
Given that correlations exist between observations in a given time series (a phenomenon known as autocorrelation), a standard neural network would treat all observations as independent, which is erroneous and would generate misleading results.
A convolutional neural network is one that applies a process known as convolution in determining the relationships between two functions. e.g. given two functions f and g, the convolution integal expresses how the shape of one function is modified by the other. Such networks are traditionally used for image classification, and do not account for sequential dependencies in the way that a recurrent neural network is able to do.
However, the main advantage of CNNs that make them suited to forecasting time series is that of dilated convolutions - or the ability to use filters to compute dilations between each cell. That is to say, the size of the space between each cell, which in turn allows the neural network to better understand the relationships between the different observations in the time series.
For this reason, LSTM and CNN layers are often combined when forecasting a time series. This allows for the LSTM layer to account for sequential dependencies in the time series, while the CNN layer further informs this process through the use of dilated convolutions.
With that being said, standalone CNNs are increasingly being used for time series forecasting, and the combination of several Conv1D layers can actually produce quite impressive results — rivalling that of a model which uses both CNN and LSTM layers.
How is this possible? Let’s find out!
The below example was designed using a CNN template from the Intro to TensorFlow for Deep Learning course from Udacity — this particular topic is found in Lesson 8: Time Series Forecasting by Aurélien Géron.
The below analysis is based on data from Antonio, Almeida and Nunes (2019): Hotel booking demand datasets.
Imagine this scenario. A hotel is having difficulty in forecasting hotel booking cancellations on a day-to-day basis. This is leading to difficulty in forecasting revenues and also in the efficient allocation of hotel rooms.
The hotel would like to solve this problem by building a time series model that can forecast the fluctuations in daily hotel cancellations with reasonably high accuracy.
Here is a time series plot of the fluctuations in daily hotel cancellation bookings:
The neural network is structured as follows:
Here are the important model parameters that must be accounted for.
The kernel size is set to 3, meaning that each output is calculated based on the previous three time steps.
Here is a rough illustration:
Setting the correct kernel size is a matter of experimentation, as a low kernel size risks poor model performance, while a high kernel size risks overfitting.
As can be seen from the diagram, three input time steps are taken and used to generate a separate output.
In this instance, causal padding is used in order to ensure that the output sequence has the same length as the input sequence. In other words, this ensures that the network “pads” time steps from the left side of the series in order to ensure that future values on the right side of the series are not being used in generating the forecast — this will quite obviously lead to false results and we will end up overestimating the accuracy of our model.
The stride length is set to one, which means that the filter slides forward by one time step at a time when forecasting future values.
However, this could be set higher. For instance, setting the stride length to two would mean that the output sequence would be approximately half the length of the input sequence.
A long stride length would mean that the model might potentially discard valuable data in generating the forecast, but increasing the stride length can be useful when it comes to capturing longer-term trends and smoothing out noise in the series.
Here is the model configuration:
model = keras.models.Sequential([  keras.layers.Conv1D(filters=32, kernel_size=3,                      strides=1, padding="causal",                      activation="relu",                      input_shape=[None, 1]),  keras.layers.LSTM(32, return_sequences=True),  keras.layers.Dense(1),  keras.layers.Lambda(lambda x: x * 200)])lr_schedule = keras.callbacks.LearningRateScheduler(    lambda epoch: 1e-8 * 10**(epoch / 20))optimizer = keras.optimizers.SGD(lr=1e-8, momentum=0.9)model.compile(loss=keras.losses.Huber(),              optimizer=optimizer,              metrics=["mae"])
Firstly, let’s make forecasts using the above model on different window sizes.
It is important that the window size is large enough to account for the volatility across time steps.
Suppose we start with a window size of 5.
The training loss is as follows:
plt.semilogx(history.history["lr"], history.history["loss"])plt.axis([1e-8, 1e-4, 0, 30])
Here is a visual of the forecasts versus actual daily cancellation values:
rnn_forecast = model_forecast(model, series[:,  np.newaxis], window_size)rnn_forecast = rnn_forecast[split_time - window_size:-1, -1, 0]plt.figure(figsize=(10, 6))plot_series(time_valid, x_valid)plot_series(time_valid, rnn_forecast)
The mean absolute error is calculated:
>>> keras.metrics.mean_absolute_error(x_valid, rnn_forecast).numpy()9.113908
With a mean of 19.89 across the validation set, the model accuracy is reasonable. However, we do see from the diagram above that the model falls short in terms of forecasting more extreme values.
What if the window size was increased to 30?
The mean absolute error decreases slightly:
>>> keras.metrics.mean_absolute_error(x_valid, rnn_forecast).numpy()7.377962
As mentioned, the stride length can be set higher if we wish to smooth out the forecast — with the caveat that such a forecast (the output sequence) will have less data points than that of the input sequence.
Unlike an LSTM, a CNN is not recurrent, which means that it does not retain memory of previous time series patterns. Instead, it can only train based on the data that is inputted by the model at a particular time step.
However, by stacking several Conv1D layers together, it is in fact possible for a convolutional neural network to effectively learn long-term dependencies in the time series.
This can be done using a WaveNet architecture. Essentially, this means that the model defines every layer as a 1D convolutional layer with a stride length of 1 and a kernel size of 2. The second convolutional layer uses a dilation rate of 2, which means that every second input timestep in the series is skipped. The third layer uses a dilation rate of 4, the fourth layer uses a dilation rate of 8, and so on.
The reason for this is that it allows the lower layers to learn short-term patterns in the time series, while the higher layers learn longer-term patterns.
The WaveNet model is defined as follows:
model = keras.models.Sequential()model.add(keras.layers.InputLayer(input_shape=[None, 1]))for dilation_rate in (1, 2, 4, 8, 16, 32):    model.add(      keras.layers.Conv1D(filters=32,                          kernel_size=2,                          strides=1,                          dilation_rate=dilation_rate,                          padding="causal",                          activation="relu")    )model.add(keras.layers.Conv1D(filters=1, kernel_size=1))optimizer = keras.optimizers.Adam(lr=3e-4)model.compile(loss=keras.losses.Huber(),              optimizer=optimizer,              metrics=["mae"])model_checkpoint = keras.callbacks.ModelCheckpoint(    "my_checkpoint.h6", save_best_only=True)early_stopping = keras.callbacks.EarlyStopping(patience=50)history = model.fit(train_set, epochs=500,                    validation_data=valid_set,                    callbacks=[early_stopping, model_checkpoint])
A window size of 64 is used in training the model. In this instance, we are using a larger window size than was used with the CNN-LSTM model, in order to ensure that the CNN model picks up longer-term dependencies.
Note that early stopping is used when training the neural network. The purpose of this is to ensure that the neural network halts training at the point where further training would result in overfitting. Determining this manually is quite an arbitrary process, so early stopping can greatly assist with this.
Let’s now generate forecasts using the standalone CNN model that we just built.
cnn_forecast = model_forecast(model, series[..., np.newaxis], window_size)cnn_forecast = cnn_forecast[split_time - window_size:-1, -1, 0]
Here is a plot of the forecasted vs. actual data.
The mean absolute error came in slightly higher at 7.49.
Note that for both models, the Huber loss was used as the loss function. This type of loss tends to be more robust to outliers, in that it is quadratic for smaller errors and linear for larger ones.
This type of loss is suitable for this scenario, as we can see that some outliers are present in the data. Using MSE (mean squared error) would overly inflate the forecast error yielded by the model, whereas MAE on its own would likely underestimate the size of the error by not taking such outliers into account. The use of a Huber loss function allows for a happy medium.
>>> keras.metrics.mean_absolute_error(x_valid, cnn_forecast).numpy()7.490844
Even with a slightly higher MAE, the CNN model has performed quite well in forecasting daily hotel cancellations, without having to be combined with an LSTM layer in order to learn long-term dependencies.
Of course, any model comes with limitations and the CNN is no exception.
Given that the model is making one-step predictions, this means that all observations up to time t in the relevant window is required to predict the value at time t+1. In this regard, a one-step CNN cannot be used to make long-range forecasts. While a multi-step CNN can be used for this purpose, it is debatable as to whether such a model will outperform an ARIMA model, for instance.
Additionally, a CNN (or any neural network model) requires a lot of data to train effectively. While many time series models come with pre-built parameters to model a range of time series, the CNN is learning entirely from scratch. In this regard, a researcher may spend a lot of time configuring a CNN model and end up with either lower accuracy relative to other standard time series models, or just a marginal gain in accuracy that does not justify the extra training time and data resources required.
In this particular example, the model was trained using 80% of the data as training data and then prediction accuracy was screened against 20% of the validation data. However, a true test of whether a model is effective involves comparing the prediction accuracy against test data, or data that is completely unseen by the model. This has not been done here, as this article has covered the model training component only.
However, neural networks are quite prone to overfitting — whereby the model performs well on data it has seen before — while performing poorly in predicting unseen data.
In this example, we have seen:
The similarities and differences between CNNs and LSTMs in forecasting time series
How dilated convolutions assist CNNs in forecasting time series
Modification of kernel size, padding and strides in forecasting a time series with CNN
Use of a WaveNet architecture to conduct a time series forecast using stand-alone CNN layers
In particular, we saw how a CNN can produce similarly strong results compared to a CNN-LSTM model through the use of dilation.
Many thanks for your time, and any questions, suggestions or feedback are greatly appreciated.
As mentioned, this topic is also covered in the Intro to TensorFlow for Deep Learning course from Udacity course — I highly recommend the chapter on Time Series Forecasting for further detail on this topic. Usual disclaimers apply — this is only a personal recommendation and I have no affiliation with the authors of this course in any way.
You can also find the full Jupyter Notebook that I used for running this example on hotel cancellations here.
The original Jupyter Notebook (Copyright 2018, The TensorFlow Authors) can also be found here.
Disclaimer: This article is written on an “as is” basis and without warranty. It was written with the intention of providing an overview of data science concepts, and should not be interpreted as professional advice in any way. The findings and interpretations are those of the author and are not endorsed by or affiliated with any third-parties mentioned in this article.