Time Series Forecasting using TensorFlow and Deep Hybrid Learning | by Aditya Bhattacharya | Towards Data Science

In my previous TDS article I described about the Machine Learning variant of Deep Hybrid Learning and how easily it can be applied for image data. If you are someone who is new to the concept of Deep Hybrid Learning, I would highly recommend you to have a look at the session recording of my past session at GIDS AI/ML 2020 conference. The recording video can also be directly accessed from my youtube channel.
In this article, we will see how Deep Hybrid Learning can be applied for time series data and whether it is as effective as it is with image data. But before we begin, in case if you are very new to time series data, I would recommend looking at my previous posts related to time series data analysis:
Time Series Anomaly DetectionTime Series SegmentationPreparing time series data from SQLTime Series 101 with R and Python
Time Series Anomaly Detection
Time Series Segmentation
Preparing time series data from SQL
Time Series 101 with R and Python
If you are new to TensorFlow and wondering how to apply TensorFlow for time series forecasting, this article from my website can be really helpful. The article does give very detailed code walkthrough of using TensorFlow for time series prediction. In this article also, I will take a similar approach of providing a very detailed approach for using Deep Hybrid Learning for Time Series Forecasting in 5 simple steps.
For this post, I am going to use the same Sunspot Data from Kaggle like my previous article. As mentioned there, the data can be easily downloaded from my GitHub project TimeSeries-Using-TensorFlow. I will encourage everyone to use google colab notebooks where the modules required are already installed and the infrastructure is ready for use. Now, let’s get started!
The data can be downloaded using a simple command in google colab –
!wget — no-check-certificate https://raw.githubusercontent.com/adib0073/TimeSeries-Using-TensorFlow/main/Data/Sunspots.csv -O /tmp/sunspots.csv
Once the download is complete, we can use pandas to load the data into dataframes.
# Loading the data as a pandas dataframe df = pd.read_csv(‘/tmp/sunspots.csv’, index_col=0) df.head()
We can take a glance at the data, which looks like this:
Now, ideally we should do an extensive Exploratory Data Analysis (EDA) to understand the trend, seasonality of the data. But for this case, to keep things simple, we will do a visual inspection for the data.
In this step we have to transform the loaded data and process it so that it can be passed as an input to the Deep Hybrid Learning model after which we can start the training process. We can consider time series forecasting as a sequential machine learning regression problem, and the time series data is converted into a set of feature values and the corresponding true or target value. Since regression is a supervised learning problem, we need the target value, in which the lagged time series data becomes the feature values like this:
We will follow a window or buffer approach in which we would have to consider an appropriate window size. Then we would move the window from left to right of the sequence or the series data. We will consider the value immediately to right of the window frame as target or the true value. So, each time-step we will shift or move the window so as to get a new row of features values and target value pairs. In this way we form the training data and training labels. In a similar way, we form the test and the validation dataset, which is typically required for a machine learning prediction model. Also, remember that for a predictive model, having a wider observation window and a narrow prediction window can give better results. Next, for the train-test-validation split ratio we will have to figure that out based on the size of the data. For this example I have used a split ratio of 0.8 and based on the seasonality of the data, we have taken a window size of 60. But these variables are all hyper-parameters, which requires some tuning to get the best possible results.
The code for the same is as follows:
# Convert the data values to numpy for better and faster processing time_index = np.array(df['Date']) data = np.array(df['Monthly Mean Total Sunspot Number'])   # Certain Hyper-parameters to tune SPLIT_RATIO = 0.8 WINDOW_SIZE = 60 BATCH_SIZE = 32 SHUFFLE_BUFFER = 1000  # Dividing into train-test split split_index = int(SPLIT_RATIO * data.shape[0])   # Train-Test Split train_data = data[:split_index] train_time = time_index[:split_index]  test_data = data[split_index:] test_time = time_index[split_index:]
Next, we will prepare a data generator that prepares the training and testing data for us.
def ts_data_generator(data, window_size, batch_size, shuffle_buffer):  '''  Utility function for time series data generation in batches  '''  ts_data = tf.data.Dataset.from_tensor_slices(data)  ts_data = ts_data.window(window_size + 1, shift=1, drop_remainder=True)  ts_data = ts_data.flat_map(lambda window: window.batch(window_size + 1))  ts_data = ts_data.shuffle(shuffle_buffer).map(lambda window: (window[:-1], window[-1]))  ts_data = ts_data.batch(batch_size).prefetch(1)  return ts_data# Expanding data into tensorstensor_train_data = tf.expand_dims(train_data, axis=-1)tensor_test_data = tf.expand_dims(test_data, axis=-1)tensor_train_dataset = ts_data_generator(tensor_train_data, WINDOW_SIZE, BATCH_SIZE, SHUFFLE_BUFFER)tensor_test_dataset = ts_data_generator(tensor_test_data, WINDOW_SIZE, BATCH_SIZE, SHUFFLE_BUFFER)
Now, we have our processed data ready for feeding this into our model.
We will use a simple version of a Deep Hybrid Learning architecture for this problem. As discussed we will use the Deep Learning variant with Late Fusion technique. The model architecture looks like this:
Here we have used a combination model of 1D CNN for extracting the initial sequential features and then combined with 2 LSTM layers for the feature extraction part and finally passed the same into tradional DNN Fully Connected Layers to produce the final output.
The code for the model architecture looks like this:
# DHL Fusion model of 1D CNN and LSTM model = tf.keras.models.Sequential([tf.keras.layers.Conv1D(filters=32, kernel_size=5,strides=1, padding="causal",activation="relu",                       input_shape=[None, 1]),tf.keras.layers.LSTM(64, return_sequences=True),   tf.keras.layers.LSTM(64, return_sequences=True),   tf.keras.layers.Dense(30, activation="relu"),   tf.keras.layers.Dense(10, activation="relu"),   tf.keras.layers.Dense(1)])
Next we would need to choose other hyper-parameters like learning rate, optimizer and loss function. I am not covering the intuition behind selection of these values in this post so as to keep it simple. But feel free to comment or drop me a note in this if you are curious to find out more about this.
optimizer = tf.keras.optimizers.SGD(lr=1e-4, momentum=0.9)model.compile(loss=tf.keras.losses.Huber(),              optimizer=optimizer,              metrics=["mae"])history = model.fit(tensor_train_dataset, epochs=200, validation_data=tensor_test_dataset)
Next we would see how to evaluate our model. But first, after the training process, it is always a better idea to plot the model loss curves to see whether the model is actually learning.
From the model loss curves, we do see clear presence of over-fitting. At the end of the article I will give some hint of how to handle this to make the model better, but we can see the model loss is decreasing with more training time, which is a good indication that the model is learning.
Now, for the model evaluation we would need to pick a metric. In a future article, I will include about various model evaluation metrics for time series data. But for this case, we will use MAE as the metric.
def model_forecast(model, data, window_size):    ds = tf.data.Dataset.from_tensor_slices(data)    ds = ds.window(window_size, shift=1, drop_remainder=True)    ds = ds.flat_map(lambda w: w.batch(window_size))    ds = ds.batch(32).prefetch(1)    forecast = model.predict(ds)    return forecastrnn_forecast = model_forecast(model, data[..., np.newaxis], WINDOW_SIZE) rnn_forecast = rnn_forecast[split_index - WINDOW_SIZE:-1, -1, 0]# Overall Errorerror = tf.keras.metrics.mean_absolute_error(test_data, rnn_forecast).numpy()print(error)
The MAE value that we got is roughly around 40. Which is not bad, but a little high for this case. And the reason why the model error is more is because of the initial over-fitting that we saw.
As a final step, let’s visualize the results that we are getting over the test data and let’s inspect if at all the model is close for predicting good results.
From the first plot, we can see that the predicted values does follow a similar seasonal pattern and trend as the actual values, but the peaks are not as high as the actual ones. Also, since time series forecast should be ranged prediction not a single point estimate, we will use the error rate to form the confidence interval or the confidence band. We can see the the error bands are wide, which means the model is not very much confident and might have some prediction error.
The code for visualization is as follows:
plt.figure(figsize=(15, 6))  plt.plot(list(range(split_index,len(data))), test_data, label = 'Test Data') plt.plot(list(range(split_index,len(data))), rnn_forecast, label = 'Predictions') plt.legend() plt.show()  plt.figure(figsize=(15, 6)) # Plotting with Confidence Intervals plt.plot(list(range(split_index,len(data))), rnn_forecast, label = 'Predictions', color = 'k', linestyle = '--')plt.fill_between(range(split_index,len(data)), rnn_forecast - error, rnn_forecast + error, alpha = 0.5, color = 'orange') plt.legend() plt.show()
And thus we do have a Deep Hybrid Learning model for time series forecasting and we have used TensorFlow for forming the model and implementing the flow.
But if you are wondering how can the result be improved, I have the following recommendations for you:
Change the window size (either increase or decrease)Use more training data (so as to solve the over-fitting problem)Use more model layers or more hidden unitsUse a different loss function and learning rateWe saw that the loss curve is not smooth. It usually happens if the batch size is small, so try with a bigger batch size.
Change the window size (either increase or decrease)
Use more training data (so as to solve the over-fitting problem)
Use more model layers or more hidden units
Use a different loss function and learning rate
We saw that the loss curve is not smooth. It usually happens if the batch size is small, so try with a bigger batch size.
Also sometimes, a simpler model may give better result. In my post Time Series Forecasting using Deep Learning with TensorFlow I got much better results just by using a simple Deep Neural Network. Now, unlike with image data, we saw that with time series data, Deep Hybrid Learner was not significantly better than a conventional Deep Learning, Machine Learning or Statistical methods. But, yes after doing thorough hyper parameter tuning, I am sure the results would be much better!
Thus, this brings us to the end of this article. The full working code and notebook can be obtained from here. Keep following me: https://medium.com/@adib0073 and my website: https://www.aditya-bhattacharya.net/ for more! Please feel free to reach for any query or comment!
**Update**: I am working on a book on applied computer vision in python and looking for contributors who can contribute in writing or reviewing content and code. Mostly I am interested to take in Graduate students and help them become an Applied Computer Vision expert by writing a book and maintaining an open source project! Please take a look at this article of mine to know more: Link. If you want to take a look at the early efforts of the initiative, please take a look my detailed hands-on workshop on applied computer vision in youtube. If you are interested to be a part of this please feel free to reach out through any mode of communication mentioned here: https://aditya-bhattacharya.net/contact-me/ with your updated resume.