Forecasting the Future Power Consumption using LSTM, Deep Neural Network and Exploratory Data Analysis | Towards Data Science

Human beings live on the time axis and design their daily life accordingly. Mostly they sleep at night and other activities are mostly done at certain times (e.g. breakfast — in the morning). When we consider language, it is known that what is spoken or written maintains and follows certain integrity of meaning. When we talk about machine learning, model training in such sequential datasets is made by using statistical methods such as ARIMA, SARIMAX, as well as using a neural network. The fundamental idea is that the data at time t is the result of several previous data points. This article explains the theoretical part of RNN — LSTM and includes a tutorial about quick exploratory data analysis of time series dataset and predicting the future power consumptions of Germany using LSTM and DNN.
Table of Contents1. Theory1.1. Recurrent Neural Networks1.1.1. Under the Hood1.1.2. Vanishing Gradient Descent1.2. Long-Short Term Memory2. Tutorial3. Conclusion4. References
In a dataset, which flows sequential, each data at time t occurs in relation to the historical data. For example, if we consider NLP, the continuation of a paragraph is shaped according to the keywords before it, that is, according to the integrity of the meaning. While reading a novel, an average of 3–4 words per second is seen and the book continues to be read fluently. Although thousands of words have been seen and read at the end of 1 hour of reading, it is difficult to repeat them by heart. However, the section read can be understood and explained easily. In machine learning, Recurrent Neural Networks can be used to extract the basic meaning & pattern from such sequential datasets. Let’s continue with different examples; movies consist of frames and these frames are in certain integrity according to the subject & content of the movie. We can easily predict the next moment or the next frame in the movie after we have understood the movie. Recurrent Neural Networks can be used to predict the future by detecting the flow pattern in sequential data. In another example, it provides time-series, that is, in time-changing graphics (stock market, spreading of coronavirus by time), to detect the flow and perform forecasting. So, how to explain mathematically what provides these possibilities using numbers?
The figure below shows how the Recurrent Neural Network process works in a sequential dataset with 4 data in 4 seconds. Its basic working principle is similar to Feedforward Neural Network. The whole system can be briefly summarized as follows:
The weight values ​​assigned to the input turn into output after the layers and are compared with the real output to determine the Loss value, that is, the distance from where it should be. According to this loss value, the value of the weight is updated and the same process is repeated. The purpose of updating the weights is to minimize the loss value and determine the optimum weight value.
RNN, on the other hand, is designed sequentially, that is, in a backward connection to detect the connection of the data at time t with (t-1). Of course, beyond just t and (t-1), perhaps its value at time t is calculated according to the last 10 values, thanks to its backward-connected structure. Now let’s dive into Figure 1 in detail.
Let’s consider 4 data points (x1, x2, x3, and x4). Yellow structures represent layers. The equation of each output is depicted above. As it is seen, the output of each layer is sequentially presented as an additional input to the next layer. The final output results are based on the selected activation function. Then, the deviation of the obtained result is calculated, the weights are derived according to the loss output and updated with the backpropagation process. The RNN structure basically follows this pattern. If we take a look at the disadvantages of this RNN structure, the vanishing gradient problem meets us.
When it is hard to converge the global minimum point, this case is called Vanishing Gradient Descent.
In RNN, this process occurs when there are too many layers and the weights are updated with the Sigmoid Function. The value obtained with the Sigmoid Function should be between 0 and 1. However, in the backpropagation process, the effect of the first inputs (first a few words for NLP, first a few times for Time Series) on the system in the learning of the sequential series is minimized, since the weights update is at such a low level that when the first layers are reached. This causes these inputs and their weights to be ignored, concentrating sequential learning on the last inputs.
If another activation function is used instead of Sigmoid Function, such as ReLu, due to the structure of ReLu, a situation opposite to vanishing gradient descent, that is, too much change in weights so that cannot reach the global minimum point safely.
LSTM is designed to prevent ignoring the effect of first inputs in the sequential series during the updating of the weights. Figure 2 shows how the LSTM model avoids the problem of not considering prior data (weights of previous data) unimportant in weights adjustments.
Logistic (Sigmoid) activation function outputs the data between 0–1. In this way, the level of significance of the data is evaluated between 0–1. 0 becomes unimportant and is reflected as 0 in the multiplied operation. 1 means it is important.
Tanh activation function takes on the task of regulation by giving the value between (-1) and (+1) as output.
Previous cell output(h(t-1)) and current input(x(t)) are passed through the logistic(sigmoid) function. The long-term state (c(t-1)) is multiplied with the resulting value (f(t)). At this stage (forget gate), it is mathematically inferred which data is important and to what extent from the long-term states.The sigmoid value obtained in the first step is also obtained here (i(t)) with the same inputs (Previous cell output(h(t-1)) and current input(x(t)). In addition, these inputs are also passed through the tanh activation function and multiplied with the sigmoid output (g(t)). At this stage, (input gate) i(t) mathematically expresses how important the g(t) value is.Mathematical expressions (Forget Gate and Input Gate) obtained in the first two cases are added and form the new cell state (c(t)). In addition, this result is regularized by passing through tanh and multiplied by the logistic(sigmoid) function result (o(t)) of the two inputs in the first two stages (Previous cell output(h(t-1)) and current input(x(t)). Returns the short term state (h(t)). At this stage (output gate), the importance of long-term states is transferred to the system.
Previous cell output(h(t-1)) and current input(x(t)) are passed through the logistic(sigmoid) function. The long-term state (c(t-1)) is multiplied with the resulting value (f(t)). At this stage (forget gate), it is mathematically inferred which data is important and to what extent from the long-term states.
The sigmoid value obtained in the first step is also obtained here (i(t)) with the same inputs (Previous cell output(h(t-1)) and current input(x(t)). In addition, these inputs are also passed through the tanh activation function and multiplied with the sigmoid output (g(t)). At this stage, (input gate) i(t) mathematically expresses how important the g(t) value is.
Mathematical expressions (Forget Gate and Input Gate) obtained in the first two cases are added and form the new cell state (c(t)). In addition, this result is regularized by passing through tanh and multiplied by the logistic(sigmoid) function result (o(t)) of the two inputs in the first two stages (Previous cell output(h(t-1)) and current input(x(t)). Returns the short term state (h(t)). At this stage (output gate), the importance of long-term states is transferred to the system.
LSTM and configured versions of it are suitable methods for many time-based datasets, especially stock market price prediction.
In this tutorial, models are trained with LSTM and DNN using hourly power consumption of the Germany dataset between 01.01.2015 and 01.09.2020. The dataset can be downloaded from the link.
The license of the dataset: CC0: Public Domain
First, the necessary libraries are imported and since the data at time t is related to the previous data, from 01.01.2015 to 31.05.2019 is reserved as the training set and 31.05.2019–01.09.2020 as the test set. Dataset consists of power consumption values every 15 minutes. Dataset is converted to hourly based by summing the 4 values (0–15 min, 15 min-30 min, 30 min-45 min, 45 min-60 min). Then it is visualized as in Figure 3.
After adding years and months columns, box plots according to years and months are visualized as in Figure 4 and in Figure 5 respectively.
In this section, the dataset is rescaled and redesigned according to the neural network model using the following strategy:
24 data (1 day = 24 hours) sequentially with last_n = 24 is designed as input, 25. data is output. For example, on the basis of the index, 0–24 is input while 24. data output; 25. Data output when 1–25 is input; 26. data output when 2–26 is input, etc.
While the LSTM model is being designed, return sequence=True is set until the last LSTM layer and the model is trained. At the end of the training, the updated weights and the training and test dataset are predicted separately by the model.
The visualization result of a random part of the test dataset by rearranging the shifting caused by the selected number while creating the input-output is shown in Figure 6.
In this section, the model is built using the Dense layers to train the model with the Deep Neural Network. The same scaler method is used in the dataset and it is trained after making the shapes suitable for the model. The processes used in LSTM are configured and made suitable for the DNN model, and the prediction of a random part of the test dataset is visualized in Figure 7.
As the name suggests, Long Short Term Memory (LSTM) is based on training the model by updating the weights, taking into account the importance of long-term historical data. The DNN model, which is created using the dense layers as above, is the updating of the weights according to the activation function between the input and output on the basis of the neural network. When we roughly compare the results above (Figure 6 and Figure 7), it can be said that the two outcomes are close to reality. However, when looked in detail, it is seen that LSTM draws a more successful graph at the lower and upper peaks, especially at the lower peak points. While almost all lower peaks of the DNN results are lower than they should have been, LSTM draws a more successful and appropriate graph even though there are small upward and downward deviations.
Since LSTM is designed to take into account the importance of previous data, it performs the learning process more effectively. Therefore, while the model is being trained, DNN can make the error pattern a part of the learning process, while LSTM presents the model more flexibly at this point. As this dataset structure roughly follows a certain order (see Figure 3), the distinction between the two models can only be seen when investigated carefully. However, the difference of LSTM can be seen more clearly for datasets with higher randomness (hard to detect pattern) and time-varying datasets.