A combination of novel hybrid deep learning model and quantile regression for short‐term deterministic and probabilistic PV maximum power forecasting

IET Renewable Power GenerationVolume 17, Issue 4 p. 794-813 ORIGINAL RESEARCHOpen Access A combination of novel hybrid deep learning model and quantile regression for short-term deterministic and probabilistic PV maximum power forecasting Nguyen Duc Tuyen, Corresponding Author Nguyen Duc Tuyen [email protected] School of Electrical and Electronic Engineering, Hanoi University of Science and Technology, Hanoi, Vietnam Department of Electrical Engineering, Shibaura Institute of Technology, Tokyo, Japan Correspondence Nguyen Duc Tuyen, School of Electrical and Electronic Engineering, Hanoi University of Science and Technology, Hanoi 11615, Vietnam. Email: [email protected]Search for more papers by this authorNguyen Trong Thanh, Nguyen Trong Thanh School of Electrical and Electronic Engineering, Hanoi University of Science and Technology, Hanoi, VietnamSearch for more papers by this authorVu Xuan Son Huu, Vu Xuan Son Huu School of Electrical and Electronic Engineering, Hanoi University of Science and Technology, Hanoi, VietnamSearch for more papers by this authorGoro Fujita, Goro Fujita Department of Electrical Engineering, Shibaura Institute of Technology, Tokyo, Japan Contribution: Conceptualization, ​Investigation, Project administration, Supervision, Writing - review & editingSearch for more papers by this author Nguyen Duc Tuyen, Corresponding Author Nguyen Duc Tuyen [email protected] School of Electrical and Electronic Engineering, Hanoi University of Science and Technology, Hanoi, Vietnam Department of Electrical Engineering, Shibaura Institute of Technology, Tokyo, Japan Correspondence Nguyen Duc Tuyen, School of Electrical and Electronic Engineering, Hanoi University of Science and Technology, Hanoi 11615, Vietnam. Email: [email protected]Search for more papers by this authorNguyen Trong Thanh, Nguyen Trong Thanh School of Electrical and Electronic Engineering, Hanoi University of Science and Technology, Hanoi, VietnamSearch for more papers by this authorVu Xuan Son Huu, Vu Xuan Son Huu School of Electrical and Electronic Engineering, Hanoi University of Science and Technology, Hanoi, VietnamSearch for more papers by this authorGoro Fujita, Goro Fujita Department of Electrical Engineering, Shibaura Institute of Technology, Tokyo, Japan Contribution: Conceptualization, ​Investigation, Project administration, Supervision, Writing - review & editingSearch for more papers by this author First published: 06 November 2022 https://doi.org/10.1049/rpg2.12634AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract Recently, the increasing penetration of renewable energy resources in power system, especially photovoltaic (PV) systems, has caused severe technical issues due to its randomness and dependency on primary sources. Therefore, precise output power forecast is vital for both system operators and PV system owners to improve grid stability and generation quality, respectively. This article proposes a novel hybrid deep learning model named EDSACL comprising four components, namely, convolutional neural network (CNN), long short-term memory (LSTM), self-attention mechanism (SAm), and residual learning (RL) strategy for short-term PV maximum power prediction with forecast horizon of 1, 2, and 4 h. This proposed model initially extracts spatial-temporal features of input data based on CNN and LSTM before using SAm to achieve hidden features of LSTM layers. RL strategy is employed to maintain the information flow entering the network. Besides, interval prediction is also implemented based on a combination of the EDSACL model and quantile regression with different probabilities. The forecast accuracy of the proposed model is validated based on two real-world data sets. The predicted results show the superiority of the proposed model over numerous predictive models; in particular, mean absolute percentage error values of the proposed hybrid model are under 6% in all case studies. 1 INTRODUCTION In recent years, rapid economic development and population growth have posed several severe issues such as the increasing electricity demand and climate changes. In addition, the primary resources based on fossil fuels are being over-exploited and the reductions in carbon emission are strictly required. Therefore, the presence of alternative energies is of great importance [1]. Among renewable energy sources (RESs), photovoltaic (PV) power has received massive attention with high penetration in electric market. Nevertheless, the integration of PV systems into power grids leads to many huge technical challenges to grid operators in maintaining the stability of the power systems due to its intermittent characteristic [2]. In particular, the PV output power converted from solar energy through photovoltaic effect varies rapidly based on various meteorological factors such as irradiance and temperature. Therefore, these stochastic fluctuations result in the uncertainties of generated PV power. This brings out numerous negative effects on the grid relating to frequency and voltage. Appropriate scheduling of power supply and load is a significant key to restrict the aforementioned consequences and ensure the power balancing of the power grid. Thus, accurate PV power forecast plays a vital role to enhance the stability and reduce the power curtailment rate of the PV systems. PV power forecast (PVPF) might be classified based on forecast horizon [3] into four types: long-term, medium-term, short-term, and very short-term [4]. Very short-term forecasts define the prediction a few minutes in advance and are utilized to ensure the stability of the power systems against rapid fluctuations as well as ramp-rate. Reference [5] developed a predictive model for very short-term high-dimensional PV power forecast combining with feature selection method called ReliefF to eliminate unimportant inputs. Short-term predictions which refer to forecasting a few hours ahead are used to facilitate serious problems such as unit commitment and economic dispatch [6]. Several studies adopt various methods for short-term PV forecast [7-9]. Medium-term forecasts defining predicting a few weeks in advance are employed for maintenance, whereas long-term predictions denote monthly or annual forecast dedicated for planning as well as power plant construction schedule [10]. Studies [11] and [12] proposed PV forecast methods for medium-term and long-term, respectively, and the predicted results of the proposed models outperformed other benchmark models. PV prediction is split into two types, namely, deterministic forecast (point forecast) and probabilistic forecast (interval forecast). The appearance of interval forecast is to quantify the uncertainties of prediction, since utilizing point forecast alone cannot lead to optimal results. Regarding deterministic predictive models, studies related to PV prediction are normally divided into four categories including physical models, persistence models, statistical models or machine learning (ML) models, and hybrid models [7, 13]. Physical models adopt mathematical calculations to illustrate the physical relation of meteorological variables [13] and contain three approaches, namely numerical weather prediction (NWP) [14], sky images [15], and satellite images [16]. These approaches are unstable due to the huge dependency on weather prediction [5]. Persistence model is frequently utilized as a benchmark model to evaluate the performance of the proposed models [17]. In PV forecast domain, myriad works used this model as a referenced model [18, 19]. Thus, the forecast accuracy of this model relies considerably on the reliability of historical data. Similar to persistence approaches, statistical models predict forthcoming values based on the historical data, and these models are adequate to forecast stochastic time-series data [10]. Numerous well-known statistical models are adopted for PV forecast such as auto-regressive integrated moving average (ARIMA) [20] and exponential smoothing (ETS) [21]. However, the adaptability of these forecast models is the major drawback for rendering them prevalent. For instance, statistical models cannot be applied to heterogeneous data, which is a mixture of numerical and categorical features. Due to the rapid development of artificial intelligence (AI), ML-based PV forecast models are applied widely and yield promising results compared to these aforementioned models. ML methods are capable of extracting non-linear relationship between target variable and input variables thanks to the computational process of the models [22]. In case of solar power forecast, several ML models are listed as follows: linear regression (LR) [23], support vector machine (SVM) [24], and artificial neural network (ANN) [25, 26]. Furthermore, deep learning (DL) models are enhanced based on ANN models with various alternative structures to stimulate the performance as well as be appropriate to different issues. In PV forecast field, apart from ANN-based models, many DL models are used such as convolutional neural network (CNN) [27], recurrent neural network (RNN) [28], and the improved structures of RNN model, including long short-term memory (LSTM) [29] and gated recurrent unit (GRU) [30]. Hybrid models are implemented to achieve higher forecast precision by combining the above approaches together or integrating them with optimization algorithms. Several studies surveyed the application and superiority of hybrid models over single ones in numerous domains such as hydrological streamflow [31] and building energy consumption forecasting [32]. Reference [33] provided a comprehensive overview on combination model techniques for solar irradiance forecast. Yinpeng et al. [34] proposed DL-based temporal distributed hybrid model containing GRU and ARIMA for PV generation prediction. The forecast results indicated that the proposed model outperforms other benchmark models. Study [13] introduced a novel hybrid model combining LSTM with wavelet packet decomposition (WPD) for predicting PV power 1 h in advance. The predicted outcomes showed that the proposed model yields more accurate performance in comparison with numerous DL models such as LSTM, GRU, RNN, and MLP. In addition, hybrid models are also utilized in other fields such as traffic congestion and wind speed prediction. For example, [35] proposed a novel combination model comprising LSTM network, no negative constraint theory (NNCT), and population extremal optimization (PEO) algorithm for short-term traffic forecasting. To be more specific, NNCT is used to aggregate the predicted outcomes of several LSTM networks while PEO algorithm is employed to optimize integration coefficients of NNCT. Study [36] presented a hybrid model for wind speed forecasting including two layers where the first layer contains three components, namely, extreme learning machine (ELM), Elman neural network (ENN), and LSTM network, and the second one is based on ELM-based nonlinear integrated mechanism in order to combine separate forecast results. To summarize, the advantages and disadvantages of the aforementioned methods are listed in Table 1. TABLE 1. Advantages and disadvantages of four types of deterministic predictive model [3] Model Advantage Disadvantage Physical model Do not require historical data; Low accuracy forecasting Short computational time Persistence model Short computational time; Low accuracy forecasting with large forecast horizon; Frequently used as a benchmark model Unable to work with non-linear data Statistical model or ML model Able to work with non-linear data Rely on the quality of historical data; High computational time Hybrid model High accuracy forecasting High computational time In terms of probabilistic forecast, there are two common methods, namely, parametric and non-parametric. The operation of parametric methods is to estimate the parameters during training process with the assumption that the target variable follows a known probability distribution. Various distributions are applied for interval renewable energy prediction, such as Cauchy distribution [37], Gama distribution [38], and Gaussian distribution [39]. Unlike parametric methods, non-parametric estimation methods require no assumptions and are frequently combined with deterministic predictive models. However, the huge disadvantage of this combination is that the interval prediction is relied predominantly on the accuracy of the point forecast outcomes. Several prominent methods can be mentioned such as quantile regression [40], bootstrapping [41], and kernel density estimation [42]. Recently, there are increasing number of studies that utilize the hybrid model combined by CNN and LSTM to extract spatial-temporal features. For example, study [43] combined multi-head attention, 1D-CNN, and LSTM to predict electric demand in different various time resolutions, whereas [18] proposed the LSTM-CNN model for solar irradiation forecasting. Nevertheless, these above studies solely focused on short time-step prediction. With respect to further time-step forecast, there are more challenges due to the uncertainties of data [44]. Therefore, the depth of the network as well as the number of parameters need to be increased to reinforce the learning ability of predictive model. Study [45] introduced bidirectional-LSTM incorporated with several residual learning strategies to solve the convergence issues of deep neural networks, but spatial features between input and target variable as well as influencing levels of different inputs on forecasting results were not considered. To overcome the weaknesses of these previous studies, this article proposes a high-accuracy hybrid deep learning model (EDSACL model) with encoder-decoder structure comprising four components, namely, CNN, LSTM, Self-attention mechanism (SAm), and residual learning strategy for short-term PV maximum power deterministic prediction with different forecast horizon. To be more specific, CNN and LSTM are used to extract spatial-temporal features, whereas SAm is adopted to recognize important information from hidden features of the Encoder. This work also employs residual learning strategy to maintain data flow of the proposed network, thus avoiding degradation and over-fitting. Besides, to quantify the uncertainties of forecast results, probabilistic or interval forecast is also conducted utilizing quantile regression. The forecast is implemented based on two data sets collected from the National Renewable Energy Laboratory (NREL) with a resolution of 5 minutes and evaluated using performance criteria. Moreover, various models are utilized as benchmark models to verify the predictive accuracy of the proposed model. The key contributions of this study are highlighted as follows: Adopt several methods for data pre-processing such as box-plot method for outlier removal, linear interpolation (LI) and day similarity (DS) for filling missing data. Implement a high-precision hybrid deep learning model with encoder-decoder structure, including CNN, LSTM, and SAm for short-term PV maximum power forecast with different forecast horizons using two PV parameter data sets from the NREL. The predicted outcomes show that the proposed model can obtain a mean absolute percentage error (MAPE) value of 1.66% in case of forecasting 1-h ahead corresponding 12 steps. Combine the aforementioned hybrid model with quantile regression to conduct interval forecast with probability of 80%, 90%, and 95%. Compare the predicted outcomes of the proposed model with numerous state-of-the-art models such as LSTM, CNN-LSTM, SAM-LSTM, transformer in various instances regarding sky conditions. The remainder of this paper are organized as the following: Section 2 details the EDSACL model regarding each component; Section 3 illustrates the proposed framework for PV maximum power forecast whereas Section 4 represents the result comparison and discussions; Section 5 is dedicated to conclusion and future works. 2 METHODOLOGY To achieve high predictive efficiency, combination models or hybrid models are frequently adopted instead of single ones. In this section, we detail our proposed model for PV maximum power forecasting, which comprises three components, namely LSTM (Section 2.1), CNN (Section 2.2), and multi-head attention mechanism (Section 2.3). To the best of the authors' knowledge, this proposed model has not been published before. Furthermore, in addition to conducting deterministic forecast, interval prediction or probabilistic forecast method named quantile regression (Section 2.6) is also presented in this section. LSTM LSTM network was initially introduced in 1997 by Hochreiter and Schmidhuber [46] and then applied in several domains. In addition to using memory cells instead of normal neurons, LSTM proposed several parameters such as input gate i t ${i_t}$ , forget gate f t ${f_t}$ , and output gate o t ${o_t}$ to facilitate capturing the flow of information in the network. This also helps LSTM overcome disadvantages of RNN, especially vanishing and exploding gradient. The operation of LSTM network is described mathematically as follows [47]: f t = σ w f . x t , h t − 1 + b f , $$\begin{equation} {f_t} = \sigma {\left({{{\rm {w}}_f}.{\left[ {{x_t},{h_{t - 1}}} \right]} + {b_f}} \right)} , \end{equation}$$ (1) C ∼ t = tanh w c . x t , h t − 1 + b c , $$\begin{equation} {\tilde{C}_t} = \tanh {\left({{{\rm {w}}_c}.{\left[ {{x_t},{h_{t - 1}}} \right]} + {b_c}} \right)} , \end{equation}$$ (2) i t = σ w i . x t , h t − 1 + b i , $$\begin{equation} {i_t} = \sigma {\left({{{\rm {w}}_i}.{\left[ {{x_t},{h_{t - 1}}} \right]} + {b_i}} \right)} , \end{equation}$$ (3) C t = f t ∗ C t − 1 + i t ∗ C ∼ t , $$\begin{equation} {C_t} = {f_t} * {C_{t - 1}} + {i_t} * {\tilde{C}_t} , \end{equation}$$ (4) o t = σ w o . x t , h t − 1 + b o a b , $$\begin{equation} {o_t} = \sigma {\left({{\rm {w}}_o}.{\left[ {{x_t},{h_{t - 1}}} \right]} + {b_o} \frac{a}{b}\right)}, \end{equation}$$ (5) h t = o t . tanh C t , $$\begin{equation} {h_t} = {o_t}.\tanh {\left({{C_t}} \right)} , \end{equation}$$ (6)where wf, wc, wi, wo indicate the weight matrices of the forget gate, memory cell, input gate, and output gate, respectively. bf, bc, bi and bo denote the biases of the forget gate, memory cell, input gate, and output gate, respectively. σ defines the Sigmoid activation function with output ranging from 0 to 1, in which 0 represents that no information is retained and 1 indicates all information is preserved. * denotes point-wise multiplication. The architecture of LSTM network is illustrated in Figure 1. FIGURE 1Open in figure viewerPowerPoint General architecture of LSTM model [46] One-dimensional convolution neural network (1D-CNN) Conventional approaches employ CNN for feature recognition of two-dimensional images, while 1D-CNN is widely adopted for local spatial feature extraction of time series data [48]. 1D-CNN might reduce the translational variance among hidden features and maintain spatial relationships between PV parameters by learning filters to extract and recognize spatial features [43]. The proposed model utilizes the main operation of 1D-CNN named convolutional layer. In particular, this layer is targeted to extract local spatial features from the target variable (i.e. P m $P_m$ ) and its associated input variables (i.e. PV module parameters and datetime) by using multiplication called “convolution.” This process is illustrated as Equation (7): c o n v i k k = r w k ⊙ x i j + b k , $$\begin{equation} conv_{ik}^k = r{\left({{{{\left({{{w}^k} \odot x} \right)}}_{{ij}}} + {b_k}} \right)} ,\end{equation}$$ (7)where w k ${{w}^k}$ and ⊙ denote the weight of the kernel and convolutional process operator, respectively, whereas r ( · ) $r(\cdot )$ defines the ReLU activation function. Convolutional kernels are designed to be suitable for each forecast horizon. With respect to further forecast horizon, great convolutional kernel is used to achieve correlation information between target and input variables. Moreover, utilizing multiple small stacked kernels instead of a huge kernel is to increase the depth of the network and decrease the number of parameters [49]. Besides, Batch normalization is also used to facilitate the training process of neural networks to be faster and more stable through normalizing of the layers' inputs [50]. Here, the operation of convolutions is packed into a module whose structure is as shown in Figure 2. Each module includes two convolutional layers and the output of each layer is fed into activation function called ReLU then normalized by Batchnorm process. Table 2 presents the selection of CNN modules' parameters. As can be observed from this table, the kernel size is chosen equalling to 9, which is the number of input variables. The number of CNN module layer is selected according to the forecast horizon. TABLE 2. Parameters of CNN modules Forecast horizon 1-h ahead 2-h ahead 4-h ahead Kernel size 9 9 9 The number of CNN module layer 1 2 3 Activation function ReLU ReLU ReLU Normalization: Batchnorm1d n-feature = 9 n-feature = 9 n-feature = 9 The number of residual connection 1 2 3 FIGURE 2Open in figure viewerPowerPoint Structure of module CNN detailing the convolutional multiplication operation Multi-head attention mechanism SAm was introduced as an effective method to improve the accuracy of encoder-decoder models, and it has indeed been successful in various domains [52]. With respect to time-series forecasting models that adopt the LSTM network, there is an internal correlation among hidden features; meanwhile, these features at different time steps have different effects on the forecast results [53]. This cannot be recognized by both the LSTM and CNN network. Thanks to assigning weights intentionally to hidden features, SAm is employed to enhance important information and reduce unimportant information. Three vectors, namely, query (q), key (k), and value (v) are defined by multiplying time-series inputs by three matrices W Q ${{W}_Q}$ , W K ${{W}_K}$ , W V ${{W}_V}$ , which are adjusted during training process. The output matrix through the attention layer is calculated as follows: A t t e n t i o n ( Q , K , V ) = s Q K T d k V , $$\begin{equation} Attention(Q,K,V) = s{\left({\frac{{Q{K^T}}}{{\sqrt {{d_k}} }}} \right)}V , \end{equation}$$ (8)where Q = [ q 1 , q 2 , … ] T $Q = {[ {{q_1},{q_2},\ldots} ]^T}$ , K = [ k 1 , k 2 , … ] T $K = {[ {{k_1},{k_2},\ldots} ]^T}$ , V = [ v 1 , v 2 , … ] T $V = {[ {{v_1},{v_2},\ldots} ]^T}$ are query, key, and value matrices, respectively, and they are merged into a matrix. To be more specific, q i ${q_i}$ , k i ${k_i}$ , v i ${v_i}$ are query, key, value vectors at ith time step ( 0 < i < L $0<i<L$ ), respectively. s ( · ) $s(\cdot )$ denotes softmax function while d k ${{d_k}}$ is the scaling factor. In fact, d model $\rm {d_{model}}$ -dimensional k, v, and q vectors with high d model $\rm {d_{model}}$ are difficult to extract hidden features. As a result, h-heads self-attention are used in parallel and then concatenated together. This method is referred to as multi-head attention mechanism [51]. The structure of multi-head attention is shown in Figure 3. This mechanism enables concentrating on information from different representational sub-spaces at time-steps by employing q, k, and v vectors in each head to extract the sub-space features of the time-series inputs. This mechanism is presented as follows: M u l t i h e a d ( Q , K , V ) = C o n c a t ( h e a d 1 , h e a d 2 , … ) × W O , $$\begin{equation} Multihead(Q,K,V) = Concat(hea{d_1},hea{d_2},\ldots)\times {{W}^O} , \end{equation}$$ (9) h e a d i = A t t e n t i o n ( Q × W i Q , K × W i K , V × W i V ) , $$\begin{equation} head_i = Attention(Q\times {W}_i^Q,K\times {W}_i^K,V\times {W}_i^V) , \end{equation}$$ (10)where W i Q ∈ R d m d × d q ${W}_i^Q \in {R^{{d_{md}} \times {d_q}}}$ , W i K ∈ R d m d × d k ${W}_i^K \in {R^{{d_{md}} \times {d_k}}}$ , W i V ∈ R d m d × d v ${W}_i^V \in {R^{{d_{md}} \times {d_v}}}$ are parameter matrices of Q, K, and V, respectively; i is the sequence number of the head; d m d $d_{md}$ , d q $d_{q}$ , d k $d_{k}$ , and d v $d_{v}$ are dimensions of time-series inputs, vector q, vector k, and vector v, respectively; W O ∈ R d v × d m d ${{W}^O} \in {R^{{d_v} \times {d_{md}}}}$ is a weight matrix. FIGURE 3Open in figure viewerPowerPoint Structure of multi-head self-attention [51] In this study, the proposed model utilizes three attention heads ( h = 3 $h=3$ ) and thus d k ${d_{k}}$ = d v ${d_{v}}$ = d m d / h ${d_{md}/h}$ = 3. Residual learning Although deep learning models might be equipped with more features by stacked layers, several disadvantages such as vanishing or exploding gradient could occur in case of the small number of training data that has a negative effect on the convergence of the model. Furthermore, degradation and over-fitting can happen after solving convergent issues. To deal with difficulties in acquiring data, numerous methods are employed. Among these approaches, residual learning is widely applied by overwhelming others such as drop-out and pooling strategy [54]. In case of utilizing residual learning, the mapping layers can be represented as in the following equation: G x = f G x + x , $$\begin{equation} G{\left(x \right)} = {f_G}{\left(x \right)} + x , \end{equation}$$ (11)where G ( x ) $G(x )$ denotes the desired mapping of the stacked layer; f G ${f_G}$ is the original stacked layer, and x is the input. In terms of conventional models, only the calculations between x and G n ${G_n}$ are performed where G n ${G_n}$ is the output of the n th $\rm {n^{th}}$ layer. Therefore, the total network aims to reduce the difference between G n ${G_n}$ and x as well as G n − x ${G_n}-x$ . Alternatively, the models employing residual learning optimize the parameters by minimizing the difference between G i ${G_i}$ and G i − 1 ${G_{i-1}}$ corresponding to the output of the i th $\rm {i^{th}}$ and ( i − 1 ) th $\rm {{(i-1)}^{th}}$ layer, respectively. By assigning optimization tasks for each layer, the learning process becomes more straightforward. Furthermore, the output of residual learning networks can be expressed as f n ( G n − 1 ) + ⋯ + f 1 ( x ) + x $ {f_n}({G_{n - 1}}) + \cdots + {f_1}(x) + x$ , whereas the output of conventional networks is f n ( G n − 1 ( … f 1 ( x ) ) ) ${f_n}({G_{n - 1}(\ldots{f_1}(x))})$ . This represents the continuity of the information flow entering the network without being stuck in the details of the training process [45]. Proposed model In this subsection, a novel deep learning model named EDSACL with the incorporation of 1D-CNN, LSTM network, self-attention mechanism, and residual learning strategy is proposed for PV maximum power forecast by modelling spatial-temporal features and overcoming inherent drawbacks of memory model in terms of calculation capability. Figure 4 illustrates the proposed model called EDSACL with encoder-decoder structure. The EDSACL model includes two main components, namely, encoder and decoder where encoder transforms input data into feature vectors containing necessary information such as spatial-temporal features and hidden correlation while decoder uses feature vectors from encoder and produces the results that best match the actual or desired output. These components are detailed more explicitly as follows: ENCODER: The encoder is responsible for receiving input and extracting feature vectors. To be more specific, the data are initially fed into CNN layer. This layer employs convolutional multiplication to the input data consisting of target value and neighbouring locations to attain spatial information of target and extract spatial features, then put through activation function called ReLU then normalized by Batchnorm process, and the resulting vectors carry the spatial features: j τ = [ j 1 , j 2 , … , j n ] ${j_\tau } = [ {{j_1},{j_2},\ldots,{j_n}} ]$ . In the proposed model, the convolutional layer is deployed as a module as illustrated in Section 2.2. These modules are linked through residual connection to maintain the flow of information throughout the training process. The selection of CNN modules' parameters is presented in Table 2 that is mentioned above. The raw data from residual connection and spatial information extracted by the CNN layer are fed into the LSTM layer. By utilizing memory cells to update hidden states, this layer extracts temporal aspects, captures long-term relationships, and maintains long-term dependencies of historical data. To sustain the network's information flow, the output of