A deep‐learning based solar irradiance forecast using missing data

IET Renewable Power GenerationVolume 16, Issue 7 p. 1462-1473 ORIGINAL RESEARCHOpen Access A deep-learning based solar irradiance forecast using missing data Shuo Shan, Shuo Shan orcid.org/0000-0002-5240-5208 Key Laboratory of Measurement and Control of Complex Systems of Engineering, Ministry of Education, Nanjing, P.R.China School of Automation, Southeast University, Nanjing, P.R.ChinaSearch for more papers by this authorXiangying Xie, Xiangying Xie Beihang University, Beijing, P.R.China State Grid Electronic Commerce Co., Ltd., Beijing, P.R.ChinaSearch for more papers by this authorTao Fan, Tao Fan State Grid Electronic Commerce Co., Ltd., Beijing, P.R.ChinaSearch for more papers by this authorYushun Xiao, Yushun Xiao The College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, Nanjing, P.R.ChinaSearch for more papers by this authorZhetong Ding, Zhetong Ding orcid.org/0000-0002-4769-0550 Key Laboratory of Measurement and Control of Complex Systems of Engineering, Ministry of Education, Nanjing, P.R.China School of Automation, Southeast University, Nanjing, P.R.ChinaSearch for more papers by this authorKanjian Zhang, Kanjian Zhang Key Laboratory of Measurement and Control of Complex Systems of Engineering, Ministry of Education, Nanjing, P.R.China School of Automation, Southeast University, Nanjing, P.R.ChinaSearch for more papers by this authorHaikun Wei, Corresponding Author Haikun Wei hkwei@seu.edu.com orcid.org/0000-0002-6667-3166 Key Laboratory of Measurement and Control of Complex Systems of Engineering, Ministry of Education, Nanjing, P.R.China School of Automation, Southeast University, Nanjing, P.R.China Correspondence Haikun Wei, Key Laboratory of Measurement and Control of Complex Systems of Engineering, Ministry of Education, Nanjing 210096, P.R.China. Email: hkwei@seu.edu.comSearch for more papers by this author Shuo Shan, Shuo Shan orcid.org/0000-0002-5240-5208 Key Laboratory of Measurement and Control of Complex Systems of Engineering, Ministry of Education, Nanjing, P.R.China School of Automation, Southeast University, Nanjing, P.R.ChinaSearch for more papers by this authorXiangying Xie, Xiangying Xie Beihang University, Beijing, P.R.China State Grid Electronic Commerce Co., Ltd., Beijing, P.R.ChinaSearch for more papers by this authorTao Fan, Tao Fan State Grid Electronic Commerce Co., Ltd., Beijing, P.R.ChinaSearch for more papers by this authorYushun Xiao, Yushun Xiao The College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, Nanjing, P.R.ChinaSearch for more papers by this authorZhetong Ding, Zhetong Ding orcid.org/0000-0002-4769-0550 Key Laboratory of Measurement and Control of Complex Systems of Engineering, Ministry of Education, Nanjing, P.R.China School of Automation, Southeast University, Nanjing, P.R.ChinaSearch for more papers by this authorKanjian Zhang, Kanjian Zhang Key Laboratory of Measurement and Control of Complex Systems of Engineering, Ministry of Education, Nanjing, P.R.China School of Automation, Southeast University, Nanjing, P.R.ChinaSearch for more papers by this authorHaikun Wei, Corresponding Author Haikun Wei hkwei@seu.edu.com orcid.org/0000-0002-6667-3166 Key Laboratory of Measurement and Control of Complex Systems of Engineering, Ministry of Education, Nanjing, P.R.China School of Automation, Southeast University, Nanjing, P.R.China Correspondence Haikun Wei, Key Laboratory of Measurement and Control of Complex Systems of Engineering, Ministry of Education, Nanjing 210096, P.R.China. Email: hkwei@seu.edu.comSearch for more papers by this author First published: 16 February 2022 https://doi.org/10.1049/rpg2.12408AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinked InRedditWechat Abstract Irradiance prediction is a vital task in the renewable energy field. Its aim is to forecast the irradiance or power of a photovoltaic plant and thus provide a reference for stabilizing the power grid. In the real scenarios, missing data can significantly reduce the accuracy of the prediction. Meanwhile, due to the unawareness of the distribution of datasets, it is difficult to choose a suitable imputation method before modeling. Also, different imputation methods do not have the same good effects on different datasets. In this article, a recurrent neural network with an adaptive neural imputation module is proposed for forecasting direct solar irradiance using missing data. The model predicts future 4-h irradiance based on the missing historical climate and irradiance data without imputing the data in pre-processing stage. The proposed model is tested on the open access datasets, with missing values generated randomly in all input series. The model performance is compared under various missing rates and different input factors with other imputation methods. The results demonstrate that the proposed methods outperform other methods under different evaluation metrics. Furthermore, the authors integrate the model with the attention mechanism and find it has better performance at high irradiance. 1 INTRODUCTION Solar photovoltaic (PV) energy, as an alternative to fossil, fuels can effectively reduce humanity's carbon footprint in nature. PV power plants convert sunlight into electricity through PV modules and inverters. The global installed PV capacity is expected to double in 2050 [1]. As PV systems develop rapidly, it is a challenge for us to utilize it efficiently for its instability. High-accuracy irradiance prediction can significantly improve energy usage and reduce power generation costs. So, there have been many advanced achievements in solar irradiance prediction tasks [2, 3]. Global horizontal irradiance (GHI) and direct normal irradiance (DNI) are both main targets of prediction, which are associated closely with the power of PV plants. Besides, GHI can be inferred by DNI and solar angles [4]. Irradiance prediction methods are classified into three categories: physical mechanism-based models, traditional statistical models, and machine learning. Numerical weather prediction (NWP) is one of the physical models [5]. NWP models atmospheric activity by complex differential equations. The results obtained by such methods are practical in large areas, but are not suitable for prediction at a specific site. Data-driven methods, such as statistical models and machine learning, extract features based on numerous historical time series of the power plants and meteorological satellite [6, 7]. Autoregressive Integrated Moving Average Model (ARIMA) [8, 9] is one of the most widely used statistical models, which initials the parametric model by determining the order of the data and learns the parameters by optimization methods. These kinds of approaches effectively utilize historical information and are easy to train compared to traditional physical modeling. However, due to the assumptions for the priori model, for example, the linear assumption, bias are introduced, which make learned models not consistent with real scenarios. Machine learning methods, especially deep learning, have been widely used in renewable energy field in recent years. Recurrent neural network (RNN) has succeeded in dealing with sequence prediction problems. Theoretically, neural networks can approximate arbitrary nonlinear functions [10]. So, compared to the previous two methods, model bias is alleviated, and little assumptions on initial model are required. However, long historical dependencies can lead to numerical problems such as vanishing gradients [11]. In order to overcome this obstacle, gate recurrent unit (GRU) and long-short term memory (LSTM) are widely used in RNN [12]. Compared with the traditional RNN, they design forgetting gates, update gates, and output gates to help the network capture extended history information. These methods can extract more valuable features of the sequence and mitigate the impact of the vanishing gradient problem. All of above methods require a large amount of actual data. There are usually two ways to obtain irradiance data and meteorological factors at PV stations. First, the satellite can provide the estimated surface irradiation data. These kinds of data are more complete but too widely distributed in space. So there is a particular difference with the actual irradiance at a specific site. In other conditions, the ground station radiometer can provide the observed data. However, there are severe deficiencies due to problems such as communication failure and instrumentation failure. Similarly, there also exists missing in meteorological data. According to statistics, there are about 40% of solar data missing in real scenarios [13]. In order to make the algorithms more practical, there are many studies in imputation models, which are applied in data pre-processing stage to make the data complete. 36 imputation methods with their impact at different temporal frequencies in mid-term horizontal solar irradiance forecasts are discussed in [14], which are mainly divided into five main categories, interpolation, Kalman filters, persistence, weighted moving average (MA), and random sample. In recent years, unsupervised imputation methods based on generative adversarial networks (GAN) have been applied for PV data [13]. This kind of methods use generative models to learn the potential distribution of the data, and can directly output the imputed sequence in the inference process. However, the above methods have several drawbacks that are difficult to overcome. On the one hand, PV data are strongly unstable and closely related to climate. So, traditional statistical-based methods usually only deal with sporadic missing values and often ignore real data distribution [15]. Further, traditional imputation methods usually fill the missing values based on simple transformations of the nearest neighbor or historical sequence. Due to the limitation of representation of nonlinear multidimensional information, they will perform worse when the target fluctuates. On another hand, short-term forecasting of irradiance requires fast response. Unsupervised-based methods require an imputation step before forecasts, which is time-consuming. Due to the unstable of training process of generative model [16], many parameters need specific optimization, which consumes much time and human labor. Also, too precise imputation for training data can produce overfitting and limit the value of the application in the field [17]. For the first problem above, multidimensional variables and nonlinear assumptions are usually used to incorporate more information and capture potential fluctuations. For the second problem, the method should simplify the network and the process. In order to solve above problems, we propose a bi-directional Gate Recurrent Unit (GRU) based on an adaptive neural imputation module for DNI prediction, called Imputed-GRU (IGRU). It is a one-stage network consisting of two parts, the pre-imputation network and GRU. The pre-imputation network completes the missing by using positional prior and delay mechanisms. Then, the completed series is directly encoded to generate high-level features instead of the original data. Finally, the target series is obtained by decoding the generated features. In training process, the weights of GRU and imputation module are finetuned together by prediction loss. So the imputation values are determined by actual tasks. This is also the main differences with the methods, which impute missing values in raw data before modeling. Furthermore, in order to cope with the high frequency variation of irradiance data, we added an attention module to IGRU, called Attention imputed GRU (AIGRU), which makes the model focus on higher energy information rather than higher frequency information. The major contributions of this paper are as follows: We propose a predicting method to forecast solar irradiance directly from the missing data, which saves steps for imputation of raw data and performs well under various missing rates. Specifically, compared with imputing missing values in raw data, pre-imputation module has the following advantages. First, it changes the process from two-stage to one-stage, the complexity of model is reduced while the accuracy is competitive with other methods. Second, the network parameters are learned based on the prediction error of data. So, the error of the results obtained by the imputation method in pre-processing is avoided to be passed into the learners, which will reduces the bias and overfitting phenomena. Finally, compared to other methods, there are not many hyperparameters need to be adjusted. For example, the nearest neighbors of KNN, the order of Kalman filtering etc., have a lot to do with the performance [18]. Thirdly, the amount of data needed for model-based imputation is usually larger than that of prediction. So, in some industrial scenarios, it is difficult to have enough accumulated original data for providing high precision imputation. So imputing during predicting do not need as many data as the model-based imputation methods to achieve usable results. We propose an adaptive neural imputation module, which generates imputation data from prediction error and can be ported to other neural networks flexibly. With this module, there is no need to calculate the imputation values before prediction. We propose an attention-based predicting method on missing data, which performs better at high irradiance. It attempts to solve the problem of imbalance between high radiation samples and low radiation samples in irradiance prediction without resampling. Also, it can make up for the disadvantage of other models in ensemble model. The case study shows that IGRU outperforms than the traditional GRU. Also, it has a better performance over various popular imputation methods in most scenarios. The rest of this paper is organized as follows. In Section 2, the irradiance prediction/regression methods based on missing data, and the time series imputation methods are reviewed. In Section 3, the traditional imputation methods applied in this paper and the main structure of the neural imputation module, IGRU and AIGRU are introduced. In Section 4, comparison experiments with the baseline model and other popular imputation methods on an open access dataset are introduced. In Section 5, the conclusion and future work are introduced. 2 RELATED WORKS 2.1 Solar irrandiance forecast/regression with missing value The study of radiation prediction based on missing data has received close attention in recent years. Imputation of irradiance has many commonalities with the common time series imputation tasks, so many of the classical methods can be used directly. However, traditional methods usually ignored the treatment of missing values and used some simple moment and neighborhood information. [19] uses two ad-hoc methods to impute the missing values. [20] uses k-nearest neighbors(KNN) to deal with time series data with missing. However, these methods do not take into account multidimensional information, for example, the relationship between irradiance and climate data, so a class of model-based methods can characterize multidimensional inputs. [21] compares multiple imputation by chained equations (MICE) and regression based imputation method, like multiple linear regressions models (MLR) and inverse distance weighting (IDW). [22] uses matrix completion to recover missing data before forecasting short-term solar radiation. [23] explore several imputation methods in day-ahead photovoltaic generation forecasting and conclude KNN is the most appropriate methods. Recently, a number of articles have tried to review some common imputation methods and compare their performance. [24, 25] compares the effect of combining multiple interpolation methods with multiple prediction methods. [24] divided the methods into six major categories, namely interpolation, Kalman filtering, persistence, weighted moving average and random sample. They test them under different forecasting horizons find that for hourly data, Kalman Filtering (KF) performs best. Recently, deep learning based imputation and prediction is achieving great concern. Unsupervised deep learning models, such as generative adversarial network (GAN), are first to be used as imputation methods in solar irradiance forecast [13]. These methods are two-stage like the traditional methods, interpolating first and then using the interpolated data for prediction. 2.2 Time series imputation Missing temporal data is a classic problem to be solved in many fields, such as medicine [26], traffic [27] etc. Interpolation, such as mean imputation [28], is still widely used due to its efficiency and simplicity. The effects caused by the missingness come mainly from the break in the frequency domain. Also, robustness to time domain noise of models is required [29]. That's why deep learning-based models have received attention. Recently, unsupervised deep learning methods based on variational autoencoders (VAE) [30] and GAN [31] become popular. They both use generative models to learn the hidden distribution of sequences. The difference is that GAN does not specifically optimize the hidden representation. Another class of RNN-based methods, such as BRITS [32], is a class of self-learning approaches in which the hidden layer information is attenuated based on the missing data positional information in the learning process. 3 METHODOLOGY This chapter is divided into three parts. The first part introduces the problem setting. The second part introduces the traditional imputation method. The third part introduces the structure of IGRU and AIGRU. 3.1 Problem setting The study is about the issue of predicting irradiance Y for a multivariate time series X = ( x 1 , … x n , y n ) $X = (x_{1}, \ldots x_{n}, y_{n})$ . The aim is to build a multi-horizon forecasting model with direct approaches, which directly predicts all outputs instead of step-by-step [33]. Here, x1 represent a single time series and n represent the dimension of feature. We focus on the task that using the first k step time lengths to forecast the last m steps, where the input contains the historical information of the output. We assume that missing values are caused by communication or sensor failure. Then, we gives some definition of important variables. M is used to represent the positional information of missing values, with zero for missing and one for complete. Also, according to M. δ t $\delta _{t}$ represents the time step of the current missing value from the nearest valid observation. 3.2 Traditional imputation methods The methods commonly used to impute missing values in PV prediction are given here, and they are also used as comparison methods in the experimental part. 3.2.1 Interpolation Fixed values such as mean or variance are interpolated directly to the missing positions, and the mean is usually taken to be the immediately adjacent k values. This type of interpolation is based on the assumption that if the data are missing completely at random (MCAR), the missing values are likely to be very close to the mean or median of the distribution since they represent the most frequent or average observations. The advantage of this type of method is that it is easy to implement. However, the original variable distribution is distorted. 3.2.2 KNN The nearest neighbor filling idea considers the distance between two samples. It selects the mean or distance weighted of the closest several observations as the filling values for the missing sample. The advantage of KNN is that it is insensitive to outliers and is more suitable for high-dimensional data compared to the mean imputation. However, it has high computational complexity, high spatial complexity, and cannot handle sample imbalance. Equation (1) gives the calculation of KNN. x ∼ m = ∑ i w i c i ∑ i w i , \begin{equation} \tilde{x}_{m}=\frac{\sum _{i} w_{i} c_{i}}{\sum _{i} w_{i}}, \end{equation} (1)where w i = 1 d i . \begin{equation} w_{i}=\frac{1}{d_{i}}. \end{equation} (2)Here, x ∼ m $\tilde{x}_{m}$ represents the missing value, i represents the i th $i^{\text{th}}$ of selected k nearest values, and d represents the distance between two samples, such as Euclidean distance. 3.2.3 Matrix factorization (MF) MF can be used to decompose a matrix X with missing values into two or three matrices. Then these decomposed matrices can be multiplied to obtain an approximation X ̂ $\hat{X}$ of the original matrix. We use the values of this approximation matrix X ̂ $\hat{X}$ to fill the missing part of the original matrix X. The decomposition usually uses a numerical class method of gradient descent to optimize the loss function J in Equation (3). J = ∥ X − X ̂ ∥ 2 = X − U V ⊤ 2 = ∑ i , j , x i ≠ n a n x i j − ∑ l = 1 k u i l v j l 2 . \begin{equation} \begin{aligned} J &=\Vert X-\hat{X}\Vert ^{2} \\[-6pt] &={\left\Vert X-U V^{\top }\right\Vert} ^{2} \\[-6pt] &=\sum _{i, j, x_{i} \ne n a n}{\left(x_{i j}-\sum _{l=1}^{k} u_{i l} v_{j l}\right)}^{2}. \end{aligned} \end{equation} (3)To make it possible to find a more suitable estimation, regularization can be used like Equation (4). J = ∥ X − X ̂ ∥ 2 + β 2 ∥ U ∥ 2 + ∥ V ∥ 2 = ∑ i , j , x i ≠ ≠ a n x i j − ∑ l = 1 k u i l v j l 2 + β 2 ∑ i , l u i l 2 + ∑ j , l v j l 2 . \begin{equation} \begin{aligned} J &=\Vert X-\hat{X}\Vert ^{2}+\frac{\beta }{2}{\left(\Vert U\Vert ^{2}+\Vert V\Vert ^{2}\right)} \\ &=\sum _{i, j, x_{i} \ne \ne a n}{\left(x_{i j}-\sum _{l=1}^{k} u_{i l} v_{j l}\right)}^{2}+\frac{\beta }{2}{\left(\sum _{i, l} u_{i l}^{2}+\sum _{j, l} v_{j l}^{2}\right)}. \end{aligned} \end{equation} (4) 3.2.4 MICE Suppose we have ( x 1 , x 2 , … x n ) $(x_{1}, x_{2},\ldots x_{n})$ variables. If x1 is missing, then it will be regressed by the other variables, x2 to x k $x_{k}$ . Then, x1 are replaced by the new values obtained by regression. Similarly, if x2 has missing values, then x1, x3 to x n $x_{n}$ variables will be used as inputs in the regression model. Later on, the missing values will be replaced with the results of regression. 3.2.5 Kalman filtering Kalman filter is a class of methods based on the state space model [24], which consists of two equations: y t = H t α t + ω t α t + 1 = F t α t + ε t , \begin{equation} \begin{aligned} &y_{t}=H_{t} \alpha _{t}+\omega _{t} \\ &\alpha _{t+1}=F_{t} \alpha _{t}+\varepsilon _{t}, \end{aligned} \end{equation} (5)where α represents the hidden state, y is the observation, H is measurement matrix, F is transition matrix and ω , ε $\omega , \varepsilon$ represent noise. When interpolating, it is usually divided into two steps. The first step is based on the observed data, the states and covariances in Equation (5) are learned. At the second step, we estimate or smooth the future target based on the historical information. Here, we use smoothing with Kalman filtering for comparison. 3.3 IGRU and AIGRU Two proposed method are introduced in this section. First, neural imputation module is introduced, which is the key to a predictive model that enables imputation. Then, GRU with neural imputation module and the motivation of using bidirectional structure are introduced. Finally, attention based GRU with neural imputation module is introduced. 3.3.1 Neural imputation module In this paper, we propose an adaptive imputation module to help neural network models cope with missing values. Inspired by traditional missing value handling methods [31], we assume that the a priori information about the positional information of the missing values is known empirically or by methods such as anomaly detection, or the vacancy value is zero. For example, the radiometer data appear in segments of zeros under general apocalypse conditions between sunrise and sunset times. When unprocessed data are directly put into the neural network with much zero in both the input and supervised information, the data will be distorted and the wrong information is learnt by the model. According to the law of large numbers, the missing values will usually be close to the most recent observations. Referring to [31], the decay coefficient is defined as γ in Equation (6) based on the missing step δ. γ t = exp − max 0 , W γ δ t + b γ , \begin{equation} \gamma _{t}=\exp {\left\lbrace -\max {\left(0, \mathbf {W}_{\gamma } \delta _{t}+\mathbf {b}_{\gamma }\right)}\right\rbrace} , \end{equation} (6)where W γ $\mathbf {W}_{\gamma }$ , b γ $\mathbf {b}_{\gamma }$ is the linear coefficient and bias of the missing step δ. First, the linear transformation of the step size is performed. Then, the transformed result is restricted by the activation function to ensure that γ is less than one and greater than zero. Here the right-hand side of the equation is not necessarily the natural exponential function but can be a class of monotone decreasing functions. Monotonically decreasing means that the larger the δ, the longer the missing value is from the last observation, the smaller the γ and the greater the decay. Based on the decay coefficients γ we obtain, the missing values are interpolated using the maximum value in the dynamic time window in Equation (7): x ̂ t = m t ⊙ x t + 1 − m t ⊙ γ t x ¯ t , \begin{equation} \mathbf {\hat{x}}_{t}=\mathbf {m}_{t} \odot \mathbf {x}_{t}+{\left(1-\mathbf {m}_{t}\right)} \odot {\left(\gamma _{t}{\bar{x}_{t}}\right)}, \end{equation} (7)where the estimated x ̂ t $\mathbf {\hat{x}}_{t}$ after imputation consists of two parts, the original value and the new estimation replacing the missing part. The new estimation is determined by the time step of the missing value from the previous observation and the maximum value within the time window. Here is a brief explanation of why the nearest observation is not used. The nearest observation is not informative enough because it is difficult to guarantee that the value at that missing position is smaller than the nearest value. So using the maximum observation within the time window allows efficient utility of historical information, such as the maximum value in the previous days. Figure 1 is a schematic representation of the imputation process. FIGURE 1Open in figure viewerPowerPoint The flow of imputation The new estimation obtained are used as input to the model. During the training process, the weights of networks are learnt by backpropagation, including the linear parameters in the decay coefficients and the coefficients of the missing steps. This will lead to a suitable estimation for missing values. Figure 2 shows the path of the loss propagation and the overall algorithm flow. FIGURE 2Open in figure viewerPowerPoint The path of loss 3.3.2 IGRU This section introduces GRU in detail. GRU belongs to recurrent neural networks, a neural network structure commonly used for sequence modeling. The core design of RNNs that can handle sequence problems is the memory state neurons. The function of such neurons is to learn historical information, and they can be written as a class of generalizations in Equation (8). h t = f h t − 1 , y t , x t , \begin{equation} h_{t}=f{\left(h_{t-1},y_{t},x_{t} \right)}, \end{equation} (8) where h represents the value of the hidden layer neuron, and f represents the memory update equation (learned update function) that needs to be learned. There are two gates in f in the GRU model: update gate and reset gate, respectively. Fewer parameters and faster training compared to LSTM ensure the memory and mitigate the gradient disappearance. r t = σ W r · h t − 1 , x t z t = σ W z · h t − 1 , x t h ∼ t = tanh W h ∼ · r t ∗ h t − 1 , x t h t = 1 − z t ∗ h t − 1 + z t ∗ h ∼ t y t = σ W o · h t , \begin{equation} \begin{aligned} &r_{t}=\sigma {\left(W_{r} \cdot {\left[h_{t-1}, x_{t}\right]}\right)} \\ &z_{t}=\sigma {\left(W_{z} \cdot {\left[h_{t-1}, x_{t}\right]}\right)} \\ &\tilde{h}_{t}=\tanh {\left(W_{\tilde{h}} \cdot {\left[r_{t} * h_{t-1}, x_{t}\right]}\right)} \\ &h_{t}={\left(1-z_{t}\right)} * h_{t-1}+z_{t} * \tilde{h}_{t} \\ &y_{t}=\sigma {\left(W_{o} \cdot h_{t}\right)}, \end{aligned} \end{equation} (9)where r , z $r,z$ represent the reset gate and update gate. The reset gate function determines the new intermediate hidden layer based on the previous moment hidden layer information and observation. Update gate determines the ratio of the weighted average with the previous moment hidden layer information, that is, the update percentage. Figure 3 summarizes the input-output relationship of the GRU model, and the internal structure of the GRU Cell. FIGURE 3Open in figure viewerPowerPoint The architecture of GRU. (a) The input and output of GRU; (b) the inside structure of GRU cell It is worth noting that the bidirection