# Three-Phase Load Prediction-Based Hybrid Convolution Neural Network Combined Bidirectional Long Short-Term Memory in Solar Power Plant

Authors: Chien-Hsing Lee; Phuong Nguyen Thanh; Chao-Tsung Yeh; Ming-Yuan Cho
Journal: International Transactions on Electrical Energy Systems
 (2022)

---

## Abstract

The economic renewable energy generations have been rapidly developed because of the sharp reduction in the costs of solar panels. It is imperative to forecast the three-phase load power for more effective energy planning and optimization in a smart solar microgrid installed on a building in the Linyuan District, Taiwan. To alleviate this problem, this article proposes a convolution neural network bidirectional long short-term memory (CNN-Bi-LSTM) to accurately predict the short-term three-phase load power in building the energy management system in the smart solar microgrid with the collected data from advanced metering infrastructure (AMI), which have not been investigated before. The three-phase load-predicting methodology is developed using weather parameters and different collected data from AMI. The project evaluates the performance of the CNN-Bi-LSTM model by utilizing hyper-parameter optimization to attain the optimum parameters. The prediction models are trained based on hourly historical input features, selected based on the Pearson correlation coefficient. The performances’ optimal structure CNN-Bi-LSTM are validated and compared with the bidirectional LSTM (Bi-LSTM), LSTM, the Gated Recurrent Unit (GRU), and the recurrent neural network (RNN) models. The obtained optimized structure of CNN-Bi-LSTM demonstrates the effectiveness of the proposed models in the short-term prediction of three-phase load power in a smart solar microgrid for building with a maximum enhancement of 68.36% and 8.81% average MSE, and 30.26% and 36.36% average MAE during the testing and validating operations.


---

## Body

## 1. Introduction

The renewable solar microgrid has been rapidly developing due to the sharp decrement in solar costs over the past decade. The rapid spread of these smart solar microgrids has created a new operating paradigm that has resulted in predicting load power in microsystems. Moreover, the electrical consumption is continuously increasing due to the growing demand, which results in system complexity and uncertainty. Therefore, load power prediction becomes an essential application for electricity distribution and is fundamental to the corresponding decision-making. The forecasts based on time intervals can be divided into short-term, medium-term, and long-term predictions. The short-term load power forecasts can range from 1 hour to many days and are critical for dispatching generation, controlling spot markets, and defining anomalies. Interest in predicting the load power demand in smart solar microgrids has considerably increased near decades. Particularly, forecasting the load power for smart solar microgrids depends on various factors, and machine learning has been utilized to predict the load power and electricity demand which could provide good performance.The applications of artificial neural networks (ANN) for short-term load power prediction have evolved [1–4], and new proposals have been proposed over time, such as statistical methods [5–7], support vector machines [8–10], and decision trees [11, 12]. These traditional techniques could be utilized with heterogeneous and incomplete databases and straightforward interpretation for a simple dataset. The statistical methods could be applied in simple applications but achieved a poor prediction due to the multiple seasonality of electricity load [6]. The support vector machine (SVM) could obtain the global solutions as conventional ANN, but it typically suffers the overfitting problems and quickly falls into local minima solution [8–10]. The SVM has difficulty in selecting the best kernel function with its corresponding parameters [13, 14]. The decision trees method is a simple model with easy implementation and interpretation [11, 12]. However, the performance and reliability of predicting results depend on the accuracy of the training process. The disadvantages of these methodologies are the difficulty in quantifying the uncertainty results and the limited prediction for continuous variables. Advancements in ANN algorithms and big data in deep learning have recently utilized and proposed higher reliability and better performance [15–19]. The deep learning machine requires a high computational cost to optimize the network structures, which are the number of hidden layers, the number of neural nodes, the learning rate, and the optimization algorithm [20–22]. However, the deep learning algorithm could adapt to enormous collected data and provide better accuracy than traditional methodologies [23–25]. An essential application regarding electricity consumption prediction is associated with electricity demand which provides a real-time surveillance system to manage energy efficiency and control energy cost [26]. Although a neural network predicting system increases the computational resource and complexity for short-term prediction, the application of a deep learning machine could combine these external variables in forecasting the three-phase load power of building in the smart solar microgrid with higher accuracy. Moreover, deep learning algorithms provide a new powerful tool for the breakthroughs of many complex applications in computer visions and natural language processes. In energy applications, deep learning machines are widely utilized in wind power [27, 28], electric load demand [29, 30], and solar power [31, 32]. Cai et al. developed the deep learning RNN and CNN methodologies to predict day-ahead load power in commercial buildings [19]. The proposed method proved the higher performance which obtains a 22.6% improvement compared with traditional methods. Zhang et al. proposed the LSTM algorithm to predict the wind power from wind turbines. The author demonstrated that the LSTM acquired the greatest accuracy with 6.37% root mean square error (RMSE) than persistent methodologies [33]. In previous research, the author developed the CNN-Bi-GRU to forecaste the load power of building, which achieved more improvement compared with other deep learning methodologies [34]. Neeraj et al. developed the singular spectrum analysis LSTM for electric load forecasting [35]. The proposed method achieved superior forecasting performance in half-an-hour and one day-ahead compared with the existing method. Can et al. utilized the LSTM to achieve the highest possible time-series power load accuracy for short-term prediction [36]. Therefore, applying deep learning algorithms to time-series data proves the potential to obtain remarkable accuracy compared with the traditional method, especially the remarkable performance of the LSTM model. However, the LSTM method only processes the historical information in time-series order and disregards future information. Therefore, the bidirectional methodology is proposed to achieve higher performance and make full utilization of historical and future data [30, 37, 38]. Petrosanu Pirjan proposed the Bi-LSTM to predict the hourly month-ahead electricity consumption, which obtained excellent forecasting accuracy with 0.0495 RMSE [39]. Diyan et al. developed the Bi-LSTM for intelligent energy management to address the uncertainty of future load [40]. The Bi-LSTM could identify the data patterns in both historical and future directions to reduce energy consumption and enhance the user comfort level [40]. Aylak et al. utilized the Bi-LSTM to predict the share of renewable energy in Turkey’s electricity sources [41]. The Bi-LSTM obtained higher performance with better relative error compared with the ANN method. Liu Lin proposed the Bi-LSTM method to predict the electricity demand in the UK with the impact of the Covid-19 pandemic [42]. The research demonstrated the effectiveness of weather parameters and the number of new coronavirus tests on the predicting accuracy. Cai et al. proposed the Bi-LSTM for time-series information to improve the accuracy of short-term load prediction [43]. The Bi-LSTM obtained higher accuracy benchmarks which were 0.405% MAPE, 0.706 RMSE, and 9.341 MAE, compared with BP, LSTM, and ELM. The Bi-LSTM combined forward and backward computation to retrieve the important data pattern, proving the effectiveness in improving the forecasting ability with the time-series data [43].Therefore, the Bi-LSTM is proved to achieve higher performance in considering the different contributions of future and historical information in time-series data.Therefore, this project improves prediction accuracy, easy implementation, and better performance for a long-term run for a three-phase building load in the smart solar microgrid. The data collecting process is essential for three-phase load prediction with the increasing usage of advanced metering sensors in the smart solar microgrid. This project utilized the advanced metering infrastructure installed in the smart solar microgrid of the building for an efficient energy management system. The general framework of prediction methodology is illustrated in Figure1. This project implements a three-phase load prediction methodology with a CNN-Bi-LSTM model. The main features of this article are the data collection, preparation, feature selection, normalization, hyper-parameter approaches, and comparisons between deep learning models. The benchmark models in the deep learning methodology are the Bi-LSTM model, the LSTM model, the GRU, and the RNN model. These models are evaluated through an 80:20 data ratio for the training-validating process with 1-hour, 2-hour, and 3-hour three-phase load prediction. The predicting results are transferred to an efficient energy management system for the building. The main contributions of this project are summarized as follows:(1)
A CNN-Bi-LSTM methodology is deployed to forecast the three-phase load power for building energy management systems utilizing 1-year hourly data from the advanced metering infrastructures which were integrated in the smart solar microgrid. The first, second, and third-hour three-phase load powers are predicted in advance and transferred to the efficient energy management system for the building.(2)
All the hourly collected data by AMI, which are the solar data, the grid data, the battery data, and the weather data, were normalized and evaluated the correlation with the three-phase load power of the building, as shown in Figure1. A correlation matrix of all collected data and target features are calculated and summarized for advanced analysis and diagnostic tools.(3)
Hyper-parameter optimization (HO) is utilized to evaluate the effectiveness of setting parameters on the predicting performance of CNN-Bi-LSTM. The numbers of sequential input data were compared and evaluated to achieve the highest accuracy and stability.(4)
The performance of CNN-Bi-LSTM, Bi-LSTM, LSTM, GRU, and RNN models are evaluated by utilizing different benchmarks for training data and validating data.Figure 1
The general timeline of the predicting approach in the smart solar microgrid of building.In the following section, the prediction methodology is described. Section3 describes the hyper-parameter process for the CNN-Bi-LSTM model. The comparative performance between data models is illustrated in Section 4. The conclusion is organized in Section 5 of this article.


## 2. Methodologies

### 2.1. Prediction Methodology

In this project, the methodology of the proposed three-phase load prediction for building in a smart solar microgrid is described in the following approaches:Step 1.
In Figure1, the 1-year collected data of solar, grid, battery, load, and weather features with 1-hour intervals are utilized to predict the three-phase load power in the following 1-hour, 2-hour, and 3-hour. The solar voltage and solar current are collected hourly for the solar data. Each phase’s grid voltage and current are collected hourly in the grid data. The AMI in the battery module collects the battery voltage, the battery-charged current, battery-discharged current, battery remainder in kWh, and the percentage battery remainder. The target three-phase load is collected every hour with the weather data: the temperature, pressure, relative humidity, wind speed, wind direction in radian, and precipitation. All the collected parameters are transferred to the data storage center, which could be easily downloaded for analyzing purposes. The missing data are applied to the k-nearest neighbor method to impute.Step 2.
ll the prepared data are normalized with the min-max normalization method as(1)x^t=xt−xminxmax−xmin,where xmax and xmin are the max and min values of selected feature x. The normalized values are used to compute the correlation coefficient with the target feature as(2)ρx^t,y^t=covx^,y^σx^σy^,where x^ and y^ are the normalized values and σx^ and σy^ are the standard deviation of x and y, respectively.Step 3.
The hyper-parameter optimization (HO) methodology is implemented to evaluate the performance of Bi-LSTM models [44]. In addition, the number of sequential input data also varied from 36-hour, 48-hour, and 60-hour and compared the effectiveness to the predicting accuracy. The collected data is split into 90:10 for the training and validating process.Step 4.
The accurate performance of the CNN-Bi-LSTM model is evaluated and compared with the following Bi-LSTM, LSTM, GRU, and RNN models with the same optimized structure. The 10-fold cross-validation is utilized to effectively assess the performance of different deep learning algorithms. These models are briefly described in the next section.Step 5.
The trained model of CNN-Bi-LSTM is exported to real-time predict the three-phase load power in the smart solar microgrid for building. The predicted values are stored in the data center to compare with the actual value.
In the next section, different time-sequential predicting models are described in detail.


### 2.2. The Time-Sequential Predicting Models

The recurrent neural network (RNN) is utilized for generating sequential recommendations and can capture long-term dependent data [45–47]. The hidden layer of RNN receives the information of the previous time step and transfers it to the next sequential time. The memory ability of RNN could be especially suitable for the time-sequential problem. The disadvantage of the RNN model is the difficulty in training because of the gradient vanishing and exploding problem [48]. However, the RNN could loopily iterate over time to maintain the encoding information in the short term, and the previous input could influent the current output in the hidden state. Therefore, the RNN is a powerful model to process the time-sequential data and could be applied to forecast the three-phase load power in a smart solar microgrid in the building. The RNN could store the time-series information in the short-term and cannot maintain the long-term sequence information. The general structure of the RNN cell is shown in Figure 2 and the output of the hidden layer is presented in equation (3), where ht is the output of the hidden state from previous state ht−1, and Whh, Whx, and bh are the weight and bias matrix during the training process of RNN model.(3)ot=ht=tanhWhhht−1+Whxxt+bh.Figure 2
The general structure of RNN cell.Another powerful model for processing the time-sequential data is the Long Short-term Memory (LSTM) neural network which contains the particular storage unit and gate unit for long-term dependence [49]. The LSTM is a deep learning algorithm that could address the vanishing or exploding gradient of loss function during the optimization process by managing the recurrent information through different gates [49–51]. The general structure of LSTM is presented in Figure 3, which contains three nonlinear gate structures: forget gate, the input gate, and the output gate of the hidden state. The cell unit ct maintains the critical information in long-term dependencies. The input gate transfers the current information to the memory units, which are the input modulation gate zt and the forget gate ft. The forget gate is necessary to reset the memory unit with the input data xt and previous hidden state ht−1. The forward flow of an LSTM cell is described by the following components from (4a) to (4e), where W=Wfx,Wfh,Wix,Wih,Wox,Woh,Wzx,Wzh,Wyh is the weights matrices for different gate connections and □ denotes the element-wise product of two matrices. The cell unit of the LSTM preserves important long-term information in time-sequential data. The short-term of the hidden layer at a time t is computed as equation (4f):(4a)ft=sigmoidWfhht−1+Wfxxt+bf,(4b)it=sigmoidWixxt+Wihht−1+bi,(4c)zt=tanhWzxxt+Wzhht−1+bz,(4d)ot=sigmoidWoxxt+Wohht−1+bo,(4e)ct=ftct−1+itzt,(4f)ht=ottanhct−1.Figure 3
The general structure of LSTM cell with gates.The gated recurrent unit (GRU) is the simple version of LSTM and includes the reset gate and update gate, which improve the capability of capturing dependencies of different sequential scales [52–54]. The reset gate adjusts the incorporation of input information with the previous hidden state, while the update gate controls the preservation of the precedent information. The GRU has a shorter learning process and fewer parameters than the LSTM model because of the shortage of output gates. The output of the GRU cell is the linear interpolation between the previous output state ht−1 and the current candidate activation state h˜t.The general structure of GRU is illustrated in Figure 4 and the reset gate and update gate are computed as the following equations, where zt is the update vector, rt is the reset vector, h˜t is the candidate activation, ht is the activation of the GRU, W and b are the weight and bias matrices, respectively.(5a)zt=sigmoidht−1,xt+bz,(5b)rt=sigmoidht−1,xt+br,(5c)h˜t=tanhWhgxt+rt⊗Whhht−1+bh,(5d)ht=1−zt⊗ht−1+zt⊗h˜t.Figure 4
The general structure of GRU cell.Another powerful deep learning algorithms are the bidirectional long short-term memory (Bi-LSTM) which is illustrated in Figure5. The Bi-LSTM could process the sequential data in both forward and backward directions. The Bi-LSTM includes two layers of LSTM, which compute the sequential data in both directions. The combination of the bidirectional process enhances the ability to learn hidden features of enormous sequential input data in a deep learning machine. The output of each layer in Bi-LSTM is computed as (6a), where h⟶t and h←t are calculated according to forward and backward directions of hidden states, as in (6b) and (6c). These backward and forward hidden states could be computed similarly to LSTM. Therefore, the Bi-LSTM could simultaneously learn information about the past and the future in the time-sequential input. The advantage of the Bi-LSTM is that they could learn the sequential input on different time scales.(6a)yt=Wh⟶h⟶tm+Wh←h←tm+by,(6b)h⟶tm=h⟶t−1m,Xt=F⟶t=sigmoidW⟶fgh⟶t−1m,Xt+b⟶fg,I⟶t=sigmoidW⟶igh⟶t−1m,Xt+b⟶ig,O⟶t=sigmoidW⟶ogh⟶t−1m,Xt+b⟶og,c⟶t=F⟶t∗c⟶t−1+I⟶t∗tanhW⟶Ch⟶t−1m,Xt+b⟶C,h⟶tm=O⟶t∗tanhc⟶t.,(6c)h←tm=h←t−1m,Xt=F←t=sigmoidW←fgh←t−1m,Xt+b←fg,I←t=sigmoidW←igh←t−1m,Xt+b←ig,O←t=sigmoidW←ogh←t−1m,Xt+b←og,c←t=F←t∗c←t−1+I←t∗tanhW←Ch←t−1m,Xt+b←C,h←tm=O←t∗tanhc←t..Figure 5
Double-layer LSTM in the bidirectional LSTM network structure.


### 2.3. The Proposed Convolution Neural Network Bidirectional Long Short-Term Memory Methodology

The convolution neural network (CNN) includes sparse connectivity and weight sharing with a feedforward processing network. The CNN is a deep learning algorithm that can interpret time-dependent data in many applications, such as image classification, natural language processing, and medicine. The general CNN contains the convolutional layers and the pooling layers to achieve a meaningful information pattern regarding the massive data [55, 56]. The convolutional layers map the previous layer’s output with the output by the ReLU activation function. The pooling layer follows each convolution layer to reduce the number of sequential data. The pooling layer decreases the number of feature maps and filters the most critical information. In many applications, max pooling is generally utilized for information filtering. The CNN successfully reduces the computational difficulties in time-series processing data, extracting the most critical data pattern in sequential information. Therefore, in this project, the one-dimensional (1D) CNN is combined with the Bi-LSTM to predict the three-phase load power in a commercial building in Taiwan, as illustrated in Figure 6. The CNN layers firstly process the sequential information to extract the most critical data, and then the output patterns are continuously transferred to Bi-LSTM layers. The CNN layers could extract valuable and practical knowledge in different data regions, which could be weighted according to their importance to the three-phase load power prediction. Finally, bidirectional processes integrate such regional data through the Bi-LSTM layers. By combining the CNN and Bi-LSTM layers, both critical information in short and long distances dependency could be evaluated in the predicting output. The output of Bi-LSTM is finally transferred to two fully connected layers, which directly connect to the output layer. The primary function of fully connected layers is to generalize the extracted features to the output region with local correlations and global dependencies.Figure 6
The general structure of proposed CNN-Bi-LSTM method.The RNN, LSTM, GRU, Bi-LSTM, and CNN-Bi-LSTM are utilized in predicting the three-phase load power of a building in the smart solar microgrid. The sequentially collected input data from AMI are implemented to evaluate the performance between predicting models.


### 2.4. The Feature Selection for Collected Data

The hourly collected data utilized in this project are from a smart solar microgrid for 1 year installed on the building in the Linyuan District, Taiwan. The included inputs in the dataset are solar data, battery data, grid data, and weather data. The three-phase load power is regarded as the target variable. A correlation matrix is to understand the correlation between collected features better, as shown in Table1. The correlation table identifies the critical components that can participate in forecasting the three-phase load power with higher accuracy. The higher accurate prediction can provide essential information in the smart solar microgrid’s efficient building energy management system. All the collected features considered for the evaluation are mentioned in Figure 1. The correlation matrix demonstrates the correlation between each variable in a graphical representation. The correlation values increase with other features represented as positive values (green color). In contrast, the correlation values that decrease with additional features are defined as negative values (red color). The absolute value of the correlation represents the linear correlation with one another features.Table 1
The correlation coefficient between collected features in smart solar microgrid of building.As shown in Table1, the three-phase load is highly correlated with the three-phase grid currents, ranging from 0.4 to 0.7. These high correlations are based on the dispatch-ability of grid current to meet the three-phase load power and maintain the other power sources. This proves that by increasing the three-phase load power, the three-phase grid current increases the commitment to maintain the building’s power supply and demand balance. Another high correlation between the three-phase load power of the building and the battery-discharged current is found because they cooperate with the grid power and solar power from the battery, which all serve the power demand of the building. The following negative correlation is obtained between load power and the three-phase grid voltage. The increase in three-phase load power could reduce the grid voltage, which could be utilized as an input feature of predicting model. The subsequent highest correlation is observed between the solar currents and the building’s three-phase load powers, which are 0.101, 0.136, and 0.065, respectively. This correlation is because of adjusting the imported solar power with load increment. It means that by increasing the solar power, the injection of solar power to the load power is increased for the efficient energy management system of the building.From Table1, the solar current proved the strongest correlation with the battery-charged current, which is 0.912. This strong correlation is the ability of an efficient energy management system to store the oversupplying solar power to the battery. In addition, the solar current positively correlates with the solar voltage, which could be considered an input feature in forecasting three-phase load power. In addition, both solar voltage and current strongly correlate with the weather parameters: temperature, relative humidity, wind speed, and wind direction. All these weather parameters are employed in predicting the three-phase load power for building in the smart solar microgrid.Therefore, in this project, the building’s three-phase load power could be predicted with sequentially collected data which are the three-phase grid voltage, three-phase grid current, battery-charged current, battery-discharged current, solar voltage, solar current, temperature, wind speed, relative humidity, and wind direction. The AMI hourly collects these selected features in the smart solar microgrid installed at buildings in Taiwan.


### 2.5. Evaluating Metrics

In this project, the mean square error (MSE) and the mean absolute error (MAE) are utilized to evaluate the accuracy of the predicting models. The mean square error is selected as the loss function during the training process of predicting models. This benchmark measures the quality of prediction, which computed the square of Euclidean distance between predicting vector and the target vector. The MAE estimates the errors between two observations, the target and predicting vectors. The MSE and MAP calculate the errors in kW, and the CP is unitless.(7)MSE=1n∑i=1nyi−y^i2,(8)MAE=1n∑i=1nyi−y^i,where yi and y^i represents the vector of actual and predicted three-phase load powers, respectively.


## 2.1. Prediction Methodology

In this project, the methodology of the proposed three-phase load prediction for building in a smart solar microgrid is described in the following approaches:Step 1.
In Figure1, the 1-year collected data of solar, grid, battery, load, and weather features with 1-hour intervals are utilized to predict the three-phase load power in the following 1-hour, 2-hour, and 3-hour. The solar voltage and solar current are collected hourly for the solar data. Each phase’s grid voltage and current are collected hourly in the grid data. The AMI in the battery module collects the battery voltage, the battery-charged current, battery-discharged current, battery remainder in kWh, and the percentage battery remainder. The target three-phase load is collected every hour with the weather data: the temperature, pressure, relative humidity, wind speed, wind direction in radian, and precipitation. All the collected parameters are transferred to the data storage center, which could be easily downloaded for analyzing purposes. The missing data are applied to the k-nearest neighbor method to impute.Step 2.
ll the prepared data are normalized with the min-max normalization method as(1)x^t=xt−xminxmax−xmin,where xmax and xmin are the max and min values of selected feature x. The normalized values are used to compute the correlation coefficient with the target feature as(2)ρx^t,y^t=covx^,y^σx^σy^,where x^ and y^ are the normalized values and σx^ and σy^ are the standard deviation of x and y, respectively.Step 3.
The hyper-parameter optimization (HO) methodology is implemented to evaluate the performance of Bi-LSTM models [44]. In addition, the number of sequential input data also varied from 36-hour, 48-hour, and 60-hour and compared the effectiveness to the predicting accuracy. The collected data is split into 90:10 for the training and validating process.Step 4.
The accurate performance of the CNN-Bi-LSTM model is evaluated and compared with the following Bi-LSTM, LSTM, GRU, and RNN models with the same optimized structure. The 10-fold cross-validation is utilized to effectively assess the performance of different deep learning algorithms. These models are briefly described in the next section.Step 5.
The trained model of CNN-Bi-LSTM is exported to real-time predict the three-phase load power in the smart solar microgrid for building. The predicted values are stored in the data center to compare with the actual value.
In the next section, different time-sequential predicting models are described in detail.


## 2.2. The Time-Sequential Predicting Models

The recurrent neural network (RNN) is utilized for generating sequential recommendations and can capture long-term dependent data [45–47]. The hidden layer of RNN receives the information of the previous time step and transfers it to the next sequential time. The memory ability of RNN could be especially suitable for the time-sequential problem. The disadvantage of the RNN model is the difficulty in training because of the gradient vanishing and exploding problem [48]. However, the RNN could loopily iterate over time to maintain the encoding information in the short term, and the previous input could influent the current output in the hidden state. Therefore, the RNN is a powerful model to process the time-sequential data and could be applied to forecast the three-phase load power in a smart solar microgrid in the building. The RNN could store the time-series information in the short-term and cannot maintain the long-term sequence information. The general structure of the RNN cell is shown in Figure 2 and the output of the hidden layer is presented in equation (3), where ht is the output of the hidden state from previous state ht−1, and Whh, Whx, and bh are the weight and bias matrix during the training process of RNN model.(3)ot=ht=tanhWhhht−1+Whxxt+bh.Figure 2
The general structure of RNN cell.Another powerful model for processing the time-sequential data is the Long Short-term Memory (LSTM) neural network which contains the particular storage unit and gate unit for long-term dependence [49]. The LSTM is a deep learning algorithm that could address the vanishing or exploding gradient of loss function during the optimization process by managing the recurrent information through different gates [49–51]. The general structure of LSTM is presented in Figure 3, which contains three nonlinear gate structures: forget gate, the input gate, and the output gate of the hidden state. The cell unit ct maintains the critical information in long-term dependencies. The input gate transfers the current information to the memory units, which are the input modulation gate zt and the forget gate ft. The forget gate is necessary to reset the memory unit with the input data xt and previous hidden state ht−1. The forward flow of an LSTM cell is described by the following components from (4a) to (4e), where W=Wfx,Wfh,Wix,Wih,Wox,Woh,Wzx,Wzh,Wyh is the weights matrices for different gate connections and □ denotes the element-wise product of two matrices. The cell unit of the LSTM preserves important long-term information in time-sequential data. The short-term of the hidden layer at a time t is computed as equation (4f):(4a)ft=sigmoidWfhht−1+Wfxxt+bf,(4b)it=sigmoidWixxt+Wihht−1+bi,(4c)zt=tanhWzxxt+Wzhht−1+bz,(4d)ot=sigmoidWoxxt+Wohht−1+bo,(4e)ct=ftct−1+itzt,(4f)ht=ottanhct−1.Figure 3
The general structure of LSTM cell with gates.The gated recurrent unit (GRU) is the simple version of LSTM and includes the reset gate and update gate, which improve the capability of capturing dependencies of different sequential scales [52–54]. The reset gate adjusts the incorporation of input information with the previous hidden state, while the update gate controls the preservation of the precedent information. The GRU has a shorter learning process and fewer parameters than the LSTM model because of the shortage of output gates. The output of the GRU cell is the linear interpolation between the previous output state ht−1 and the current candidate activation state h˜t.The general structure of GRU is illustrated in Figure 4 and the reset gate and update gate are computed as the following equations, where zt is the update vector, rt is the reset vector, h˜t is the candidate activation, ht is the activation of the GRU, W and b are the weight and bias matrices, respectively.(5a)zt=sigmoidht−1,xt+bz,(5b)rt=sigmoidht−1,xt+br,(5c)h˜t=tanhWhgxt+rt⊗Whhht−1+bh,(5d)ht=1−zt⊗ht−1+zt⊗h˜t.Figure 4
The general structure of GRU cell.Another powerful deep learning algorithms are the bidirectional long short-term memory (Bi-LSTM) which is illustrated in Figure5. The Bi-LSTM could process the sequential data in both forward and backward directions. The Bi-LSTM includes two layers of LSTM, which compute the sequential data in both directions. The combination of the bidirectional process enhances the ability to learn hidden features of enormous sequential input data in a deep learning machine. The output of each layer in Bi-LSTM is computed as (6a), where h⟶t and h←t are calculated according to forward and backward directions of hidden states, as in (6b) and (6c). These backward and forward hidden states could be computed similarly to LSTM. Therefore, the Bi-LSTM could simultaneously learn information about the past and the future in the time-sequential input. The advantage of the Bi-LSTM is that they could learn the sequential input on different time scales.(6a)yt=Wh⟶h⟶tm+Wh←h←tm+by,(6b)h⟶tm=h⟶t−1m,Xt=F⟶t=sigmoidW⟶fgh⟶t−1m,Xt+b⟶fg,I⟶t=sigmoidW⟶igh⟶t−1m,Xt+b⟶ig,O⟶t=sigmoidW⟶ogh⟶t−1m,Xt+b⟶og,c⟶t=F⟶t∗c⟶t−1+I⟶t∗tanhW⟶Ch⟶t−1m,Xt+b⟶C,h⟶tm=O⟶t∗tanhc⟶t.,(6c)h←tm=h←t−1m,Xt=F←t=sigmoidW←fgh←t−1m,Xt+b←fg,I←t=sigmoidW←igh←t−1m,Xt+b←ig,O←t=sigmoidW←ogh←t−1m,Xt+b←og,c←t=F←t∗c←t−1+I←t∗tanhW←Ch←t−1m,Xt+b←C,h←tm=O←t∗tanhc←t..Figure 5
Double-layer LSTM in the bidirectional LSTM network structure.


## 2.3. The Proposed Convolution Neural Network Bidirectional Long Short-Term Memory Methodology

The convolution neural network (CNN) includes sparse connectivity and weight sharing with a feedforward processing network. The CNN is a deep learning algorithm that can interpret time-dependent data in many applications, such as image classification, natural language processing, and medicine. The general CNN contains the convolutional layers and the pooling layers to achieve a meaningful information pattern regarding the massive data [55, 56]. The convolutional layers map the previous layer’s output with the output by the ReLU activation function. The pooling layer follows each convolution layer to reduce the number of sequential data. The pooling layer decreases the number of feature maps and filters the most critical information. In many applications, max pooling is generally utilized for information filtering. The CNN successfully reduces the computational difficulties in time-series processing data, extracting the most critical data pattern in sequential information. Therefore, in this project, the one-dimensional (1D) CNN is combined with the Bi-LSTM to predict the three-phase load power in a commercial building in Taiwan, as illustrated in Figure 6. The CNN layers firstly process the sequential information to extract the most critical data, and then the output patterns are continuously transferred to Bi-LSTM layers. The CNN layers could extract valuable and practical knowledge in different data regions, which could be weighted according to their importance to the three-phase load power prediction. Finally, bidirectional processes integrate such regional data through the Bi-LSTM layers. By combining the CNN and Bi-LSTM layers, both critical information in short and long distances dependency could be evaluated in the predicting output. The output of Bi-LSTM is finally transferred to two fully connected layers, which directly connect to the output layer. The primary function of fully connected layers is to generalize the extracted features to the output region with local correlations and global dependencies.Figure 6
The general structure of proposed CNN-Bi-LSTM method.The RNN, LSTM, GRU, Bi-LSTM, and CNN-Bi-LSTM are utilized in predicting the three-phase load power of a building in the smart solar microgrid. The sequentially collected input data from AMI are implemented to evaluate the performance between predicting models.


## 2.4. The Feature Selection for Collected Data

The hourly collected data utilized in this project are from a smart solar microgrid for 1 year installed on the building in the Linyuan District, Taiwan. The included inputs in the dataset are solar data, battery data, grid data, and weather data. The three-phase load power is regarded as the target variable. A correlation matrix is to understand the correlation between collected features better, as shown in Table1. The correlation table identifies the critical components that can participate in forecasting the three-phase load power with higher accuracy. The higher accurate prediction can provide essential information in the smart solar microgrid’s efficient building energy management system. All the collected features considered for the evaluation are mentioned in Figure 1. The correlation matrix demonstrates the correlation between each variable in a graphical representation. The correlation values increase with other features represented as positive values (green color). In contrast, the correlation values that decrease with additional features are defined as negative values (red color). The absolute value of the correlation represents the linear correlation with one another features.Table 1
The correlation coefficient between collected features in smart solar microgrid of building.As shown in Table1, the three-phase load is highly correlated with the three-phase grid currents, ranging from 0.4 to 0.7. These high correlations are based on the dispatch-ability of grid current to meet the three-phase load power and maintain the other power sources. This proves that by increasing the three-phase load power, the three-phase grid current increases the commitment to maintain the building’s power supply and demand balance. Another high correlation between the three-phase load power of the building and the battery-discharged current is found because they cooperate with the grid power and solar power from the battery, which all serve the power demand of the building. The following negative correlation is obtained between load power and the three-phase grid voltage. The increase in three-phase load power could reduce the grid voltage, which could be utilized as an input feature of predicting model. The subsequent highest correlation is observed between the solar currents and the building’s three-phase load powers, which are 0.101, 0.136, and 0.065, respectively. This correlation is because of adjusting the imported solar power with load increment. It means that by increasing the solar power, the injection of solar power to the load power is increased for the efficient energy management system of the building.From Table1, the solar current proved the strongest correlation with the battery-charged current, which is 0.912. This strong correlation is the ability of an efficient energy management system to store the oversupplying solar power to the battery. In addition, the solar current positively correlates with the solar voltage, which could be considered an input feature in forecasting three-phase load power. In addition, both solar voltage and current strongly correlate with the weather parameters: temperature, relative humidity, wind speed, and wind direction. All these weather parameters are employed in predicting the three-phase load power for building in the smart solar microgrid.Therefore, in this project, the building’s three-phase load power could be predicted with sequentially collected data which are the three-phase grid voltage, three-phase grid current, battery-charged current, battery-discharged current, solar voltage, solar current, temperature, wind speed, relative humidity, and wind direction. The AMI hourly collects these selected features in the smart solar microgrid installed at buildings in Taiwan.


## 2.5. Evaluating Metrics

In this project, the mean square error (MSE) and the mean absolute error (MAE) are utilized to evaluate the accuracy of the predicting models. The mean square error is selected as the loss function during the training process of predicting models. This benchmark measures the quality of prediction, which computed the square of Euclidean distance between predicting vector and the target vector. The MAE estimates the errors between two observations, the target and predicting vectors. The MSE and MAP calculate the errors in kW, and the CP is unitless.(7)MSE=1n∑i=1nyi−y^i2,(8)MAE=1n∑i=1nyi−y^i,where yi and y^i represents the vector of actual and predicted three-phase load powers, respectively.


## 3. Parameter Optimization for CNN-Bi-LSTM

This section describes the methodology for grid search optimization for hyper-parameter values. The grid search investigates model combinations of hyperparameters that take many configurations to be tuned [44]. Finally, the best hyper-parameter model, which produces the highest performance of benchmark values, is obtained. In this project, the HO methodology is utilized to process the tuning hyperparameters, which are the number of hidden layers and the number of neurons in each layer. These setting parameters significantly affect the accurate prediction of the CNN-Bi-LSTM model. Table 2 summarizes the setting parameters for the CNN-Bi-LSTM model during the HO process. In addition, the number of sequential input data, the window size, is also varied to evaluate and compare the effects on the predicting performance, as shown in Figure 6. [50]. In the CNN-Bi-LSTM structure, the window of sequential input data goes into the input layer, which produces the three-phase load power of building at the outputs. The CNN-Bi-LSTM model trains the sequential input data, and the HO observes the predicting accuracy rate to produce the best structure. In addition, the window size varies from 36, 48, and 60 to evaluate the effectiveness of the performance of the CNN-Bi-LSTM model.Table 2
The summary of tuned parameters for Bi-LSTM.
HyperparametersList of valuesCNN layers1, 2, 3Number of filters in CNN32, 64, 128Convolutional window3Number of Bi- LSTM layers2, 3, 4Number of neural nodes64, 128, 256Number of sequential inputs36, 48, 60Full-connected layers2Learning rate0.001Batch size32Activation functionReLUKernel initializationUniformMax epochs50The number of CNN layers ranges from 1, 2, and 3, and the number of filters in convolutional layers changes from 32, 64, and 128. The convolutional window filter is set as three, and the max pooling layer is utilized. The number of Bi-LSTM varies from 2, 3, to 4 layers, and each layer contains 64, 128, or 256 neural nodes. The learning rate is 0.001, and the maximum epochs for each combination are 50. The activation function is ReLU, and the uniform Kernel initialization is utilized before the training process. In this project, the grid search is utilized to determine the effectiveness of hyper-parameter on the deep learning algorithm in forecasting the three-phase load power. All 243 combinations of hyperparameters are simulated to evaluate the effectiveness of essential parameters on the CNN-Bi-LSTM performance. The summarized hyper-parameter optimization results of all varieties are presented in Figures7 and 8. Table 3 shows the average and standard deviation for each tuning parameter.Figure 7
The hyper-parameter optimization on the effectiveness of CNN-Bi-LSTM during learning and validating operations: (a) effects of window size, (b) effects of number Bi-LSTM layers, (c) effects of number neural nodes.
(a)(b)(c)Figure 8
The hyper-parameter optimization on the capabilities of CNN-Bi-LSTM during learning and validating operations: (a) effects of CNN layers, (b) effects of number CNN filters.
(a)(b)Table 3
The summarized performance benchmarks for the deep structure CNN-Bi-LSTM model with the best hyper-parameter tuning.
MSEMAETrainingValidatingTrainingValidatingAverageStandDevAverageStandDevAverageStandDevAverageStandDevWindow size360.004190.001070.008640.001540.025990.003180.032890.00577480.004000.000920.008310.001100.025170.002900.031490.00598600.004160.001510.008470.001280.025710.005300.032110.00686Number layers20.003290.000660.008500.001490.023370.002390.031700.0055330.004020.001250.008500.001130.025260.004730.032210.0051940.005040.000860.008420.001340.028240.002580.032570.00769Neural nodes640.004410.001520.008430.001220.026720.005180.032830.005781280.004040.001010.008390.001100.025410.003080.031640.006642560.003880.000900.008590.001610.024720.002910.031930.00622CNN layers10.004280.001420.007510.000850.025960.005060.030590.0064220.004200.000890.008650.000820.026120.002630.032540.0051430.003850.001170.009250.001500.024750.003620.033280.00677Filters in CNN320.004490.001400.008460.001370.026970.004910.033270.00638640.004030.000910.008470.001130.025310.002760.031070.005671280.003790.001110.008480.001470.024480.003440.032150.00623Figure7(a) presents the boxplots of the RSE and MAE during the training and validating activities for the window size, the number of Bi-LSTM layers, and the nodes in each layer. The 48-sequential input improves the prediction accuracy compared with other configurations for tuning the window size hyper-parameter. The 48-window size obtains the minor average and standard deviation of MSE in both processes, as shown in Table 3. Moreover, the 48-window size reduces the average MAE in the training operations compared with 36- and 60-window sizes. This concludes that the 48-sequential window size could be considered a suboptimal solution in predicting the load power, which obtains the highest performance and increases the stability in predicting results. Moreover, the hyper-parameter of window size potentially decreases the average and standard deviation MSE and MAE in both learning and validating operations. The window size hyper-parameter tuning is more sensitive to the performance of CNN-Bi-LSTM in forecasting the three-phase load power.In the parameter tuning for number layers, the 2-layers obtain lower statistical MSE and MAE values than the 3- and 4-layers, as shown in Figure7(b) and Table 3. The 2-layers parameter reduces the average and standard deviation of MSE and MAE compared with 3- and 4-layers, which are 0.00329 and 0.00066 MSE, and 0.00850 and 0.00149 MAE for average and standard deviation values, respectively. Although the 2-layer could not enhance the accuracy in the validating operation, the 2-layer obtains better stability compared with 3- and 4-layers. The 2-layer significantly improves the predicting accuracy and stability of CNN-Bi-LSTM with better statistical benchmarks. The increment of Bi-LSTM layers could lead to a decrement in the predicting accuracy of CNN-Bi-LSTM. Therefore, the number layer hyper-parameter tuning is significantly sensitive to the outcome of the predicting model, and the 2-layers are considered the optimum solution to achieve the most significant enhancement.Moreover, the neural node’s hyper-parameter performances are also summarized in Table3 and Figure 7(c). The experiment results prove that the 256-neural nodes achieve the most enhancement compared with 64- and 128-hyperparameters. The 256-neural nodes perform the optimum performance during the training operation, which is 0.00388 and 0.00090, and 0.02472 and 0.00291 for average and standard deviation MSE and MAE, respectively. The 256-neural node hyper-parameter is more sensitive to the performance of CNN-Bi-LSTM compared with 64- and 128-neural nodes. The experiment result proves that the neural node hyper-parameter potentially improves the statistical benchmarks in the predicting performance. Therefore, the 256-neural node is considered appropriate setting parameters for predicting three-phase load power in CNN-Bi-LSTM.Figure8(a) illustrates the effect of the CNN layers on the overall performance CNN-Bi-LSTM model. The box plot chart proves that the number of CNN layers has a significant relative impact against the increase in the predicting accuracy in general 253 combinations. Although the 2-CNN layers could not achieve the average MSE and MAE in both learning and validating processes, this setting parameter obtained a lower standard deviation in both validating benchmarks. The predicting accuracy tends to oscillate with an increase in CNN layers. While the 3-CNN layer decreases the MSE and MAE error between predicting and actual values, the standard deviation of MSE and MAE increases significantly. Therefore, the 2-CNN layers can improve the performance even though the average benchmarks increase, and 2-CNN is an appropriate setting parameter for the CNN-Bi-LSTM methodology.The box plot in Figure8(b) shows the overall effect of the number of CNN filters on the 253 combinations for hyper-parameter optimization.The number of CNN filters immensely contributes to the performance of CNN-Bi-LSTM in forecasting the three-phase load power. The 64-filter in CNN layers obtains an average MSE and MAE that is not greater than 32- and 128-filters in both validating and training processes but achieves an optimum standard deviation of statistical benchmarks. The addition of the CNN filter oscillates the general performance of CNN-Bi-LSTM in predicting the load power. Therefore, the 64-CNN filter is considered a global optimum solution of CNN-Bi-LSTM during the hyper-parameter optimization.In this section, the HO is performed by simulating 243 combinations in CNN-Bi-LSTM, which combine the number of window sizes, the number of Bi-LSTM layers, the neural cell, and the number of CNN layers, and the number of filters. The learning and validating performance with MSE and MAE benchmarks are summarized. The CNN-Bi-LSTM achieves the optimum average and standard deviation benchmark with 48 window sizes, 2 Bi-LSTM layers, 256 neural nodes, 2-CNN layers, and 64 filters in each CNN. These setting parameters are utilized to develop the architecture of CNN-Bi-LSTM to predict the three-phase load power in the smart solar microgrid.


## 4. Comparing Simulation from Various Techniques

In this section, the optimum CNN-Bi-LSTM architecture is evaluated and compared with other algorithms, the RNN, GRU, LSTM, and Bi-LSTM. These methodologies use the network structure with the best hyper-parameter optimization. A comparison of the selected methods with the 10-fold cross-validation is implemented to evaluate their performance over the entire dataset. The collected data is randomly split into 10 equal parts to assess with 9 training sets and 1 test set. The simulating performance with 10-fold cross-validation is presented in Figure9 and Table 4. Furthermore, the MSE and MAE during the training and testing process are summarized and illustrated.Figure 9
The summarized performance for different deep learning algorithm with best hyper-parameter structures.Table 4
The summarized comparison between the proposed methods.
MSEMAETrainingTestingTrainingTestingMin trgMax trgAvg trgMin testMax testAvg testMin trgMax trgAvg trgMin testMax testAvg testRNN0.00620.00730.00670.00530.00850.00620.03290.03650.03430.02420.05150.0365GRU0.00300.00370.00320.00560.00810.00640.02280.02660.02430.02390.03730.0294LSTM0.00330.00410.00370.00620.00710.00670.02330.02660.02500.02550.03710.0294Bi-LSTM0.00330.00400.00360.00590.00700.00650.02370.02580.02450.02290.02990.0266CNN-Bi-LSTM0.00160.00260.00210.00530.00650.00620.02110.02690.02390.02210.03010.0254Improvement (%)MSEMSEMin trgMax trgAvg trgMin testMax testAvg testMin trgMax trgAvg trgMin testMax testAvg testRNN73.3964.6968.360.6922.961.4135.8126.4230.268.4341.6130.36GRU44.2329.8434.584.6319.934.407.19−1.071.847.3119.2113.52LSTM50.6337.1243.7014.318.198.519.12−1.044.5113.2118.8713.49Bi-LSTM50.3035.6941.9910.307.435.1210.94−4.002.463.42−0.624.28CNN-Bi-LSTMThe RNN model produced the most outstanding performance with 0.0067 MSE and 0.0343 MAE during training operations and 0.0062 MSE and 0.0365 MAE during testing. The best performances of GRU methodology are 0.0032 and 0.0064 MSE during the training operations, and 0.0243 and 0.0294 MAE during the testing data, on average. Moreover, the LSTM algorithm could provide better predicting accuracy, which are 0.0037 MSE, 0.0250 MAE in training operation, and 0.0067 MSE, 0.0294 MAE in the testing data. The average MSE and MAE with the Bi-LSTM algorithm with 10-fold cross-validation are 0.0036, 0.0065 average MSE and 0.0245, 0.0266 average MAE during the training and testing procedure. The predicting accuracy of LSTM and Bi-LSTM methods are almost identical. However, the GRU model, which combines the hidden state and cell state, is excellent at forecasting the three-phase load power. The CNN-Bi-LSTM outperforms another deep learning algorithm which obtains the most optimum error during the training and testing procedures. The CNN-Bi-LSTM achieves the best performance compared with other methodologies, which are 0.0021, 0.0062, 0.0239, and 0.0254 average MSE and MAE through the training and testing procedures. Therefore, the combination of CNN and Bi-LSTM enhances the predicting sensitivity of testing operations due to the appropriate selection of hyper-parameter optimization. Moreover, the CNN with the feature extraction in enormous input data is a suitable combination of Bi-LSTM, improving the predicting results in the testing operation.The CNN-Bi-LSTM with the best architecture through hyper-parameter optimization prioritizes other models and provides the most excellent performance during the training and testing procedures.Compared with previous algorithms, the CNN-Bi-LSTM could significantly improve the prediction of the three-phase load power. The CNN-Bi-LSTM obtains 41.99% and 5.12% MSE improvement compared with the Bi-LSTM methodology with the feature extraction in CNN layers. With the same deep learning structure, the CNN-Bi-LSTM outperforms the traditional Bi-LSTM, which proves the higher effectiveness of the combined hybrid method. The CNN layers could recognize the three-phase load patterns with large-scale window sizes. The traditional Bi-LSTM could not perform well in identifying the critical information patterns in massive input data. Compared with conventional LSTM, the combination between CNN and Bi-LSTM could perform very well with outperformed average predicting MSE and MAE respectively of 43.70%, 4.51%, 8.51%, and 13.49% during testing and validating operations. Although the Bi-LSTM could not work well in predicting the three-phase load power compared with LSTM, the combination with CNN could provide better behavior of the data information. In general, the combination between CNN and Bi-LSTM obtains the highest enhancements, which are 68.36% and 8.81% average MSE and 30.26% and 36.36% average MAE during the testing and validating operations.Although the collected data from the solar power plant could not provide the standard dataset for the training and validating process, the 10-fold cross-validation significantly affected the overall performance of deep learning algorithms. The CNN-Bi-LSTM can overcome an unbalanced prediction class structure that achieves super performance compared with other algorithms. Although the optimization processes require the computational cost to complete the appropriate hyper-parameter combinations, the simulating performances prove that the CNN-Bi-LSTM could improve performance with better accuracy and higher stability. Therefore, the application of CNN-Bi-LSTM could be considered for load forecasting for performance enhancement in the smart solar microgrid. Figure10 presents the comparative MSE value in testing data with different time horizons. All previous prediction approaches have higher MSE values for different time horizons. The CNN-Bi-LSTM prediction shows the best performance with higher accuracy scores. Therefore, the CNN-Bi-LSTM is the best possible approach for effectively predicting three-phase load power in a smart solar plant.Figure 10
The comparative MSE with different time horizons.


## 5. Conclusions

This project demonstrates applying CNN-Bi-LSTM in forecasting the three-phase load power in a smart solar plant. The AMI integrated into the smart solar plant has collected sensitive data for more than 1 year in Taiwan. The hyper-parameter optimization is deployed to achieve the optimum structure of the CNN-Bi-LSTM method. Moreover, the experiment results prove that the CNN-Bi-LSTM obtained the highest enhancement compared with RNN, GRU, LSTM, and Bi-LSTM in predicting the three-phase load power. Therefore, the CNN-Bi-LSTM could produce better accuracy than recent deep learning algorithms. The experiment results prove that the hybrid CNN and Bi-LSTM achieve a maximum enhancement of 68.36% and 8.81% average MSE and 30.26% and 36.36% average MAE during the testing and validating operations. The CNN-Bi-LSTM could be utilized to effectively predict three-phase load power in the smart solar plant project with higher accuracy and better stability than other algorithms.



---

*Source: 2870668-2022-09-16.xml*