# Development of Modified LSTM Model for Reservoir Capacity Prediction in Huanggang Reservoir, Fujian, China

Authors: Bibo Dai; Jiangbin Wang; Xiao Gu; Chunyan Xu; Xin Yu; Haosheng Zhang; Canming Yuan; Wen Nie
Journal: Geofluids
 (2022)

---

## Abstract

The Huanggang Reservoir capacity is affected by a variety of factors. In order to accurately understand the Huanggang Reservoir capacity change, we develop a new hydrological prediction model based on the LSTM (Long-Short-Term Memory) method, which is used to predict the capacity of the reservoir. In this modified model, we choose to input multidimensional factors, two fully connected layers, selecting the optimal number of the hidden neurons, the optimizer, and adding the attention mechanism. The result of using the Developed LSTM and usual LSTM shows that the prediction curve of the Developed LSTM model can fit the true value better than the usual LSTM model, and the mean relative error of the Developed LSTM model decreased by 1.15%-3.82%, comparing with the usual LSTM model. Thus, we realize that the Developed LSTM model can make accurately prediction in some reservoir capacity estimations.


---

## Body

## 1. Introduction

Reservoir, a water conservancy project, which plays an important role in agricultural irrigation, flood control during flood season, hydroelectric power generation, and water supply [1–3]. Therefore, the safety of reservoirs will directly affect economic development and people’s livelihood. With the environment changing, extreme weather occurring frequently, the probability of reservoir safety hazards has greatly increased [4–6].With climate change and the country’s emphasis on water conservancy projects, the prediction of reservoir capacity has become a hot research spot in recent years. To clarify the changes in reservoir water volume, a large number of fruitful studies have been carried out. There are three commonly used methods of reservoir volume forecasting: (1) Reservoir volume prediction based on the principle of reservoir flood regulation calculation; (2) Hydrodynamic methods; (3) Artificial Neural Networks predict the complex nonlinear reservoir capacity [7]. The first method is to estimate the water level using the principle of reservoir flood regulation calculation. The principle of reservoir flood regulated calculation was used widely and predicted the water level of the reservoir initially. If the reservoirs’ inflow is back-calculated by the balance equation, the errors would occur [8]. The second method depends on the hydrodynamic method. An excellent hydrological model can simulate the water level pretty well. However, the hydrodynamic model requires amounts of data, and the exactitude of the input factors will determine the accuracy of the model. In addition, the hydrodynamic process is complex and changeable, which could increase the difficulty of reservoir capacity simulation. These factors limit the development of this hydrodynamic model [9, 10]. For the third method, the reservoir capacity prediction depends on the Artificial Neural Networks. With the continuous development of various monitoring and forecasting technologies [11–14], especially the emergence of the LSTM (Long Short-term Memory) neural network, it stores historical information by cyclic feedback and has a strong ability to solve time series problems. It has attracted attention in the prediction of hydrological time series problems [15, 16]. Lin [17] et al. compared the LSTM algorithm with the hydrological physical model who found the LSTM model is better than the traditional physical model; Mich [18] et al. compared the accuracy of several different neural network algorithms in water level prediction; Chen [19] Compared the water level prediction with LSTM and SVM models, it found that LSTM has advantages over general neural networks; Gu [20] et al. used LSTM to predict the water level of Poyang Lake by using 7 lakes’ data. The accuracy of predicting the water level of Poyang Lake based on the amount of rivers’ data that is much better than one single river ‘s. The traditional LSTM model mainly focuses on the prediction of time series of hydrological information, while external factors are often ignored. Factors, which are closely related to the prediction, are usually not involved in the structure of the model. This ignorance directly made the model prediction have a large deviation from the real when the correlation factor changes greatly. Therefore, considering the influence of the correlation factor on the model, we optimize the LSTM model for the prediction of reservoir capacity.In the study area, Huanggang Reservoir capacity (Fujian, China) has many influence factors which are complex nonlinear changing. LSTM model usually predicts the storage capacity only by one input factor (historical reservoir capacity). We use the multiple inputting factors of developed LSTM including historical storage capacity and rainfall. In addition, an attention mechanism is introduced in the developed LSTM model, which can give different weights to different factors in the process of LSTM decoding and coding, so that the calculation results are more accurate. In the following, we explore whether they can improve the accuracy of the model in reservoir capacity prediction.


## 2. Materials and Methods

The study area is located in the upper reaches of the Jiulong River water system, which belongs to Longyan City, Fujian Province, China. The city has a population of 300,000, a land of about 150,000 acres, many industrial and mining enterprises [21]. Huanggang Reservoir is an important reservoir that is mainly used for irrigation, combined with comprehensive utilization of flood control, power generation, and aquaculture [22, 23]. The study area is affected by a subtropical monsoon climate with sufficient rainfall throughout the year. Sufficient water flow, typhoons, heavy rainfall, and other extreme weather usually occur in summer [24]. The total storage capacity of the reservoir area reaches 28.6 million m3 [25, 26], and the location map of Huanggang Reservoir is shown in Figure 1.Figure 1
Location of Huanggang Reservoir.


### 2.1. Research data

The daily hydrological data of the Huanggang Reservoir from 2001-to 2019 was obtained from the Longyan Reservoir Management Center. These data included the historical storage capacity of the reservoir and the rainfall value around Huanggang Reservoir. The relationship between rainfall and reservoir capacity from 2001-to 2019 is shown in Figure2.Figure 2
The data of reservoir capacity and rainfall from 2001 to 2019.


### 2.2. Basic principles of LSTM

LSTM is a special deep learning neural network that is based on RNN, which makes up for the problem of gradient disappearance and explosion in the process of long sequence training (Figure3). In the hydrological prediction of long time series, the LSTM model has been widely used [27]. Compared with other algorithm models, it has some significant advantages and the simulation effect is excellent in hydrological prediction [28].Figure 3
LSTM model structure [27].The long-short-term memory neural network is composed of memory cells. Its key is the state of these memory cells and the structure of the “gate” in the process. The cell state transmits the information in the time series that likes a conveyor belt and then passes through three gates: the input gate, the forget gate, and the output gate. The input gate determines how much input data of the network at the current moment is saved in the unit state; the forget gate determines how much of the unit state at the previous moment is saved to the current moment; the output gate determines the number of the control unit state which is outputted to the current output value of LSTM.The basic process of LSTM neural network information transmission is as follows (only one neural network node is used as an example to illustrate): (1) The forget gate controls the input information through the sigmoid function to determine which information can enter the current cell state unit; (2) A candidate value that can be added to the current cell state unit is generated by the tanh layer, and then the candidate value, the input information of the previous step, and the cell unit state of the previous moment are bitwise operated to obtain a new cell unit state value; (3) After obtaining the initial output information through the sigmoid function, the tanh layer is used to process the new cell unit state value to make it between -1 and 1. The information of the initial output and the state value of the processed cell unit are calculated and output.(1)it=σWxixt+Whiht−1+bift=σWxfxt+Whfht−1+bfot=σWxoxt+Whoht−1+boct=ft×ct−1+it×tanhWxcxt+Whcht−1+bcht=ot×tanhctit - the output value of the hidden layer at the previous moment in the input gate and the value of the input value at the current moment processed by the sigmoid function; ft - the output value of the hidden layer at the previous moment in the forget gate and the value of the input value at the current moment processed by the sigmoid function; ot - the output value of the hidden layer at the previous moment in the output gate and the value of the input value at the current moment processed by the sigmoid function; ct - the state value of the cell unit after the update at the current moment; ht - the output value of the hidden layer at the current moment; σ and tanh - cyclic activation functions; ht-1 - the output value of the hidden layer at the previous moment; xt - the input value at the current moment; W - different weight matrices; b—different bias parameters.


#### 2.2.1. LSTM neural network backpropagation

The LSTM neural network model finds the optimal solution by continuous optimization and making the model converge. The Loss function is used to compare the true value and the prediction and it is an important indicator for evaluating the quality of models. The Loss function, which reaches an infinitely close to 0 degree, determines the pros and cons of the model. The neural network backpropagation algorithm uses gradient descent to iteratively update the weight value of the neural network:(2)Loss=1m∑i=1myi−y¯i2In the formula, Loss is the Loss function,yi and y^i are the true value and predicted value at the time i, respectively.


### 2.3. Data processing method

We got the Developed LSTM model multidimensional inputting factors historical reservoir capacity and rainfall that are from 2001 to 2018. Due to the unusual characteristics of the original data (the historical reservoir capacity and rainfall), we processed the original data, deleted outliers, supplemented missing values, and minimized the abnormal samples of the data that had a large negative impact on the calculation results, such as the model calculation does not converge, the expected effect of the model is worse and other phenomena. To ensure that the data are in the same dimension, we normalize the data as eq (3):
(3)X∗=X−XminXmax−XminHere we adopt the Min-Max Normalization method. In the formula,Xi is the normalized result, X is the original data, Xmax is the maximum value in the original data, and Xmin is the minimum value in the original data.


## 2.1. Research data

The daily hydrological data of the Huanggang Reservoir from 2001-to 2019 was obtained from the Longyan Reservoir Management Center. These data included the historical storage capacity of the reservoir and the rainfall value around Huanggang Reservoir. The relationship between rainfall and reservoir capacity from 2001-to 2019 is shown in Figure2.Figure 2
The data of reservoir capacity and rainfall from 2001 to 2019.


## 2.2. Basic principles of LSTM

LSTM is a special deep learning neural network that is based on RNN, which makes up for the problem of gradient disappearance and explosion in the process of long sequence training (Figure3). In the hydrological prediction of long time series, the LSTM model has been widely used [27]. Compared with other algorithm models, it has some significant advantages and the simulation effect is excellent in hydrological prediction [28].Figure 3
LSTM model structure [27].The long-short-term memory neural network is composed of memory cells. Its key is the state of these memory cells and the structure of the “gate” in the process. The cell state transmits the information in the time series that likes a conveyor belt and then passes through three gates: the input gate, the forget gate, and the output gate. The input gate determines how much input data of the network at the current moment is saved in the unit state; the forget gate determines how much of the unit state at the previous moment is saved to the current moment; the output gate determines the number of the control unit state which is outputted to the current output value of LSTM.The basic process of LSTM neural network information transmission is as follows (only one neural network node is used as an example to illustrate): (1) The forget gate controls the input information through the sigmoid function to determine which information can enter the current cell state unit; (2) A candidate value that can be added to the current cell state unit is generated by the tanh layer, and then the candidate value, the input information of the previous step, and the cell unit state of the previous moment are bitwise operated to obtain a new cell unit state value; (3) After obtaining the initial output information through the sigmoid function, the tanh layer is used to process the new cell unit state value to make it between -1 and 1. The information of the initial output and the state value of the processed cell unit are calculated and output.(1)it=σWxixt+Whiht−1+bift=σWxfxt+Whfht−1+bfot=σWxoxt+Whoht−1+boct=ft×ct−1+it×tanhWxcxt+Whcht−1+bcht=ot×tanhctit - the output value of the hidden layer at the previous moment in the input gate and the value of the input value at the current moment processed by the sigmoid function; ft - the output value of the hidden layer at the previous moment in the forget gate and the value of the input value at the current moment processed by the sigmoid function; ot - the output value of the hidden layer at the previous moment in the output gate and the value of the input value at the current moment processed by the sigmoid function; ct - the state value of the cell unit after the update at the current moment; ht - the output value of the hidden layer at the current moment; σ and tanh - cyclic activation functions; ht-1 - the output value of the hidden layer at the previous moment; xt - the input value at the current moment; W - different weight matrices; b—different bias parameters.


### 2.2.1. LSTM neural network backpropagation

The LSTM neural network model finds the optimal solution by continuous optimization and making the model converge. The Loss function is used to compare the true value and the prediction and it is an important indicator for evaluating the quality of models. The Loss function, which reaches an infinitely close to 0 degree, determines the pros and cons of the model. The neural network backpropagation algorithm uses gradient descent to iteratively update the weight value of the neural network:(2)Loss=1m∑i=1myi−y¯i2In the formula, Loss is the Loss function,yi and y^i are the true value and predicted value at the time i, respectively.


## 2.2.1. LSTM neural network backpropagation

The LSTM neural network model finds the optimal solution by continuous optimization and making the model converge. The Loss function is used to compare the true value and the prediction and it is an important indicator for evaluating the quality of models. The Loss function, which reaches an infinitely close to 0 degree, determines the pros and cons of the model. The neural network backpropagation algorithm uses gradient descent to iteratively update the weight value of the neural network:(2)Loss=1m∑i=1myi−y¯i2In the formula, Loss is the Loss function,yi and y^i are the true value and predicted value at the time i, respectively.


## 2.3. Data processing method

We got the Developed LSTM model multidimensional inputting factors historical reservoir capacity and rainfall that are from 2001 to 2018. Due to the unusual characteristics of the original data (the historical reservoir capacity and rainfall), we processed the original data, deleted outliers, supplemented missing values, and minimized the abnormal samples of the data that had a large negative impact on the calculation results, such as the model calculation does not converge, the expected effect of the model is worse and other phenomena. To ensure that the data are in the same dimension, we normalize the data as eq (3):
(3)X∗=X−XminXmax−XminHere we adopt the Min-Max Normalization method. In the formula,Xi is the normalized result, X is the original data, Xmax is the maximum value in the original data, and Xmin is the minimum value in the original data.


## 3. Developed LSTM model structure

In the Developed LSTM structure, we considered a variety of correlation factors as the model inputting, getting the optimal number of hidden neurons and the best type of the optimizer in the LSTM model, and the introduction of attention mechanism in the model.


### 3.1. Model structure

The model was structured by using a both way two-layer fully connected layer and multi-factor inputting parameter optimized. We selected the two factors including Huanggang reservoir capacity and the rainfall data as the input factor from 2013-2018. The data from 2013-2016 was training set, while the data of 2017-2018 was used as the test data to predict the changes in reservoir capacity for 7days, 14 days, and 30 days, respectively. The developed LSTM neural network model structure is shown in Figure4. xi is the inputting factor. It changes the dimension of the xi matrix, xi ={ai,ci}, ai=storage capacity, ci=rainfall, to realize multi-factor input; Vi⟵ stands for data to participate in forward operation; Vi⟶ stands for data to participate in opposite operation; Si⟵, Si⟶ represents the value of the hidden layer in forward and opposite operation; Viis the value after attention processing; Y is the prediction result in Developed LSTM model.Figure 4
Developed LSTM model structure diagram.


### 3.2. Model parameter optimization

In the model parameter optimization processing, we selected optimal number of hidden neurons, the activation function, the optimizers and introduced the attention mechanism in the LSTM model.


#### 3.2.1. The influence of the number of hidden neurons

The LSTM algorithm has high requirements for the number of hidden neurons [28]. The number of hidden neurons is one of the most important factors that affect the accuracy of the LSTM model. If only few neurons were used in the hidden layer, the model could not be fully trained; while too many hidden neurons would cause a time-consuming training, which may lead the model to take a phenomenon of non-convergence or over-fitting.Finding the appropriate number of neurons had an important impact on the quality of model training. We took a total of 10 kinds of neural network numbers (1, 10, 20, 30, 50, 100, 150, 200, 500, 800) to predict the future reservoir capacity by inputting the historical storage capacity and rainfall of the Huanggang Reservoir. It found that the RMSE value of the 1, 10, 20 items varies from 1 to 7. And the RMSE value of 30 and 50 items ranges from 0.975 to 0.27 and the results have great fluctuations (in the box plot would have a negative effect). Thus, the result of 100-800 items are only considered as shown in Figure5.Figure 5
Hidden Neuron box plot.With the number of neurons increase, the RMSE value of the model becomes smaller and gradually stabilizes, and the number of abnormal points decreases. When the number of neurons exceeds 500 to 800, the increased neurons number has a small effect on the model and the number of outliers becomes more, so we finally use 500 hidden neurons.


#### 3.2.2. The influence of choosing a good optimizer

The optimizer plays a vital role in improving the speed of model training and accelerating model convergence to find the optimal solution [29–31]. Increasing of complex neural networks directly lead to huge model parameters. A good optimizer can speed up the update and calculate the network parameters output of the model so that the model can approximate and find the optimal solution. We randomly selected 7 kinds of functions (“adam”, “adagrad”, “adamax”, “SGD”, “Nadam”, “Rmsprop”, “Adadelta” 7 kinds of optimizers), run 7 times under random conditions, and recorded the RMSE generated in each operation (Figure 6).Figure 6
Optimizers’ RMSE value.The running results of 7 kinds of optimizers shows that the “SGD” optimizer was the worst since there are many failures to converge. Although the results of the “Adadelta” and “Nadam” optimizers have converged, the RMSE was too large. The result greater than 1 means the effect is very poor. The effect of the remaining three optimizers is well, and the value of RMSE is less than 1. The calculation result of “Adam” is more stable and works better by comparing with the other two optimizers, so the suitable optimizer is “Adam”.According to the description of hyperparameters in sections 3.2.1 and 3.2.2, and combined with the experimental objects (Huanggang Reservoir and Hejiapo Reservoir), we set appropriate hyperparameters to train each model to convergence. The final hyperparameter selections are shown in Table1.Table 1
Hyperparameters of LSTM.
Experimental objectsHuanggang ReservoirHejiapo ReservoirSingle-factorTwo-factorSingle-factorTwo-factorNumber of hidden layers2222Number of neurons in hidden layer 1500500100100Number of neurons in hidden layer 25005002020Activation function of hidden layer 1tanhtanhtanhtanhActivation function of hidden layer 2relurelulinearlinearOptimizeradamadamadamadamTraining times20060500240


#### 3.2.3. Introduction of attention mechanism

In the LSTM model, the same vector is assigned to each input unit during the decoder decoding process [31]. In the encoder process, the output is the same fixed-length semantics. In the long-short-term prediction process of the model, these factors directly limit the model’s accuracy, so we considered adding the attention mechanism into LSTM. The attention mechanism comes from the human learning process, in which a certain part of our learning always appears to be the most important content. To speed up learning efficiency and improve accuracy, we focus our attention on that part of the information and suppress other irrelevant information in the re-learning process. The weights, which are assigned by the attention mechanism to the input sequence, can determine the most relevant aspects that affect the prediction data, thus improving prediction accuracy. The attention mechanism can effectively improve the situation in which the LSTM loses information because of long sequences, and simultaneously replace the original method of randomly assigning weights with that of assigning probabilities [32, 33]. Firstly, the attention machine value obtains a proportion by calculating the similarity between each output and input. Then, the mechanism accumulates all the inputs based on these factors’ proportion. By comparing the LSTM model with the attention and without the attention, we run 7 times under the same initial conditions and recorded the RMSE (Figure 7). We find that input of the attention mechanism in LSTM has a significant improvement than the LSTM without attention.Figure 7
Comparison plot of RMSE values of LSTM models with and without attention.


## 3.1. Model structure

The model was structured by using a both way two-layer fully connected layer and multi-factor inputting parameter optimized. We selected the two factors including Huanggang reservoir capacity and the rainfall data as the input factor from 2013-2018. The data from 2013-2016 was training set, while the data of 2017-2018 was used as the test data to predict the changes in reservoir capacity for 7days, 14 days, and 30 days, respectively. The developed LSTM neural network model structure is shown in Figure4. xi is the inputting factor. It changes the dimension of the xi matrix, xi ={ai,ci}, ai=storage capacity, ci=rainfall, to realize multi-factor input; Vi⟵ stands for data to participate in forward operation; Vi⟶ stands for data to participate in opposite operation; Si⟵, Si⟶ represents the value of the hidden layer in forward and opposite operation; Viis the value after attention processing; Y is the prediction result in Developed LSTM model.Figure 4
Developed LSTM model structure diagram.


## 3.2. Model parameter optimization

In the model parameter optimization processing, we selected optimal number of hidden neurons, the activation function, the optimizers and introduced the attention mechanism in the LSTM model.


### 3.2.1. The influence of the number of hidden neurons

The LSTM algorithm has high requirements for the number of hidden neurons [28]. The number of hidden neurons is one of the most important factors that affect the accuracy of the LSTM model. If only few neurons were used in the hidden layer, the model could not be fully trained; while too many hidden neurons would cause a time-consuming training, which may lead the model to take a phenomenon of non-convergence or over-fitting.Finding the appropriate number of neurons had an important impact on the quality of model training. We took a total of 10 kinds of neural network numbers (1, 10, 20, 30, 50, 100, 150, 200, 500, 800) to predict the future reservoir capacity by inputting the historical storage capacity and rainfall of the Huanggang Reservoir. It found that the RMSE value of the 1, 10, 20 items varies from 1 to 7. And the RMSE value of 30 and 50 items ranges from 0.975 to 0.27 and the results have great fluctuations (in the box plot would have a negative effect). Thus, the result of 100-800 items are only considered as shown in Figure5.Figure 5
Hidden Neuron box plot.With the number of neurons increase, the RMSE value of the model becomes smaller and gradually stabilizes, and the number of abnormal points decreases. When the number of neurons exceeds 500 to 800, the increased neurons number has a small effect on the model and the number of outliers becomes more, so we finally use 500 hidden neurons.


### 3.2.2. The influence of choosing a good optimizer

The optimizer plays a vital role in improving the speed of model training and accelerating model convergence to find the optimal solution [29–31]. Increasing of complex neural networks directly lead to huge model parameters. A good optimizer can speed up the update and calculate the network parameters output of the model so that the model can approximate and find the optimal solution. We randomly selected 7 kinds of functions (“adam”, “adagrad”, “adamax”, “SGD”, “Nadam”, “Rmsprop”, “Adadelta” 7 kinds of optimizers), run 7 times under random conditions, and recorded the RMSE generated in each operation (Figure 6).Figure 6
Optimizers’ RMSE value.The running results of 7 kinds of optimizers shows that the “SGD” optimizer was the worst since there are many failures to converge. Although the results of the “Adadelta” and “Nadam” optimizers have converged, the RMSE was too large. The result greater than 1 means the effect is very poor. The effect of the remaining three optimizers is well, and the value of RMSE is less than 1. The calculation result of “Adam” is more stable and works better by comparing with the other two optimizers, so the suitable optimizer is “Adam”.According to the description of hyperparameters in sections 3.2.1 and 3.2.2, and combined with the experimental objects (Huanggang Reservoir and Hejiapo Reservoir), we set appropriate hyperparameters to train each model to convergence. The final hyperparameter selections are shown in Table1.Table 1
Hyperparameters of LSTM.
Experimental objectsHuanggang ReservoirHejiapo ReservoirSingle-factorTwo-factorSingle-factorTwo-factorNumber of hidden layers2222Number of neurons in hidden layer 1500500100100Number of neurons in hidden layer 25005002020Activation function of hidden layer 1tanhtanhtanhtanhActivation function of hidden layer 2relurelulinearlinearOptimizeradamadamadamadamTraining times20060500240


### 3.2.3. Introduction of attention mechanism

In the LSTM model, the same vector is assigned to each input unit during the decoder decoding process [31]. In the encoder process, the output is the same fixed-length semantics. In the long-short-term prediction process of the model, these factors directly limit the model’s accuracy, so we considered adding the attention mechanism into LSTM. The attention mechanism comes from the human learning process, in which a certain part of our learning always appears to be the most important content. To speed up learning efficiency and improve accuracy, we focus our attention on that part of the information and suppress other irrelevant information in the re-learning process. The weights, which are assigned by the attention mechanism to the input sequence, can determine the most relevant aspects that affect the prediction data, thus improving prediction accuracy. The attention mechanism can effectively improve the situation in which the LSTM loses information because of long sequences, and simultaneously replace the original method of randomly assigning weights with that of assigning probabilities [32, 33]. Firstly, the attention machine value obtains a proportion by calculating the similarity between each output and input. Then, the mechanism accumulates all the inputs based on these factors’ proportion. By comparing the LSTM model with the attention and without the attention, we run 7 times under the same initial conditions and recorded the RMSE (Figure 7). We find that input of the attention mechanism in LSTM has a significant improvement than the LSTM without attention.Figure 7
Comparison plot of RMSE values of LSTM models with and without attention.


## 3.2.1. The influence of the number of hidden neurons

The LSTM algorithm has high requirements for the number of hidden neurons [28]. The number of hidden neurons is one of the most important factors that affect the accuracy of the LSTM model. If only few neurons were used in the hidden layer, the model could not be fully trained; while too many hidden neurons would cause a time-consuming training, which may lead the model to take a phenomenon of non-convergence or over-fitting.Finding the appropriate number of neurons had an important impact on the quality of model training. We took a total of 10 kinds of neural network numbers (1, 10, 20, 30, 50, 100, 150, 200, 500, 800) to predict the future reservoir capacity by inputting the historical storage capacity and rainfall of the Huanggang Reservoir. It found that the RMSE value of the 1, 10, 20 items varies from 1 to 7. And the RMSE value of 30 and 50 items ranges from 0.975 to 0.27 and the results have great fluctuations (in the box plot would have a negative effect). Thus, the result of 100-800 items are only considered as shown in Figure5.Figure 5
Hidden Neuron box plot.With the number of neurons increase, the RMSE value of the model becomes smaller and gradually stabilizes, and the number of abnormal points decreases. When the number of neurons exceeds 500 to 800, the increased neurons number has a small effect on the model and the number of outliers becomes more, so we finally use 500 hidden neurons.


## 3.2.2. The influence of choosing a good optimizer

The optimizer plays a vital role in improving the speed of model training and accelerating model convergence to find the optimal solution [29–31]. Increasing of complex neural networks directly lead to huge model parameters. A good optimizer can speed up the update and calculate the network parameters output of the model so that the model can approximate and find the optimal solution. We randomly selected 7 kinds of functions (“adam”, “adagrad”, “adamax”, “SGD”, “Nadam”, “Rmsprop”, “Adadelta” 7 kinds of optimizers), run 7 times under random conditions, and recorded the RMSE generated in each operation (Figure 6).Figure 6
Optimizers’ RMSE value.The running results of 7 kinds of optimizers shows that the “SGD” optimizer was the worst since there are many failures to converge. Although the results of the “Adadelta” and “Nadam” optimizers have converged, the RMSE was too large. The result greater than 1 means the effect is very poor. The effect of the remaining three optimizers is well, and the value of RMSE is less than 1. The calculation result of “Adam” is more stable and works better by comparing with the other two optimizers, so the suitable optimizer is “Adam”.According to the description of hyperparameters in sections 3.2.1 and 3.2.2, and combined with the experimental objects (Huanggang Reservoir and Hejiapo Reservoir), we set appropriate hyperparameters to train each model to convergence. The final hyperparameter selections are shown in Table1.Table 1
Hyperparameters of LSTM.
Experimental objectsHuanggang ReservoirHejiapo ReservoirSingle-factorTwo-factorSingle-factorTwo-factorNumber of hidden layers2222Number of neurons in hidden layer 1500500100100Number of neurons in hidden layer 25005002020Activation function of hidden layer 1tanhtanhtanhtanhActivation function of hidden layer 2relurelulinearlinearOptimizeradamadamadamadamTraining times20060500240


## 3.2.3. Introduction of attention mechanism

In the LSTM model, the same vector is assigned to each input unit during the decoder decoding process [31]. In the encoder process, the output is the same fixed-length semantics. In the long-short-term prediction process of the model, these factors directly limit the model’s accuracy, so we considered adding the attention mechanism into LSTM. The attention mechanism comes from the human learning process, in which a certain part of our learning always appears to be the most important content. To speed up learning efficiency and improve accuracy, we focus our attention on that part of the information and suppress other irrelevant information in the re-learning process. The weights, which are assigned by the attention mechanism to the input sequence, can determine the most relevant aspects that affect the prediction data, thus improving prediction accuracy. The attention mechanism can effectively improve the situation in which the LSTM loses information because of long sequences, and simultaneously replace the original method of randomly assigning weights with that of assigning probabilities [32, 33]. Firstly, the attention machine value obtains a proportion by calculating the similarity between each output and input. Then, the mechanism accumulates all the inputs based on these factors’ proportion. By comparing the LSTM model with the attention and without the attention, we run 7 times under the same initial conditions and recorded the RMSE (Figure 7). We find that input of the attention mechanism in LSTM has a significant improvement than the LSTM without attention.Figure 7
Comparison plot of RMSE values of LSTM models with and without attention.


## 4. Result

### 4.1. Model evaluation criteria

For judging the prediction effect of the model, we used the most common indicator of hydrological models: the Nash-Sutcliffe Efficiency (NSE) [34] and the Mean Relative Error (MRE) [35]. As an important indicator of hydrological model evaluation, NSE is used to verify the quality of the hydrological model simulation. The value range is from negative infinity to 1. The closer of value is to 1, the better quality and reliability of the hydrological model.
(4)NSE=1−∑i=1NQsim,i−Qobs,i2∑i=1NQobs,i−Q¯obs2In the formula,Qsim,i is the true value of the reservoir capacity, Qobs,i is the predicted value of the reservoir capacity, and Q¯obs is the total average of the true value of the reservoir capacity.MRE is used to measure the average relative error between the true value and the predicted value, and then to judge the pros and cons of the model fitting.(5)MRE=1n∑i=1nxi−xviIn the formula, n represents the total number of validation set data, andxi represents the reservoir monitoring capacity of a reservoir on the i day. xvi represents the reservoir capacity predicted by the model on the i-th day, and i represents 1, 2, 3, ..., n.


### 4.2. Model accuracy comparison

Previously, we have optimized the model parameters by selecting the model structure with excellent effects. And considering the relevant factors in the reservoir capacity change, we have established a Developed LSTM model. We would verify whether the developed LSTM model had an improvement compared with the traditional LSTM model in predicting the change of reservoir capacity. We selected the Huanggang Reservoir data from 2013 to 2018, and used 70% of the data as the model training set and the rest 30% for model validation. LSTM model considers inputting the single-factor (historical reservoir capacity), while the developed LSTM model considers inputting the two-factors (historical reservoir capacity and rainfall) and introduction of attention mechanism. We used the model to predict the Huanggang Reservoir’s capacity change in 7days, 14days, and 30days, respectively. The prediction of reservoir capacity and the relative error of both models are as shown in Figure8. (The relative error described in this article refers to the value obtained by multiplying the ratio of the absolute error caused by the measurement to the true value of the measurand multiplied by 100%.)Figure 8
The prediction of reservoir capacity and the relative error of both models.Va represents the actual reservoir capacity, V7,V14 and V30 represent the predicted reservoir capacity for 7, 14, and 30 days.
(a)
Single-factor input LSTM model(b)
Two-factor input LSTM model(c)
Single-factor input LSTM relative error curve(d)
Two-factor input LSTM relative error curveThe result of Figure8 shows that developed LSTM model with inputting the two-factor (historical reservoir capacity and rainfall) fits measurement data higher than the LSTM model with inputting single-factor (historical reservoir capacity). The two-factor input prediction model has few deviations between the predicted reservoir capacity and the real reservoir capacity only at a few points while at other points the prediction are well. NSE is an important indicator for hydrological model evaluation. The value is closer to 1 means that a model has a higher credibility. MRE is an indicator to judge the degree of agreement between the predicted value and the true value. We can see the NSE and MRE values of developed LSTM model are better than that of the LSTM model for predictions of 7-day, 14-day, and 30-day. In addition, as the increase of prediction period (from 7, 14 to 30 days), the increase of error is obviously slow for the developed LSTM model than LSTM model (Figure 9). We also test the performances of these models during the typhoon season, which shows the similar results (Figure 10).Figure 9
Comparison plot of accuracy between LSTM model and developed LSTM model.
(a)(b)Figure 10
Comparison plot of accuracy of the LSTM model and the developed LSTM model in typhoon season.


## 4.1. Model evaluation criteria

For judging the prediction effect of the model, we used the most common indicator of hydrological models: the Nash-Sutcliffe Efficiency (NSE) [34] and the Mean Relative Error (MRE) [35]. As an important indicator of hydrological model evaluation, NSE is used to verify the quality of the hydrological model simulation. The value range is from negative infinity to 1. The closer of value is to 1, the better quality and reliability of the hydrological model.
(4)NSE=1−∑i=1NQsim,i−Qobs,i2∑i=1NQobs,i−Q¯obs2In the formula,Qsim,i is the true value of the reservoir capacity, Qobs,i is the predicted value of the reservoir capacity, and Q¯obs is the total average of the true value of the reservoir capacity.MRE is used to measure the average relative error between the true value and the predicted value, and then to judge the pros and cons of the model fitting.(5)MRE=1n∑i=1nxi−xviIn the formula, n represents the total number of validation set data, andxi represents the reservoir monitoring capacity of a reservoir on the i day. xvi represents the reservoir capacity predicted by the model on the i-th day, and i represents 1, 2, 3, ..., n.


## 4.2. Model accuracy comparison

Previously, we have optimized the model parameters by selecting the model structure with excellent effects. And considering the relevant factors in the reservoir capacity change, we have established a Developed LSTM model. We would verify whether the developed LSTM model had an improvement compared with the traditional LSTM model in predicting the change of reservoir capacity. We selected the Huanggang Reservoir data from 2013 to 2018, and used 70% of the data as the model training set and the rest 30% for model validation. LSTM model considers inputting the single-factor (historical reservoir capacity), while the developed LSTM model considers inputting the two-factors (historical reservoir capacity and rainfall) and introduction of attention mechanism. We used the model to predict the Huanggang Reservoir’s capacity change in 7days, 14days, and 30days, respectively. The prediction of reservoir capacity and the relative error of both models are as shown in Figure8. (The relative error described in this article refers to the value obtained by multiplying the ratio of the absolute error caused by the measurement to the true value of the measurand multiplied by 100%.)Figure 8
The prediction of reservoir capacity and the relative error of both models.Va represents the actual reservoir capacity, V7,V14 and V30 represent the predicted reservoir capacity for 7, 14, and 30 days.
(a)
Single-factor input LSTM model(b)
Two-factor input LSTM model(c)
Single-factor input LSTM relative error curve(d)
Two-factor input LSTM relative error curveThe result of Figure8 shows that developed LSTM model with inputting the two-factor (historical reservoir capacity and rainfall) fits measurement data higher than the LSTM model with inputting single-factor (historical reservoir capacity). The two-factor input prediction model has few deviations between the predicted reservoir capacity and the real reservoir capacity only at a few points while at other points the prediction are well. NSE is an important indicator for hydrological model evaluation. The value is closer to 1 means that a model has a higher credibility. MRE is an indicator to judge the degree of agreement between the predicted value and the true value. We can see the NSE and MRE values of developed LSTM model are better than that of the LSTM model for predictions of 7-day, 14-day, and 30-day. In addition, as the increase of prediction period (from 7, 14 to 30 days), the increase of error is obviously slow for the developed LSTM model than LSTM model (Figure 9). We also test the performances of these models during the typhoon season, which shows the similar results (Figure 10).Figure 9
Comparison plot of accuracy between LSTM model and developed LSTM model.
(a)(b)Figure 10
Comparison plot of accuracy of the LSTM model and the developed LSTM model in typhoon season.


## 5. Discussion

In this section, two aspects of the developed LSTM model will be discussed: the inputting influence factors of LSTM model, and the further applicability of model.


### 5.1. Influence factors of LSTM model

For multi-factors input, we considered that the influencing factors of reservoir capacity are often diverse rather than one-single. Thus, physically it is impossible to make good predictions for considering only one factor (reservoir capacity) input. Considering the impact of multi-correlation factors, the model can make good adaptability for some predictions of mutation. The input of multi factors is not blind, and the influencing factors need to have sufficient correlation. If the input influencing factors are irrelevant or negatively related to the reservoir capacity prediction, it will directly lead to the reduction of the accuracy of the model. Specially, one factor of historical reservoir capacity is used in the LSTM model. It considers this single factor containing the previous information of rainfall and drainage which mainly affect the balance of reservoir capacity. In two factors inputting consideration, besides the historical reservoir capacity, we regards the time-series rainfall, the most important factor to improve the reservoir capacity, as the one independent factor.


### 5.2. Application of model in another case

In order to further illustrate the applicability of the developed LSTM model, we obtained the reservoir data of Hejiapo in Longyan City. The data contained the Hejiapo historical reservoir capacity and the historical rainfall from 2015 to 2019 (Figure11). By using the data of 2015-2017 as the training set and the data of 2018-2019 as the validation set, we used the Developed LSTM model and the LSTM model for the prediction of the reservoir capacity of the Hejiapo (7 days, 14 days, and 30 days) (Figure 12).Figure 11
Location of Hejiapo Reservoir.Figure 12
The prediction of reservoir capacity.Va represents the actual reservoir capacity of the reservoir, V7,V14,andV30 represent the predicted reservoir capacity of the 7, 14 and 30 days, respectively, and Rainfall is the rainfall in the reservoir area.The Figure13 indicates that the developed two-factor LSTM model has a lower MRE and a higher NSE values than the LSTM model for 7-day, 14-day, and 30-day.Figure 13
Two models’ accuracy comparison.
(a)(b)Comparing the MRE of the single-factor LSTM model and the two-factor developed LSTM model during the typhoon season, the Figure14 shows that the developed LSTM model makes a more accurate predictions in extreme weather.Figure 14
Two models’ accuracy comparison in typhoon season.


## 5.1. Influence factors of LSTM model

For multi-factors input, we considered that the influencing factors of reservoir capacity are often diverse rather than one-single. Thus, physically it is impossible to make good predictions for considering only one factor (reservoir capacity) input. Considering the impact of multi-correlation factors, the model can make good adaptability for some predictions of mutation. The input of multi factors is not blind, and the influencing factors need to have sufficient correlation. If the input influencing factors are irrelevant or negatively related to the reservoir capacity prediction, it will directly lead to the reduction of the accuracy of the model. Specially, one factor of historical reservoir capacity is used in the LSTM model. It considers this single factor containing the previous information of rainfall and drainage which mainly affect the balance of reservoir capacity. In two factors inputting consideration, besides the historical reservoir capacity, we regards the time-series rainfall, the most important factor to improve the reservoir capacity, as the one independent factor.


## 5.2. Application of model in another case

In order to further illustrate the applicability of the developed LSTM model, we obtained the reservoir data of Hejiapo in Longyan City. The data contained the Hejiapo historical reservoir capacity and the historical rainfall from 2015 to 2019 (Figure11). By using the data of 2015-2017 as the training set and the data of 2018-2019 as the validation set, we used the Developed LSTM model and the LSTM model for the prediction of the reservoir capacity of the Hejiapo (7 days, 14 days, and 30 days) (Figure 12).Figure 11
Location of Hejiapo Reservoir.Figure 12
The prediction of reservoir capacity.Va represents the actual reservoir capacity of the reservoir, V7,V14,andV30 represent the predicted reservoir capacity of the 7, 14 and 30 days, respectively, and Rainfall is the rainfall in the reservoir area.The Figure13 indicates that the developed two-factor LSTM model has a lower MRE and a higher NSE values than the LSTM model for 7-day, 14-day, and 30-day.Figure 13
Two models’ accuracy comparison.
(a)(b)Comparing the MRE of the single-factor LSTM model and the two-factor developed LSTM model during the typhoon season, the Figure14 shows that the developed LSTM model makes a more accurate predictions in extreme weather.Figure 14
Two models’ accuracy comparison in typhoon season.


## 6. Conclusions

(1)
This study verified the better performance of the developed LSTM model than the LSTM model for the Huanggang Reservoir capacity prediction for 7-day, 14-day, and 30-day. In addition, the Developed LSTM model was well applied to the Hejiapo Reservoir capacity estimation, which also shows an accurate prediction(2)
The prediction accuracy of the LSTM neural network model based on two-factor input has a great relationship with the parameter settings. Different parameter settings of the same model could produce different results. The input of multi factors is not blind, and the influencing factors need to have sufficient correlation. If the input influencing factors are irrelevant or negatively related to the reservoir capacity prediction, it will directly lead to the reduction of the accuracy of the model(3)
Due to the limitation of data collected in this study, the developed LSTM only considers two factors. There could be more factors related to the change of reservoir capacity. The question that with the increasing of number of factors, whether the prediction accuracy of the model continually improves will be investigated in our future work



---

*Source: 2891029-2022-07-23.xml*