# Load Forecasting with Machine Learning and Deep Learning Methods
## 3 Methodology
### Modeling
#### 3.2.5 Long Short-Term Memory  
Recurrent neural networks are a derivative of ANNs, so the learning process is carried out in a similar way to that presented in the previous section. Thus, the minimization process is governed, as in the previous case, by Equation (8). The RNN and the ANN are usually used together, where the first layers of the problem are part of the RNN. These layers extract and retain all possible information from the data, and the subsequent layers form the ANN. Because of the internal complexity of the RNNs arising from the need to train a larger number of parameters, they tend to cause problems of overfitting [67,68]. The main characteristic of RNNs is the retention of important information from the dataset [69]. The best known and commonly used model is LSTM. This model consists of a series of internal mechanisms to regulate the flow of relevant information in the short and long term to make accurate forecasts. This flow of information is transferred in sequence by the cell state, where the retention of relevant information and the elimination of irrelevant information is controlled by gates [70,71,72]. In each neuron, the flow of the LSTM is controlled by 3 types of gates: forget gates, input gates, and output gates. If a node has more than one input, its function corresponds to the sum of the resulting values of each of its connections. This process propagates from the input neurons to the output neurons through the interconnections [70]. The forget gate, given in (11) by \(f^{[t]}\), decides which information that has been previously retained should be kept or deleted when new data are received. \[f^{[t]}=g\Big{(}w_{fg}\Big{[}h^{[t-1]},x^{[t]}\Big{]}+b_{fg}\Big{)} \tag{11}\]where \(w_{fg}\Big{[}h^{[t-1]},x^{[t]}\Big{]}\) is the kernel, a function of the information previously kept, \(h^{[t-1]}\), and the new data, \(x^{[t]}\), and \(b_{fg}\) is the bias of the forget gate. \(g(\cdot)\) is the activation function. The sigmoid activation function is commonly used because it compresses the information in the range \([0,1]\). This allows the gate to recognize when the information is important, i.e., whether the value is close to or equal to \(1\), or when it is irrelevant, i.e., whether the value is close to or equal to \(0\). The input gate decides what new information should be stored in the cell state through the simultaneous operation of \(2\) steps. Firstly, (12) determines what information should be updated, \(i^{[t]}\). Then, through (13), the sequence, \(\widetilde{c}^{[t]}\), is regulated. \[i^{[t]}=g\Big{(}w_{ig}\Big{[}h^{[t-1]},x^{[t]}\Big{]}+b_{ig}\Big{)} \tag{12}\]  
where \(w_{ig}\Big{[}h^{[t-1]},x^{[t]}\Big{]}\) and \(b_{ig}\) are the weights of the phase of information retention of the input gate, kernel, and bias, respectively. As in the forget gate, the sigmoid activation function is frequently employed to retain important information. \[\widetilde{c}^{[t]}=g\Big{(}w_{cs}\Big{[}h^{[t-1]},x^{[t]}\Big{]}+b_{cs}\Big{)} \tag{13}\]  
where \(w_{cs}\Big{[}h^{[t-1]},x^{[t]}\Big{]}\) and \(b_{cs}\) are the weights of the phase of sequence regulation of the input gate, kernel, and bias, respectively. In this case, the tanh activation function is commonly utilized because it compresses the information in the range \([-1,1]\). This adds a weight to the values obtained in the previous phase in the cell state. By combining the results of the previously described gates and the previous information kept in the cell state, \(c^{[t-1]}\), the value of the cell state, \(c^{[t]}\) in Equation (14), is updated. \[c^{[t]}=f^{[t]}c^{[t-1]}+i^{[t]}\widetilde{c}^{[t]} \tag{14}\]  
Finally, the output gate decides the output of the neuron. This is the combination of the information previously kept together with the new data and information resulting from the cell state (15).