# A Review on Neural Network Based Models for Short Term Solar Irradiance Forecasting
## 3 The Architecture of Deep Learning Algorithms
### Long Short-Term Memory (LSTM)  
The LSTM network is considered to be an extended and improved structure of RNN that can effectively solve its time-series forecasting issues and inability to deal with long-term data dependencies because of vanishing gradient and gradient explosion issues [105]. Therefore, LSTMs are frequently utilized in power systems for time-series forecasting, including load forecasting, demand response, and renewable power generation forecasting [106]. The LTSM network concept was introduced by [43] to resolve the limitation of a vanishing gradient. This approach can be seen in Figure 6, where the LTSM network consists of inputs, outputs, memory cells, and forget gates that work together to regulate information flow. The forget gate separates information into two categories: data that should be deleted and data that should be retained. The input gate is responsible for updating the cells, whereas the output gate determines their next concealed state. The gates use the sigmoid function as their activation function, and they will generate a value between 0 and 1 to selectively enable information to pass through them. The gates can be used to indicate how much data from the current input is allowed to pass through the structure. As previously stated, the whole structure serves as a gate; when it is open, all information is permitted to pass through it (sigmoid output is one). On the other hand, when the  
Figure 5: The architecture of RNN [94].  
gate is closed (having a sigmoid output of zero), no information can pass through it. The mathematical description and information distribution are described in Equation (13):  
* The forget gate, \(f_{t}\), helps the LSTM to determine which information should be eliminated from the cell state by applying a sigmoid function to the output of the prior \(h_{t-1}\) and input data \(x_{t}\): \[f_{t}=(w_{f}.h_{t-1},w_{f}.x_{t}+b_{f})\] (13) where (.) signifies the sigmoid activation function, \(w_{t}\) signifies the weight matrix, \(h_{t-1}\) signifies the prior state, \(x_{t}\) signifies the memory cell of the input vector at time \(t\), and \(b_{f}\) signifies the bias vector.  
The input gate, \(i_{t}\), determines which data are stored in the new candidate cell state \(\widetilde{C}_{t}\). The hyperbolic tangent function is symbolized as tan\(h\) () in Equation (14).  
\[\widetilde{C}_{t}=\tan h\ (w_{c}.h_{t-1},w_{c}.x_{t}+b_{c}) \tag{14}\]  
\[I_{t}=(w_{i}.h_{t-1},w_{i}.x_{t}+b_{i}) \tag{15}\]  
The input gate applies the sigmoid function, (\(\sigma\)), to determine which values to write, and it updates the gate using the tan\(h\) activation function to generate new cell values.  
The new candidate's cell state \(\widetilde{C}_{t}\) and the prior cell state \(C_{t-1}\) are incorporated to update the latest cell state, \(C_{t}\), which is expressed as shown in Equation (16):  
\[C_{t}=I_{t}*\widetilde{C}_{t}+f_{t}*C_{t-1} \tag{16}\]  
Finally, the output gate regulates the output of a cell and merges it with a cell state that is activated via the tan\(h\) function to determine the final output, \(h_{t}\), which is expressed in the Equations (17) and (18):  
\[O_{t}=\sigma\ (w_{o}.h_{t-1},w_{o}.x_{t}+b_{o}) \tag{17}\]  
\[h_{t}=\tan h\ (C_{t}).\ O_{t} \tag{18}\]  
Figure 6: The architecture of the LSTM network [107].