# Short-Term Forecasting of Power Production in a Large-Scale Photovoltaic Plant Based on LSTM
## 2 Theoretical Background
### Recurrent Neural Networks and Long Short-Term Memory  
These transforms are inverted on forecasts to return them into their original scale before calculating and error score. An RNN is defined as an ANN with arbitrary connections between neurons, usually fully connected between adjacent layers. It models temporal dependencies present in time series data through the use of feedback connections and aims to'remember' the values at previous time steps. As shown in Figure 2, it gives a simple RNN with one input unit, one output unit, and one recurrent hidden unit unfolded into a full network. \(o_{t}\) is the output at time step \(t\), \(x_{t}\) is the input at time \(t\), \(s_{t}\) is the state at time step \(t\), and it is the'memory' of RNN. \(W\), \(U\), and \(V\) are parameters in different network layers, and an RNN can share the same parameters (\(W\), \(U\), and \(V\) above) across all steps [28]. The process can be expressed as  
\[s_{j}(t)=f(\sum\
olimits_{i}^{t}x_{i}(t)v_{ji}+\sum\
olimits_{h}^{m}s_{h}(t- 1)u_{ji}+b_{j}) \tag{1}\]  
where, \(f\) (\(\cdot\)) is map function which is usually nonlinear such as tanh or rectified linear unit (ReLU). \(\sum\
olimits_{i}^{t}x_{i}(t)v_{ji}\) is the input layer at the time of \(t\). \(\sum\
olimits_{h}^{m}s_{h}(t-1)u_{ji}+b_{j}\) is the input of hidden layer of time \((t-1)\). \(b_{j}\) is the bias. The LSTM algorithm belongs to the family of recurrent neural networks (RNN). RNNs commonly face a problem called the \"exploding/vanishing gradients problem\". LSTM is designed to overcome above question by creating a special type of structures called memory cells and gate units. So it is good for remembering information for long time. To understand the functioning of LSTM memory units, Figure 3 depicts a single localized LSTM cell in the first layer of a network at a time step \(t\). Module has the output values input (\(\mathbf{i_{t}}\)), update (\(\mathbf{g_{t}}\)), output (\(\mathbf{o_{t}}\)) and forget (\(\mathbf{f_{t}}\)) of four types of gates as shown in Figure 3.  
Figure 3: A LSTM (long short-term memory) unit in the first layer for time step (t). Figure 2: General overview of recurrent neural networks. First of all, the decision which information will be thrown away from the cell state is made by forget gate. Equations (2)-(6) are governing equations, which we explain below. Please note that \(W_{f}\) and \(U_{f}\) are weight matrices of input and output data from the previous LSTM cells. \(b_{n}\) represents bias vector parameters which need to be learned during training. \(f_{t}\) is the activation vector of forget gate. The sigmoid activation (linearly approximated standard sigmoid) has a range of 0 (forget all) to 1 (remember all) and is governed by Equation (2) [28]. \[f_{t}=sigm(W_{f}x_{t}+U_{f}h_{t-1}+b_{f}) \tag{2}\]  
Next, once an input \(x_{t}\) enters the LSTM cell, it is separately passed through an activation function, \(g_{t}\) and input gate \(i_{t}\). The cell can decide which values to write through the input gate and create a vector of new cell values through the update gate. Equations (3) and (4) represent their working, note \(g_{t}\) and \(i_{t}\) separately use hyperbolic tangent (tanh) and hard sigmoid (tanh) activation [28]. \[g_{t} =tanh(W_{f}x_{t}+U_{f}h_{t-1}+b_{f}) \tag{3}\] \[i_{t} =sigm(W_{i}x_{t}+U_{i}h_{t-1}+b_{i}) \tag{4}\]  
Cell memory (see Figure 2) updates itself recursively by interaction of its old value (\(t-1\)) with forget and write gates' values. After these, combine these two parts to create an update to the states, \"\(\odot\)\" represents scalar multiplication. It follows (5) [28]  
\[C_{t}=f_{t}\odot C_{t-1}+i_{t}\odot C_{t} \tag{5}\]  
Lastly, the output gate decides which part of the cell states will be outputted by Equation (6). Ultimately, the output \(h_{t}\) of LSTM unit at time \(t\) can be obtained by applying Rectified Linear Unit (ReLU) to cell states through and multiplying it by output gate values as Equation (7) shows. Where (\(W_{0}\), \(b_{0}\)) are the weights and bias of output gate.