## 2 Background Theory
### Predictive Methods
#### 2.2.2 Recurrent Neural Networks  
A different approach for time series forecasting is through Recurrent Neural Network (RNN). It represents a class of artificial neural networks characterized by the connection between the nodes along a temporal sequence. This allows for exhibiting temporal dynamic behavior. In detail, RNN's can use an \"internal memory state\" to process variable-length sequences of inputs in order to predict probabilistic future trends in the data [48]. The simplest RNN consists of just three layers: one input layer with \(d\) input neurons, one hidden layer with \(c\) hidden neurons, and one output layer with \(q\) output neurons (see Figure 2). At each time point \(t\), the network includes three matrices of weight, \(W_{ih}\), \(W_{hh}\), and \(W_{ho}\) that connect, the input to hidden layer, recurrent hidden layer to hidden layer itself, and hidden layer to output, respectively. The hidden layer captures information about what happened in the previous time step. It can be considered as the \"memory\" of the network. In order to determine how well the RNN predicts the output during the training phase, a Loss function (\(L_{t}\)) is computed that includes the differences between predicted and actual output (\(y_{t}\) and \(y^{\prime}_{t}\), respectively) at each iteration. At each time step, \(L_{t}\) is given by cross-entropy, defined as follows:  
\[L_{t}=-y_{t}\,\log(y^{\prime}_{t}) \tag{6}\]  
Figure 2: Schematic and simplified structure of an RNN. The final Loss is the sum of the loss at all steps. For instance, in case of \(T\) steps (and \(T\) layers, as in Figure 2), the final Loss is:  
\[L=\sum_{j=1}^{T}L_{j} \tag{7}\]  
The Loss function is minimized by finding the optimal weights for all the nodes forming out the RNN. For instance, the gradient descent algorithm can be applied for determining the optimal weights matrixes. According with that algorithm, the rule for updating the weights is:  
\[W=W-\alpha\frac{\partial L}{\partial W} \tag{8}\]  
Here, \(W\) indicates, generically, each one of the weights matrixes defined above (\(W_{ih}\), \(W_{hh}\) and \(W_{ho}\)). The parameter \(\alpha\) is called \"learning rate\". It scales the gradient and thus controls the step size. When the gradient of the Loss function is calculated with respect to the weight matrixes, generally it is possible to incur in the well-known problem of the vanishing gradient. In fact, when the RNN is formed by many neural layers, there is the risk that the gradient tends to become smaller and smaller at each derivative step. Consequently, since the gradient tends to vanish over time, the network cannot learn properly about long-term time information kept in memory. This is a typical problem in all the types of deep neural networks formed by many hidden layers, and not exclusively in RNNs. In order to face that problem, a variant of RNN called \"Long-Short Term Memory\" (LSTM) algorithm can be used [49; 50; 51]. This allows for the network to include both short-term and long-term dependencies at a given time from many time steps before the current time and can solve the vanishing gradient problem effectively. A typical LSTM cell consists of three special \"gates\" called, respectively, the input gate, forget gate, and output gate. These allow for deciding what information must be included or excluded from the network memory and used for calculating the output. In other words, the effectiveness of the network is guaranteed by selecting the length of the sequence of historical data to be used as \"memory information\". The LSTM network is designed to keep information in the memory only if required. The input gate is responsible for selecting the part of information that must be stored in memory and used for calculating the output. Instead, the forget gate is responsible for deciding what information is not relevant for calculating the output and should be excluded from the output computation. Finally, the output gate is set for selecting what information must be shown at each time step.