## 3 Theoretical Background
### Deep Learning Models
#### 3.2.1 Feed-Forward Neural Networks  
FFNNs, also known as Multi-Layer Perceptrons (MLPs), are the most basic DL architectures. FFNNs are able to model complex relationships between input-output pairs by composing together many functions [25] and natively support multi-output regression problems. Moreover, as the universal function approximation theorem states [26], under mild conditions, they are capable of finding the function that relates input to output with a certain level of precision. FFNNs consist of interconnected layers composed of different numbers of neurons, in which the information is progressively broken down into features and flows in one direction. In Figure 3, we show a schematic diagram of a three-layered FFNN for illustration purposes. On it, the following are differentiated: the input layer, which is responsible for receiving the input vector and passing it to the next layer; the hidden layer, which performs all the computations; and finally, the output layer, which generates the predicted values. Each layer consists of many units, called neurons, that are fully connected with the immediately adjacent layers in one direction; note that we may add many hidden layers, which tends to increase the prediction performance when enough data are available [25]. The fundamental computational unit of a neural network is referred to as a neuron, or unit. As described in [27], a neuron is composed of three main elements: the connecting links, weighted by a numerical value known as weight, \(w_{ji}\), which indicates the relevance of such connection in the output value; the summing junction, which sums all the incoming input values; and the activation function, \(\varphi(\cdot)\), which limits the dynamic range of the output value, which depends itself on the chosen function. Some of the most applied activation functions are the linear function, the Sigmoid, the Hyperbolic tangent or the Rectified Linear Unit. In addition, there is a scalar value known as bias, \(b_{i}\). Moreover, each neuron has a single output and receives many inputs. Thus, the output is a non-linear combination of all the inputs, and the equation that computes the output value \(\xi_{i}^{j}\) of neuron \(i\) in layer \(j\) can be described as  
\[\xi_{i}^{j}=\varphi\left(\sum_{k=1}^{P}w_{k,i}\xi_{k}^{j-1}+b_{i}\right) \tag{3}\]  
where \(\xi_{k}^{j-1}\) are the input values to layer \(j\) neurons, i.e., the output of the \(P\) neurons of the previous layer \(j-1\), and \(\varphi\) represents the activation function chosen. The most popular method to train FFNNs is the Back-Propagation (BP) algorithm. The training proceeds in two phases [27]: the forward and the backward propagation. In the former, weights and biases are randomly initialized, and input values are passed through the layers performing operation 3 on each neuron until the output layer is reached. Then, the error between the predicted value and the actual value is computed. This error is propagated backwards towards the input layer, and the weights and bias values of each neuron are adjusted until the error converges to an acceptable level. This is referred to as the backward propagation and is an optimization problem in which a certain loss function has to be minimized by adjusting the weights and biases of the NN. As a loss function, we use the Mean Squared Error (MSE), which is widely used in regression problems and is described by the following equation:  
\[\text{MSE}=\frac{1}{NM}\sum_{i=1}^{N}\sum_{j=1}^{M}\left(y_{i}^{j}-y_{i}^{j} \right)^{2} \tag{4}\]  
where \(y_{i}^{j}\) represents the true values and \(y_{i}^{j}\) are the predictions of the network for trajectory \(i\) and time step \(j\). Detailed computations of the Back-Propagation algorithm are presented in [25; 27]. The optimization algorithm that we use is Adam [28]. It is a first-order method based on gradient, whose name is derived from the estimation of the adaptive momentum.