## 3 Popular ANN Architectures Employed for Time Series Forecasting Purposes
### Feedforward Neural Networks
#### 3.1.2 FFNN Training  
In summary, the ANN has a set of parameters that must be set to determine how the input data are processed and the output generated: weights and biases. The weights are related to the control of the connection between two neurons. The weight value determines the magnitude and direction of the impact of a given input on the output. Biases can be defined as the constant that is added to the product of features and weights. It helps models change the activation function to the positive or negative side [36]. FFNNs are trained, or \"taught\", with the help of supervised machine learning algorithms. _Backpropagation_ (BP) is probably the most popular machine learning algorithm employed to train FFNN models. Next, we describe briefly how BP works. The idea is to adjust all weights and biases in the FFNN model so that, in principle, they minimize some fitting criterion \(E\); for example, the mean squared error:  
\[E=\dfrac{1}{n}\sum_{p=1}^{n}\left(y^{(p)}-\hat{y}^{(p)}\right)^{2}, \tag{5}\]  
where \(n\) is the number of example input patterns available for training our model, \(y^{(p)}\) is the target (desired value) for the \(p\)th example input pattern, and \(\hat{y}^{(p)}\) is the FFNN output also for the \(p\)th example input pattern, \(p=1,2,\ldots,n\). FFNN weights are adjusted according to the backpropagation rule: \(\Delta w=-\eta\dfrac{\partial E(\tilde{W})}{\partial w}\), where meta-parameter \(\eta\) is a small positive number called \"learning rate\", \(\tilde{W}\) is a vector containing all FFNN weights, and \(w\) is a single FFNN weight (i.e., \(w\) can be any single component of \(\tilde{W}\)). Basically, the BP algorithm consists of the following steps:  
1. Initialize \(\tilde{W}\) randomly;
2. Repeat (a) and (b) until \(E(\tilde{W})\) is below a given threshold \(T\) or a pre-established maximum number of iterations \(M\) has been reached\[\text{(a)}\quad\quad\vec{W}_{old}\leftarrow\vec{W};\] \[\text{(b)}\quad\quad\text{Update all network weights: }w\gets w+\left[-\eta\frac{\partial E(\vec{W}_{old})}{\partial w}\right];\]
3. Return \(\vec{W}\). From this algorithm outline, we see that BP starts from a random point \(\vec{W}_{0}\) in search space \(W\) and looks for the global minimum of the error surface \(E(\vec{W})\), say \(\vec{W}^{*}\), by taking small steps, each towards a direction opposite to the derivative (gradient) of the multivariate error function \(E\), evaluated at the current location in the search space \(W\) where BP has advanced so far. If \(E(\vec{W})\) is smooth enough, BP will descend monotonically when moving from one step to the next; this is why BP is said to be a stochastic gradient descent procedure. In summary, the goal of BP is to iteratively adjust \(\vec{W}\), applying the gradient descent technique, so that the output of the FFNN is close enough to the target values in the training data [29]. For convex error surfaces, BP would do a nice job; unfortunately, \(E(\vec{W})\) often contains local minima, and in such situations, BP can easily become stuck into one of those minima. A heuristic approach to facing this issue is to run BP several times, keeping all FFNN settings fixed (e.g., set of training data, number of inputs \(d\), number of hidden neurons \(q\), and BP meta-parameters). Another important issue we face when training FFNNs with BP is that of _overfitting_. This condition occurs when a model adapts too well to the local stochastic structure of (noisy) training examples but produces poor predictions for inputs not in the training examples. If we strictly aim for \(E(\vec{W})\) global minimum, then we focus only on _interpolating exactly_ all training data examples.