## 3 Popular ANN Architectures Employed for Time Series Forecasting Purposes
### Feedforward Neural Networks  
#### 3.1.1 Basic Model  
Also known in the state of the art by the name of multilayer perceptron (MLP), a feedforward neural network (FFNN), is an ANN architecture where information flows in one direction only, from one layer to the next. Typically, an FFNN is composed of an input layer, one or more hidden layers, and an output layer. In each hidden layer and the output layer, there are neurons (nodes) that are usually connected to all the neurons in the next layer. Figure 1 shows a single FFNN hidden layer with inputs \(x_{1},x_{2},\ldots,x_{d}\) and output \(\hat{y}\).  
In the structure shown in Figure 1, information is transmitted from left to right; the inputs \(x_{1}\), \(x_{2}\), \(\ldots\), \(x_{d}\) are transformed into the output \(\hat{y}\). In the context of time series modeling, focusing on the AR model, FFNN inputs \(x_{1}\), \(x_{2}\), \(\ldots\), \(x_{d}\) correspond to time series values \(y_{t}\), \(y_{t-1}\), \(y_{t-2}\), \(\ldots\), \(y_{t-d+1}\), while FFNN output \(\hat{y}\) typically corresponds to a prediction value \(\hat{y}_{t+1}\), which attempts to approximate future time series value \(y_{t+1}\) (see Section 2.2).  
Each arrow in the FFNN structure in Figure 1 represents a weight \(w\) for the input value \(x\) that enters on the left of the arrow, and exits on the right side with the value \(w\cdot x\). The input layer is formed by the independent variables \(x_{1}\), \(x_{2}\), \(\ldots\), \(x_{d}\) and a constant value  
Figure 1: Single hidden layer feedforward neural network.  
known as the _intercept_. For each neuron, all its inputs are summed, and then this sum \(\Sigma\) is transformed by applying a nonlinear activation function \(\sigma\). One of the most-used activation functions in ANNs is the logistic (sigmoid) function \(\sigma(s)=\dfrac{1}{1+e^{-s}}\). The hyperbolic tangent is another frequently used activation function \(\sigma(s)=\dfrac{e^{2s}-1}{e^{2s}+1}\). The functionality of each neuron in the hidden layer \(h\) (\(h=1,2,\ldots,q\)) is described by  
\[\sigma\left(\sum_{i=0}^{d}w_{i,h}\cdot x_{i}\right), \tag{3}\]  
where each \(w_{i,h}\) represents the weight that corresponds to the arrow that connects the input node \(x_{i}\) with the neuron \(h\). In Figure 1, the output layer contains only one neuron; this is the number of output units needed if we are interested in predicting a single scalar time series value, but we can employ more output units if we need multiple prediction horizons (scalar or vector). Neurons in an output layer have identical functionality to those in hidden layers, although output neurons sometimes employ the identity function \(I(z)=z\) as an activation function (especially for time series forecasting tasks). It is also possible, however, to use as an activation function for output units the same sigmoidal activation functions employed by hidden units (i.e., logistic or hyperbolic tangent). It is even possible to employ different activation functions for units in the same layer. The output layer functionality for our FFNN, depicted in Figure 1, assuming an identity activation function, is described by  
\[\hat{y}=w_{0,1}+\sum_{h=1}^{q}w_{h,1}\cdot\sigma\left(\sum_{i=0}^{d}w_{i,h} \cdot x_{i}\right), \tag{4}\]