## 4 Brief Literature Survey on ANNs Applied to Time Series Modeling
### Combining Feedforward Neural Networks and Particle Swarm Optimization for Time Series Forecasting
#### 4.1.1 Particle Swarm Optimization for Artificial Neural Networks  
In Algorithm 1, the basic PSO algorithm proposed by [PERSON] and [PERSON] [101] is presented. ```
for each particle \(i=1,2,...,N\) in the swarm do  initialize particle's position: \(\vec{x}_{i}\leftarrow\) uniform random vector in \(\mathbb{R}^{\mathbb{D}}\)  initialize particle's best-known position: \(\vec{P}_{Best,i}\leftarrow\vec{x}_{i}\) if\(f(\vec{P}_{Best,i})<f(\vec{G}_{Best})\)then  update swarm's best-known position: \(\vec{G}_{Best}\leftarrow\vec{P}_{Best,i}\) endif  initialize particle's velocity: \(\vec{v}_{i}\leftarrow\) uniform random vector in \(\mathbb{R}^{\mathbb{D}}\) endfor repeat for each particle \(i=1,2,...,N\) in the swarm do for each dimension \(d=1,2,...,D\)do  pick random numbers \(r_{p},r^{\delta}\sim U(0,1)\)  update particle's velocity: \(v_{i},d\leftarrow\omega v_{i,d}+c_{1}r_{p}(\vec{P}_{Best,i,d}-x_{i,d})+c_{2}r_{g }(\vec{G}_{Best,d}-x_{i,d})\) endfor  update particle's position: \(\vec{x}_{i}\leftarrow\vec{x}_{i}+\vec{v}_{i}\) if\(f(\vec{x}_{i})<f(\vec{P}_{Best,i})\)then  update particle's best-known position: \(\vec{P}_{Best,i}\leftarrow\vec{x}_{i}\) if\(f(\vec{P}_{Best,i})<f(\vec{G}_{Best})\)then  update swarm's best-known position: \(\vec{G}_{Best}\leftarrow\vec{P}_{Best,i}\) endif endfor until a termination criterion is met  Now, \(\vec{G}_{Best}\) holds the best found solution
```  
**Algorithm 1** Basic particle swarm optimization (PSO) algorithm  
The basic PSO algorithm shown in Algorithm 1 can be applied to ANNs as a multilayer perceptron, where each particle's position \(\vec{x}_{i}\) represents the set of weights and biases of the ANN for the current iteration. Each particle moves in the weighting space trying to minimize the learning error during the training phase, and also maintains the historically best position \(\vec{p}_{i}\) in memory along its exploration path. When the particle changes position, it is analogous to updating the weights of the ANN controller to reduce the tracking error [108]. The termination criterion can be defined as the scope of a predefined MSE value condition [109]. Finally, the best position reached by the swarm \(\vec{g}\) can be expressed as the optimal solution for the ANN. Now let us take a look at some FFNN-PSO time series models proposed in the literature. [PERSON], [PERSON], and [PERSON] [81] assess the effectiveness of FFNN and Elman networks when trained with the PSO algorithm for the prediction of univariate seasonal time series. In this context, a PSO particle moves in the search space \(\{\tilde{W}\}\) defined by the weights of the ANN model to be trained, while the PSO cost function is the same cost function employed in backpropagation; thus, we could say that a PSO particle is structurally identical to an ANN. de M. [PERSON], [PERSON], [PERSON], and [PERSON] [92] take a slightly different approach: they propose a basic PSO optimizer in which each particle is a single hidden layer FFNN designed to produce one-step-ahead forecasts for univariate time series, plus some extra meta-parameters. The search space for their proposed hybrid system consists of the following discrete and continuous variables:  
* Relevant time lags for autoregressive inputs (the total number of relevant time lags defines the FFNN's input dimension \(d\));
* Number \(q\) of hidden units in the FFNN;
* Training algorithm employed: 1. Levenberg-Marquardt [110], 2. RPROP [111], 3. scaled conjugate gradient [112], or 4. one-step secant [113], all refinements of the basic BP algorithm;
* Variant of FFNN architecture employed: 1. An FFNN architecture identical to the one outlined in Section 3.1, with a linear output unit, 2. an FFNN with structural modifications proposed by [PERSON], [PERSON], [PERSON], and [PERSON] [114], or 3. the same FFNN architecture as in 1, but with a sigmoidal output unit;
* Initial FFNN weights and meta-parameter configuration.