## 4 Brief Literature Survey on ANNs Applied to Time Series Modeling
### Combining Feedforward Neural Networks and Particle Swarm Optimization for Time Series Forecasting
#### 4.1.2 Particle Swarm Optimization Convergence  
To prevent the basic gradient descent method (applied in BP) from being caught at the local minimum, we can apply PSO to ANN to optimize the values of the weights' and biases' parameters. Although the PSO algorithm has been shown to perform well, researchers have not adequately explained how it works. In 2003, [PERSON] and [PERSON] [124] made a comparison of the BP and PSO algorithms, analyzing the computational requirements when used in ANN training. They concluded that when the PSO algorithm is used, the FFNN weights converge faster, outperforming the BP algorithm. [PERSON], [PERSON], [PERSON], and [PERSON] [125] used a BP-ANN based on the PSO algorithm (PSO-BP). They demonstrated that their proposed PSO-BP algorithm outperforms a BP trained based on the Levenberg-Marquardt (LM) algorithm for training ANNs. [PERSON], [PERSON], and [PERSON] [126] proposed to find not only the weights of an FFNN but also the optimal architecture of the network, applying the MD-PSO algorithm: a modified version of the PSO. Their approach was to find the optimal number of dimensions for the search space simultaneously searching for the optimal solution in that proposed search space. The MD-PSO algorithm chose the global optimal solution among the optimal solutions found for each dimension. Several works have proposed variants of PSO against BP to optimize ANN [127; 128; 129; 130]. However, like other evolutionary algorithms, PSO has some disadvantages, such as an imbalance between exploration and exploitation, sensitivity to parameters, and premature convergence [36]. These problems have been mostly solved by including new parameters, modifying the algorithm with additional operators, or creating hybrid versions with other algorithms [131]. Since its original version in 1995, the PSO algorithm has been expanded to solve a variety of different problems [100]. Multimodal, constrained, and multiobjective optimization problems are some of the most prominent applications that have been addressed with the PSO algorithm [132]. But the question is, why does PSO outperform BP? The study of PSO to optimize ANN has had very good results, but there is no in-depth research on theoretical aspects. Nevertheless, there are works that have tried to explain the PSO convergence. In 2002, [PERSON] and [PERSON] [107] analyzed the full stochastic system of a deterministic version of PSO to supply knowledge about its search mechanism. They proposed reducing the particle velocity formula (Equation (15)) by redefining it as follows:  
\[v_{t+1}=\omega*v_{t}+\phi*(P-x_{t}) \tag{17}\]where \(P=\frac{c_{1}*U(0,1)*P_{Best}+c_{2}*U(0,1)+C_{Best}}{c_{1}*U(0,1)+c_{2}*U(0,1)}\) and \(\phi=c_{1}*U(0,1)+c_{2}*U(0,1)\). This is algebraically identical to the standard two-term form [133]. The analysis begins by removing all coverings from a particle, for example, a population of a one-dimensional deterministic particle, with a constant \(P\) and a constant \(\phi\). [PERSON] [133] observed that the value of the parameter \(\phi\) controls the trajectory of the particle and recognized that the explosion of the system depended on randomness. In [134], the authors analyzed the same system and concluded that the particle trajectories follow periodic sinusoidal waves. In their work, [PERSON] and [PERSON] [107] analyzed the movement in discrete time of the PSO, advancing to its visualization in continuous time. These analyses lead to a proposal of controlling the convergence tendencies of the system through a set of coefficients, resulting in a generalized model of the algorithm. When they re-introduced randomness into the PSO with constriction coefficients, the deleterious effects of randomness were seen to be controlled.