# Fitness Landscape Analysis of Product Unit Neural Networks
## 5 Training Algorithms for Product Unit Neural Networks
### Particle Swarm Optimization  
PSO is a population-based stochastic search algorithm inspired by the flocking behavior of birds [39]. A swarm of particles is maintained, where the position of each particle is adjusted according to its own experience and that of its neighbors while trying to maintain the previous search direction [14]. The positions of the particles are adjusted by adding a velocity, \(\mathbf{v}_{i}(t)\), to the current position, \(\mathbf{v}_{i}(t)\), as follows:  
\[\mathbf{x}_{i}(t+1)=\mathbf{x}_{i}(t)+\mathbf{v}_{i}(t+1) \tag{6}\]  
The optimization process is driven by the velocity vector, which reflects the experiential knowledge of the particle and socially exchanged information from the particle's neighborhood. The experiential knowledge of a particle is generally referred to as the cognitive component and the socially exchanged information is referred to as the social component of the velocity equation. The inertia global best (_gbest_) PSO [40] is considered for the purposes of this study, for which the social component of the particle velocity update reflects information obtained from all the particles in the swarm. The social information is the best position found by the swarm, and is referred to as \(\mathbf{\hat{y}}(t)\)[14]. The velocity of particle \(i\) is calculated as  
\[v_{ij}(t+1)=\omega v_{ij}(t)+c_{1}r_{ij}(t)(y_{ij}(t)-x_{ij}(t))+c_{2}r_{2j}(t) (\hat{y}_{j}(t)-x_{ij}(t)) \tag{7}\]  
where \(v_{ij}(t)\) and \(x_{ij}(t)\) are the velocity and position, respectively, in dimension \(j\) at time \(t\). The inertia weight is given by \(\omega\), while \(c_{1}\) and \(c_{2}\) are the acceleration constants. The stochastic element of the PSO is incorporated with the random variables \(r_{1 ij}(t)\), \(r_{2 ij}(t)\), which are sampled from a uniform distribution over \([0,1]\). Finally, \(y_{ij}(t)\) and \(\hat{y}_{j}(t)\) denote the personal and global best positions, respectively, in dimension \(j\) for particle \(i\).  
The performance of the PSO algorithm has been shown to be sensitive to the choice of control parameter values [41]. The control parameter values must be chosen such that a good balance of exploration and exploitation is obtained. [PERSON] and [PERSON] [42] found empirically that \(\omega=0.7298\) and \(c_{1}=c_{2}=1.496\) lead to convergent trajectories. This control parameter value combination satisfies theoretically derived stability conditions and exhibits strong performance characteristics [43].  
In the context of NN training, each particle represents the weights and biases for one NN. Pseudocode for the _gbest_ PSO is provided in Algorithm 2.  
```
Create and initialize an \(n_{x}\)-dimensional swarm; whilestopping condition(s) not truedo for each particle i = 1,...,\(n_{s}\)do // set the personal best position; if\(f(x_{i})<f(y_{i})\)then \(y_{i}=x_{i}\);  end if // set the global best position; if\(f(y_{i})<f(\hat{y})\)then \(\hat{y}=y_{i}\);  end if  end for for each particle i = 1,...,\(n_{s}\)do  update the velocity using equation (17);  update the position using equation (16);  end for  end for
```  
**Algorithm 2**Pseudocode for the inertia _gbest_ PSO.