## 2 Literature Review
### Multi-Start Gradient Descent Algorithms  
A recently common method for mitigating the suboptimality problem in gradient-based algorithms is multi-start gradient-based optimization. This approach involves initiating different potential solutions from various positions in the cost function landscape and running the gradient-based search for all potential solutions simultaneously, rather than sequentially as seen in restart-based algorithms. This method helps escape local optima with less computational effort. However, running multiple instances of the algorithm simultaneously raises concerns about execution time penalties, which motivates research into improving the computational efficiency of multi-start algorithms. For instance, [PERSON] et al. in 2012 [31] introduced the idea of replacing the computationally expensive cost function with a surrogate model. While this approach is innovative, it still requires the additional step of fitting the surrogate model. Another issue that increases the complexity of this algorithm is the use of de-clustering for the initialization of potential solutions, which itself represents an extra computational cost and complicates implementation. In 2021, [PERSON] et al. [10] developed an algorithm called METOD (Multi-start with Early Termination Of Descents), which is similar to sequential restart algorithms but incorporates memory, terminating any new point that belongs to an already explored region. However, the sequential updating of instances and the iterations needed for the local search of each instance can significantly increase execution time. Additionally, there are computational costs associated with checking whether each instance is located in an already explored region, alongside the extra hyperparameter tuning required to control this process. Swarm-Based Optimization Algorithms and Their Application as Alternatives to Gradient Descent in Neural Network Training  
Swarm-based algorithms are a subset of metaheuristic optimization techniques inspired by the collective behavior of decentralized, self-organized systems found in nature [32], such as flocks of birds, schools of fish, and colonies of ants. These algorithms have proven effective in solving complex optimization problems, particularly those that are difficult to address with traditional methods [33] (p. 1627). One of the earliest swarm-based algorithms, the Ant Colony Optimization (ACO), was introduced by [PERSON] et al. in 1996 [34], inspired by the foraging behavior of ants. Similarly, the Particle Swarm Optimization (PSO) algorithm was developed by [PERSON] et al. in 1995 [11], drawing inspiration from the social behavior of bird flocks and fish schools. [PERSON] et al. [35] developed the Artificial Bee Colony (ABC) algorithm in 2005, which is inspired by the foraging behavior of honeybees. Another effective algorithm for global optimization problems is the Cuckoo Search (CS) algorithm, introduced by [PERSON] et al. in 2009 [36]. This algorithm employs Levy flights to enhance global search capabilities, thereby effectively avoiding local minima. The Grey Wolf Optimizer (GWO), introduced by [PERSON] et al. in 2014 [37], mimics the leadership hierarchy and encircling behavior of wolf packs to balance exploration and exploitation during the search for optimal solutions. Numerous researchers have explored the application of swarm-based optimizers as alternatives to gradient descent algorithms for various local optimal problems. [PERSON] et al. [36]tested the CS and PSO algorithms on multiple benchmark mathematical functions with multiple local minima, demonstrating their efficiency in finding better solutions. In 2010, [PERSON] et al. [38] introduced the BAT algorithm, proving its effectiveness in identifying global optima. Similarly, [PERSON] et al. [37] validated the efficiency of the GWO algorithm for problems with multiple local optima. In 2019, [PERSON] et al. [39] developed a new algorithm called HACPSO, derived from APSO and CS algorithms, specifically for training artificial neural networks (ANNs).