## 2 Particle Swarm Optimization
### Background Information Based on Recent Research on the Parallelization of the PSO  
The parallelisation of the PSO algorithm for training neural networks represents an efficient and scalable approach with several practical applications and notable advantages. These include the ability to process a DLNN with many layers of intermediate neurons in a much shorter time frame than with traditional implementations that do not take advantage of the massive parallelism currently provided by multicomputers or, in our study, by general graphics processing units (GPGPU). This renders it an invaluable instrument in the domain of machine learning and optimisation, particularly in comparison with other methodological approaches to solving the same problem, as illustrated in Table 1.  
There are currently many implementations of PSO using the CUDA/GPU environment. One of them, called PSO-GPU [1], presents a generic and customisable implementation of a PSO algorithm on top of the CUDA architecture. This implementation exploits the thousands of threads running on the GPU to reduce execution time and increase performance through parallel processing. FastPSO [17] implements the PSO algorithm using CUDA with a Structure of Arrays (SoA) layout. This approach is designed to achieve optimal performance by leveraging shared memory and tensor cores. The utilisation of shared memory markedly enhances the performance of the PSO algorithm on GPUs by minimising the latency associated with global memory accesses. However, shared memory is inherently constrained in capacity. To address this challenge, the provided implementation segments the arrays into tiles of smaller size than the original arrays. Each tile is loaded into shared memory, and tile-wise instructions are executed at high speeds. The results are stored back into global memory. To further improve performance, tensor cores are used for matrix operations. This approach ensures efficient parallel processing and optimises the GPU's capabilities to accelerate the PSO algorithm. The CuPSO [18] algorithm employs a shared queue for all thread blocks and per-block data structures to store the intermediate results, allowing each block of the GPU grid to independently track and store the best solutions found by the threads within that block. This design leverages both shared memory for collaboration and per-block storage for independent evaluation, thereby facilitating efficient parallel optimisation. One strategy currently employed to enhance the performance of PSO algorithm implementations is to adjust their control parameters, thus, achieving enhanced efficiency without compromising precision in determining the objective, as described in [13]. In the article [2], an asynchronous parallel PSO algorithm is presented that significantly improves parallel efficiency. The majority of parallel PSO algorithms have been implemented synchronously, where the fitness functions, positions, and velocities of all particles are evaluated in one iteration before the next iteration starts. These latter approaches usually result in a modest increase in the speed of parallel computation over the sequential variant in cases where a heterogeneous parallel environment is used and/or where the execution time depends on the element of the algorithm being computed. Therefore, the work presented in [2] was the first work to investigate the potential of an an asynchronous parallel implementation of the PSO algorithm and, at the time of its publication, it demonstrated a notable enhancement of PSO implementations. This approach had analogous objectives to those of our study, which concentrated on accelerating the asynchronous PSO algorithm in comparison to the synchronous one. In [20], a hybrid mechanism combining