## 3 Materials and Methods
### Distributed Deep Learning  
DDL is a methodology that plays a fundamental role in training large datasets, a task that often requires harnessing the computational power of multiple CPUs and GPUs. This approach is especially crucial when dealing with image quantities. By distributing the workload across multiple processing units, DDL enables the simultaneous training of CNN models or architectures, significantly reducing the time required for their convergence [38]. The scalability offered by DDL not only addresses the computational or processing demands posed by large datasets but also improves the training efficiency and accuracy. Additionally, the methodology allows practitioners to leverage the capabilities of parallel computing, unlocking the potential for accelerated training of CNN models and improved performance when addressing complex ML tasks. To distribute CNN models among devices, DDL employs synchronization via communication mechanisms. This crucial aspect of DDL ensures that model parameters and updates are consistently shared and coordinated across distributed devices during the training process, thereby maintaining the integrity and accuracy of the CNN model formed by complex features [39]. DDL leverages various communication methods, such as message passing, to facilitate the exchange of information between devices. In addition to message passing, DDL often incorporates collective communication operations like all-reduce. The all-reduce algorithm is particularly valuable for DDL as it combines information across multiple devices efficiently. This operation helps synchronize the gradients and model parameters among devices by performing a reduction operation, such as summation or averaging, across all participating nodes [40]. Synchronization points are strategically placed to harmonize the training progress across different processing units to achieve a unified understanding of the model state and collectively move toward convergence. Additionally, effective communication in DDL enables task parallelization, allowing devices to collaboratively handle the parts of the dataset. This parallel processing not only speeds up training but also optimizes resource utilization. To implement both message passing and all-reduce in the context of DDL, libraries such as the Message Passing Interface (MPI) and NVIDIA Collective Communication Library (NCCL) are commonly utilized to distribute processes across devices [41]. These libraries provide essential functionalities for communication, synchronization, and coordination in distributed computing systems. MPI is a widely adopted standard for message-passing in distributed computing. It allows processes to communicate with each other, making it suitable for DDL implementations that involve distributed training across multiple CPU nodes. The MPI enables the exchange of messages among different processes, thereby facilitating seamless communication and synchronization. MPI supports collective communication operations, such as broadcast, scatter, gather, and reduce, which enable coordination and data exchange among a group of processes [42]. NCCL was specifically developed to optimize collective communication operations in environments with multiple NVIDIA GPUs. Its design is geared toward maximizing performance in DDL implementations that leverage the parallel processing capabilities of GPUs. NCCL offers highly efficient implementations of collective communication primitives, such as all-reduce, broadcast, and reduce, which are critical for synchronizing model parameters and gradients across GPU devices, thus facilitating efficient scalability in heterogeneous clusters [9]. Within parallel computing, Amdahl's Law is a fundamental principle that helps define and identify the limits of applications or the scalability of distributed computing infrastructures [43]. Because DDL is a parallel computing application, it also adapts to Amdahl's Law, which provides insights into the potential speedup achievable through parallelization.