# Collective Communication Performance Evaluation for Distributed Deep Learning Training
## 2 Background
### Collective Communication Libraries for Distributed Deep Learning  
MPI, GLOO, and NCCL are three prominent collective communication libraries employed in distributed deep learning. MPI is a widely adopted standard interface for high-performance parallel computing [21]. It facilitates message exchange and synchronization among multiple processes within a distributed memory system. MPI is instrumental in enabling parallel operations, with each process possessing a unique rank and the ability to send and receive messages to and from other processes. This framework provides various synchronization functions to control the order of operations and prevent deadlock situations. MPI is available in multiple programming languages (such as C, C++, and Fortran) and finds extensive use in high-performance computing clusters and supercomputers. Additionally, MPI serves as a communication library not only in standalone programs but also within distributed deep learning frameworks.  
Figure 3: Data exchange behaviors in five communication primitives.  
GLOO is a distributed parallel processing framework developed by Facebook, primarily integrated into the PyTorch deep learning framework. It specializes in facilitating distributed deep learning training. GLOO's primary role is to exchange and synchronize data among multiple worker nodes. It is optimized for handling small data exchange tasks, particularly the parameter updates of deep learning models. GLOO provides a range of distributed algorithms designed to efficiently communicate between multiple workers. Its exceptional performance is evident when each worker engages in small-sized data exchange tasks. GLOO supports CPU-based distributed processing and can be used in conjunction with MPI or other communication libraries. It enjoys widespread use in distributed deep learning model training and serves as the foundation for PyTorch's distributed training functionality.  
NCCL is a specialized library developed by NVIDIA for enabling collective communication between GPUs. Its primary purpose is to optimize distributed processing that involves multiple GPUs during deep learning model training. NCCL is compatible with CUDA-based GPU operations and excels in streamlining data transfer and synchronization among GPUs. This optimization significantly accelerates operations like parameter updates in deep learning models. NCCL demonstrates excellent performance in multi-GPU systems, making it particularly valuable for expediting training tasks involving large-scale deep learning models and datasets. It can also be seamlessly integrated with other communication libraries and is commonly used in conjunction with Horovod [22], a distributed deep learning framework.  
In summary, MPI, GLOO, and NCCL are collective communication libraries employed for parallel computing and distributed processing, each serving distinct fields and purposes. MPI is a standard interface in high-performance computing, while GLOO and NCCL are primarily tailored for distributed deep learning training, with GLOO optimized for CPU-based processing and NCCL specialized for GPU-based operations.