# Collective Communication Performance Evaluation for Distributed Deep Learning Training
## 1 Introduction  
Deep learning performance depends on two main factors: the number of parameters in the model and the size of the training dataset. As these parameters and data continue to grow, we see an increase in the size of deep learning models and training data [1]. For instance, GPT-3, a massive deep learning model, required 2048 CPUs and 2048 high-performance GPUs to train, with around 175 billion parameters and a 650 GB dataset [2]. To train such large-scale deep learning models, we need specialized GPU clusters. However, when setting up a GPU cluster, latency issues arise due to the communication between cluster components. When deep learning tasks are executed on a large GPU cluster, the time spent on collective communication can be more than ten times the time spent on actual computation [3]. This performance bottleneck affects the overall training process. Furthermore, different communication libraries employ various methods for collective communication, which means that it is crucial to choose the right communication library that suits the specific environment to optimize distributed deep learning cluster training. PyTorch supports several libraries for collective communication, including the Message Passing Interface (MPI) [4], \"GLOO\" [5], and the NVIDIA Collective Communication Library (NCCL) [6]. MPI is a standard interface used for distributed memory computing. It consists of a message-based communication protocol and a library set. MPI is versatile, working across various platforms and finding applications in cluster computing,supercomputers, and cloud environments. It excels at handling large datasets and parallel computations. GLOO, on the other hand, is a collective communication library developed by Facebook that is specifically tailored for distributed deep learning. It focuses on efficient communication in large-scale deep learning models while maintaining scalability based on the cluster and model size. Unlike MPI, GLOO is not a general-purpose library; its specialization lies in deep learning tasks. NCCL, or the NVIDIA Collective Communications Library, was developed by NVIDIA to enhance efficient communication between GPUs. NCCL is compatible with CUDA-based applications and provides swift communication on NVIDIA GPUs. It enables direct communication between GPUs, ensuring rapid data exchange. Moreover, NCCL optimizes data movement between GPUs by employing various communication patterns and algorithms. This enhancement translates into faster training speeds and improved performance for distributed deep learning models. Consequently, NCCL proves invaluable for training large-scale GPU clusters. Given these distinctions, it is essential to analyze these three collective communication libraries in the context of each type of distributed deep learning environment. Latency in collective communication primarily involves two key factors. First, it involves the delay incurred when invoking functions of collective communication libraries for communication between nodes. This latency depends on factors like communication protocols, network conditions, the distance between nodes, and more. Second, there is latency in copying data (such as parameters and gradients) needed for collective communication between the system memory and GPU memory [7]. This latency is influenced by factors like the amount of data and the speed of memory transfers. To understand these fundamental latency components, we conducted a performance analysis in the Linux shell environment. Additionally, since distributed deep learning typically involves deep learning frameworks, it is essential to examine collective communication latency in these environments. Therefore, in this paper, we conducted additional experiments in the PyTorch environment, which is a widely used deep learning framework. Cloud-based GPU cluster servers, like those offered by Amazon Web Services and Google Cloud, are gaining popularity in the business world.